Turbo charge logs with ELK and Graylog2

Who?
●

●

●

●

●

Ex-pat Englishman, now living in
Southern Ontario.
Web developer for 5 years, mostly
PHP.
(Almost) senior software engineer at
TribeHR.
Co-organiser of Guelph PHP User
Group.
Ex-professional musician.

Your app is trying to talk to
you!
●

●
●

●

Logs are the way your app speaks
to you.
Ignore log messages at your peril...
Logging is the L in LUCID
development.

http://crisscott.com/2012/09/11/lucid-developme

Using log data is hard
●

Not in a human friendly format.

●

A lot of data.

●

Many log files.

●

Potentially many servers.

Use your logs proactively

How can we stop using log
data reactively and start
using it pro-actively?

The 'ideal' logging setup
●
●

Centralised.
Accepts logs from application code,
software and the OS.

●

Performant.

●

Scalable.

●

Easily searchable.

●

Alarms and alerting.

RFC 5424 logging levels
●

Debug

●

Info

●

Notice

●

Warning

●

Error

●

Critical

●

Alert

●

Emergency

What's wrong with
error_log?
●

Nothing at all but...

●

It's limited:
–

Have to format the message yourself.

–

Limited number of destinations.

–

Doesn't support logging levels defined
in RFC 5424.

Introducing Monolog
●

PHP 5.3+ logging library by Jordi
Boggiano based on Python's Log
Book library.

●

PSR-3 compliant.

●

Supports RFC 5424.

Installing Monolog
●

●

●

Symfony2, Laravel4, Silex and PPI
all come with Monolog.
CakePHP and Slim have have plugins to use it.
Most easily installed with
Composer:

Monolog concepts
●

Channels.

●

Handlers.

●

Formatters.

●

Processors.

Channels
●

●

●

A channel is a name or category for
a logger.
Each logger instance is given a
channel when instantiated.
Allows for multiple loggers, each
with a different channel.

Handlers
●

●

●

●

Handlers write log messages to a
storage medium.
Multiple handlers can be attached to
each logger.
Set lowest level handler logs at and
if it 'bubbles'.
Many handlers available or you can
write your own.

Example handlers
Files/Syslog

Notifications

●

Stream Handler

●

Mail handlers

●

Rotating File Handler

●

Pushover Handler

●

Syslog Handler

●

HipChat Handler
Debugging

Networked Logging
●

Socket Handler

●

AMQP Handler

●

Gelf Handler

●

Zend Monitor Handler

●

●

FirePHP Handler
ChromePHP
Handler

Formatters
●

●

Processes a log message into the
appropriate format for a handler.
Each handler has a default
formatter to use but this can be
overridden.

Processors
●

●

●

Used to amend or add to the log
message.
PHP callable, called when a
message is logged.
Built in processors available:
–

IntrospectionProcessor

–

WebProcessor

–

MemoryUsageProcessor

–

MemoryPeakUsageProcessor

–

ProcessIdProcessor

–

UidProcessor

Where does this get us
to?
●
●

Centralised. Maybe...
Accepts messages from application
code, software and the OS.

●

Performant. Maybe...

●

Scalable. Maybe...

●

Easily searchable.

●

Alarms and alerting. Yes but crude.

Why Syslog?
●

●

Loggable events don't only happen
in code!
To get a full picture of what's going
on we need to monitor what's going
on in other services too.

Syslog basics
●

●

●

●

OS daemon to process log
messages.
Messages are assigned a facility,
such as auth, authpriv, daemon or
cron or a custom one.
Messages are also assigned a
severity, defined in RFC 5424.
Messages can be sent to files,
console or a remote location.

Which Syslog daemon
to use?
●

In part will depend on your OS.

●

Things to consider:
–

Syslog is the oldest with not as many
features.

–

Syslog-ng is produced under a dual
license.

–

Rsyslog fully featured and open
source.

Introduction to Rsyslog
●

Fork of syslog by Rainer Gerhards.

●

Drop in replacement for syslog.

●

●

Many, many features including
plugin system for extending.
Default syslogger in Debian, can be
installed on other distros too.

Remote logging with
Rsyslog
●

Rsyslog can be configured to work
in a client-server setup.
–

–

●

One or more machines are setup as
clients to forward log messages.
One machine is setup to receive and
store them.

Probably want to filter sender on the
receiving machine...

Leveling up with Rsyslog
●

●

Apache can send all error logs to syslog
directly.

Rsyslog can also monitor other log files
using the Text File Input module.
–

Example of monitoring Apache access log at
https://gist.github.com/joseph12631/2580615

Where does this get us?
●
●

Centralised. Yes.
code, software and the OS.
Possibly.

●

Performant. Depends.

●

Scalable. Depends.

●

Easily searchable.

●

Alarms and alerting. Yes but crude.

Taking it further with
Logstash

What is Logstash?
●

●

●

Tool to collect, filter and output log
messages.
Built in web interface or richer web
interface project called Kibana
available.
Full information at
http://logstash.net/ and Kibana
demo at http://demo.logstash.net/

Installing Logstash
●

●

Current release is 1.3.3 and can be
downloaded from here.
Run from cli, use supervisord or an
init.d/upstart script (cookbook entry
on how to do this at
http://cookbook.logstash.net/).

Inputs, filters and outputs
Inputs
–

AMQP/RabbitMQ

–

Syslog

–

Varnishlog
Outputs

–

Statsd

–

XMPP

–

AMQP/RabbitMQ

–

Nagios

–

Graphite

Filters
–

Anonymize

–

Grok

–

Geoip

–

Mutate

Logstash config
●

●

●

When starting specify the path to a
config file for Logstash to use.
Three main sections: input, filter
and output.
Each section may have multiple
instances of each type.

Sample configuration file
input {
file {
path => "/var/log/apache2/*access.log"
type => "apache"
}
}
filter {
if [type] == "apache" {
grok {
pattern => "%{COMBINEDAPACHELOG}"
}
}
}
output {
redis { host => "10.0.0.5" data_type => "list" key => "logstash" }
}
●

See http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/

Where does this get us?
●
●

Centralised. Yes.
code, software and the OS. Yes.

●

Performant. Yes.

●

Scalable. Yes.

●

Easily searchable. Possibly.

●

Alarms and alerting. Yes.

What is Graylog2?
●
●

●

●

Log storage and search application.
Can accept thousands of messages
per second and store terabytes of
data.
Web interface for searching and
analytics.
Built in alerting and metrics.

Installing Graylog2
●

Components:
–
–

Graylog2 server

–

●

MongoDb

–

●

Elasticsearch

Graylog2 web interface

Full info on installing at
http://support.torch.sh/help/kb
Live demo at
http://public-graylog2.taulia.com/login

Getting log messages into
Graylog2
●

Can accept log messages in 3
ways:
–

–

Syslog via UDP or TCP.

–
●

Graylog Extended Log Format (GELF)
via UDP .
AMQP.

Multiple Graylog2 server instances
can be run in parallel.

Graylog2 web interface
●

●

Main view shows recent log
messages and graphs of recent
message numbers.
Single message can be clicked on
to view all details for it.

●

Dashboard views.

●

Full search functionality.

●

Analytics dashboard and metrics.

Details of an individual
message

Searches and streams
●

●

●

Web interface allows fine grained
searching by different fields.
Frequently used searches can be
saved as streams.
Streams can be marked as
favourites by users and can be
viewed as dashboards.

Stream alarms
●

●

Alarms can be sent for a stream
with user defined sensitivity.
Plugins for sending alarms include:
–
–

PagerDuty

–

HipChat

–

Twilio SMS

–
●

Email

Jabber/XMPP

You can also write your own

Where does this get to?
●
●

Centralised. Yes.
code, software and the OS. Yes.

●

Performant. Yes.

●

Scalable. Yes.

●

Easily searchable. Yes.

●

Alarms and alerting. Yes.

Thanks for listening
●

Contact me:
–
–

●

jeremycook0@gmail.com
@JCook21

Questions?

Putting it all together
A few possible implementations.

Turbo charge logs with ELK and Graylog2

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Turbo charge logs with ELK and Graylog2

Similar to Turbo charge logs with ELK and Graylog2 (20)

More from Jeremy Cook

More from Jeremy Cook (8)

Recently uploaded

Recently uploaded (20)

Turbo charge logs with ELK and Graylog2

Editor's Notes