ELK & Nagios Part1: How to get your Application Logs to Redis

 

The easiest way to collect your Application logs (WebSphere, TDI, DB2…) from your servers and send them to Logstash for processing is to use Filebeat as shipper.

Filebeat gives you the possibilty to output your logs directly to Logstash but I prefer to send them first to a message broker. Reason for this is that the message broker can store all messages even if logstash isn’t available and therefore acts as a perfect buffer.

So first step is to install Filebeat on your servers, fortunately almost every OS is supported by Filebeat: https://www.elastic.co/downloads/beats/filebeat

Next step is to configure filebeat to send your logs to Redis, this is my example which I use for my WebSphere servers:

As you can see Filebeat will collect all log messages from SystemOut.log in all folders which can be found under  /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/

Filebeat even supports to join multiline error stack messages to one line, here I check if the line doesn’t start with [ and if true, the line will be added to the line before.

At the end I send all messages my Redis server which listens on port 32768 (and which by the way is a docker container).

In the next part I will cover how to bring theses messages from Redis to Logstash and how to process them, stay tuned.