Setting up your own ELK solution is easy, there are many resources available, for example this DrupalCon NOLA presentation Drupal and Logstash: centralised logging. But how do you make it cope with a massive spike in number of logs, when you need to be able to process hundreds of events per second? How do you make it scale?
This session was presented at DrupalCon Dublin (youtube video is there).
This will be a technical talk targeting sysadmims and systems savvy developers, presenting a possible High Available ELK solution, capable of receiving logs and metrics from different servers and Drupal environments.
We will cover some advanced topics and common problems, for example
- designing scalable ELK stack
- logstash indexer autoscaling
- preventing elasticsearch to run out of diskspace (S3 backup, curator)
- securing log transmission with TLS/SSL, ssl offloading tricks, ELB
- upgrading your ELK stack without downtime
- different ways of getting logs from Drupal to logstash
