Thursday, January 18, 2018

Scalable ELK Architecture


  1. Nginx logs
  2. Application logs
  3. System logs ( To be decided )  

This setup will unify the application logging and nginx logs.

Events Producers
These are our standard instances which will produce logs.

Elastic Beats will be used as specialized shippers here and it has the capacity to send logs directly to Kafka cluster.

Kafka Cluster
Like many other message brokers, it deals with publisher-consumer and queue semantics by grouping data into topics. As an application, you write to a topic and consume from a topic. An important distinction, or a shift in design with Kafka is that the complexity moves from producer to consumers, and it heavily uses the file system cache.

Kafka has a dependency on Apache ZooKeeper, so will need access to a ZooKeeper cluster.

Logstash Indexers
This cluster will consume data, at its own throttled speed, while performing expensive transformations like Grok, DNS lookup and indexing into Elasticsearch. This cluster is called the Indexer.

ElasticSearch Cluster
This will be our main Elasticsearch cluster.

Kibana Node
Kibana will be installed on this node and it will contact Elasticsearch cluster for the data.


Post a Comment