Topics Discussed:
- Introduction to Observability
- ELK Running on Kubernetes
- Agents Stack Deployment
- Kafka into Elastic Stack
- Gitops Driven
- Monitoring Cluster of Elastic Stack
- ElasticSearch Cluster Upgrade
- Demo
Introduction to Observability
Module 1 – ELK Running on Kubernetes
● Elastic cloud operator for Kubernetes.
● ECK Operator available in official elastic github repo.
● Multi Data Tier Architecture (Using Nodesets) for managing index properly
● ElasticsearchRef should be mentioned in kibana manifest
Agents Stack Deployment
● Filebeat
Aggregates container logs to elastic search
● Metricbeat
Aggregates metrics to elastic search
Kafka into Elastic Stack
● Intermediate layer between source and destination.
● We can stage our data in Kafka for optimal performance
● Kafka has a retention period which helps reliability, fault tolerance even if Elastic search fails ( attacks like ddos ) for some times. ● Overloading to the cluster can be avoided.
● Kafka logstash plugin is available for log processing.
Gitops Driven
● Integration with ArgoCD for automatically detecting the manifests configuration changes and apply into the kubernetes cluster.
Monitoring Cluster of Elastic Stack
● Capture the metrics of the performance of Elastic search clusters.
● Rate of ingestion
● Rate of querying by user
● How elastic cluster is performing inside the kubernetes cluster
ElasticSearch Cluster Upgrade
● Check the snapshots to have the latest logs
● Upgrade the operator
● Upgrade elastic search (update the release versions in the manifests) to have a rolling update (Zero downtime).
● Upgrade the kibana
Demo:-
Menu -> Devtools:-
● GET _cat/indices – Gives all indices shipped into the elastic search with details (status, name, replica_status, size, replica_size)
Menu -> Data -> Index Management
● Manage index lifecycle
Menu -> Visualise Library -> Create new Visualisation -> Lens
● Can create dashboards