Что такое Log Aggregation
Log aggregation — practice сбора logs из multiple services в central searchable store. Причина: grep по 50 servers — не scale. Stack options: ELK (Elasticsearch + Logstash + Kibana) — powerful но expensive, Loki (Grafana, cheaper), Splunk (enterprise $$$), CloudWatch/DataDog Logs (SaaS). Critical features: search, alerts, retention, correlation с traces.
Ниже: подробности, пример, смежные термины, FAQ.
Подробности
- Collector (node level): Filebeat, Fluent Bit, Vector, Promtail
- Pipeline: parsing (JSON, multiline), enrichment (host, trace_id), routing
- Storage: Elasticsearch (indexed, $$), Loki (chunks, $), S3 + Athena (archive, cheapest)
- Retention: hot (7d, fast) + warm (30d, slower) + cold (1y+, S3)
- Cost volatility: DEBUG logs в prod → 10× spend. Log level discipline critical
Пример
# Fluent Bit config
[INPUT]
Name tail
Path /var/log/nginx/access.log
[OUTPUT]
Name loki
Host grafana-loki:3100
Labels host=$HOSTNAME,service=nginxСмежные термины
Больше по теме
Часто задаваемые вопросы
ELK vs Loki?
ELK: full-text indexed, fast search, expensive at scale. Loki: Prometheus-like labels + grep при query time, 10× cheaper. Для high-volume — Loki. Для complex search — ELK.
Cost control?
Sampling (drop 90% INFO logs), log level discipline (INFO/WARN/ERROR не DEBUG в prod), TTL (< 30 days hot).
Centralize из multi-region?
Ingestion в nearest region + async replication. Или separate stores + federated search (Loki federation, CloudWatch cross-account).