Fix Kafka Consumer Lag
Lag — the count of unprocessed messages in a partition. Diagnose: kafka-consumer-groups.sh --describe. Four causes: (1) consumer is too slow (CPU/DB bottleneck), (2) fewer partitions than consumers in the group (idle consumers), (3) max.poll.interval.ms < processing time (rebalance loop), (4) slow downstream (DB, external API). Fixes: scale up partitions, parallelize, tune max.poll.records.
Below: details, example, related terms, FAQ.
Details
- kafka-consumer-groups.sh --describe shows lag per partition
- Partitions > consumers: under-utilization. Partitions < consumers: idle ones
- max.poll.records default 500 — lower it for heavy processing
- max.poll.interval.ms default 5min — make it > worst-case processing time
- Monitoring: Burrow, Cruise Control, Kafka-UI for visual
Example
# Diagnose lag
$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--group my-consumer --describe
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
my-consumer orders 0 10000 15000 5000
# Increase partitions (non-reversible)
$ kafka-topics.sh --alter --topic orders --partitions 12Related
Why teams trust us
How it works
Enter site URL
Analyse 10+ factors
Get overall score
What is Health Score?
Health Score is a comprehensive assessment of site technical health across 20+ parameters: SSL, security headers, response speed, SEO technical factors, and availability.
Comprehensive Score
20+ parameters in one number: SSL, headers, speed, SEO technical factors.
Detailed Breakdown
Each parameter with explanation — what is checked, what was found, how to fix.
Score Trends
Compare Health Score at different dates — see progress or regression.
Health Monitoring
Set up automated Health Score checks and get notified when the score drops.
Who uses this
Developers
quick pre-release audit
SEO
technical baseline score
Marketers
client site check
Security
header security audit
Common Mistakes
Best Practices
Get more with a free account
Health Score check history and real-time site health monitoring.
Sign up freeLearn more
Frequently Asked Questions
How many partitions are enough?
Rule of thumb: 10× expected throughput in MBps. 1 partition = ~10 MB/s. 100 MB/s throughput → 10+ partitions.
Can I reduce partitions?
No, it's non-reversible. Create a new topic with fewer, switch producers, migrate consumers.
Compaction vs retention?
Retention: deletes by time (7 days default). Compaction: keeps the latest value per key — useful for event sourcing / state stores.