Kafka Consumer Lag Troubleshooting and Offset Recovery: A Practical Guide

🌏 閱讀中文版本

This article provides a concrete process for safely resetting Kafka offsets and establishing stop conditions when accumulated Consumer Lag causes system latency in an Apache Kafka cluster. The goal is to adjust the consumption boundary and restore service availability.

Reproduction environment requirements:

• Kafka Broker: 2.8+ (CLI syntax is based on kafka-consumer-groups.sh)
• CLI availability: The operations host must be able to run kafka-consumer-groups.sh. Confirm that its package version or existing deployment inventory is compatible with the target Broker. Stop if the command is unavailable, cannot run, or the version is incompatible.
• Target cluster and authentication: --bootstrap-server must point to the approved cluster for this operation. SSL, SASL, or other existing authentication settings must be loaded by the same operating identity. Run --describe --state with the same connection parameters. Continue only when it returns the target Group status successfully and shows no connection timeout or authentication error.
• Offset operation permissions: The operating identity must be allowed to query Consumer Groups, read target Topic metadata, and alter committed offsets. Confirm this through the existing permission inventory or approval record, then observe whether --describe --state, --describe, and the dry run produce authorization errors. Stop if any check fails; do not work around it with another identity.
• Recovery file directory: The export directory must exist, and the same operating identity must be able to create a new file without overwriting existing content. If it cannot create the file, space is insufficient, or the target filename already exists, stop the reset and preserve the existing record.
• Consumer Group: Inactive (Empty or Dead, with #MEMBERS equal to 0)
• Target operation: Export and preserve the original boundary with --export, then use --from-file with --dry-run and --execute to reset or roll back offsets for the designated Topic, passing all four success criteria.

If any connection, authentication, authorization, CLI, or file-write check does not pass, do not enter the offset-write process.

Lag is the difference between the latest message offset in a Partition and the committed offset of a Consumer Group. Changes in this number reveal system dynamics: if Lag continues to grow, producers are outpacing consumers or consumers are blocked; if Lag declines slowly, a consumer performance bottleneck remains; if Lag jumps or returns to zero, it is usually related to a rebalance, an offset reset, or an updated measurement definition.

This article provides a structured troubleshooting framework for safely adjusting offsets while keeping data boundaries clear and restoring system balance. With standardized tools and a decision matrix, teams can reduce risk quickly during an incident while clearly recording the boundary conditions and recovery path for every operation.

Pre-Reset Investigation and Stop Conditions

Before resetting any offset, first clarify the cause and current state of the Lag. An unverified offset adjustment directly changes the consumption boundary and can introduce duplicate or missing data, so clear stop conditions and observation metrics are essential.

Definition: Consumer Lag refers to the difference between the latest offset in a partition and the current committed offset of a consumer group.

Quantify Business Impact: How Urgent Is It?

Before deciding whether an Offset Reset is needed, quantify the actual delay Lag creates for the business. This is essential for PMs and decision-makers evaluating data freshness.

Formula:

$$\text{Estimated Delay (Time)} = \frac{\text{Current Lag (Messages)}}{\text{Consumption Rate (Messages/Second)}}$$

This formula applies only when there is no incoming traffic, or when the production rate is known and stable. It represents the time needed to clear the existing backlog. If producers continue adding messages, calculate the actual time to clear it with Lag ÷ (Consumption Rate - Production Rate).

For example, if the current Lag is 1,000,000 messages and consumers process 5,000 msg/s, assuming a relatively stable production rate, the system delay has reached 200 seconds. If the business scenario, such as payment confirmation, requires latency below five seconds, this is a high-priority event that needs immediate intervention. If it is only log collection, observing the trend may be sufficient.

Note: When a Group is inactive, Lag not increasing only means that the Log End Offset has not advanced or monitoring has not updated. It does not mean production and consumption are balanced, because the consumption rate is zero at that point. Measure actual delay with message timestamps rather than relying only on the Lag value.

Confirm Consumer Group State and Boundaries

Before any write or reset operation, confirming that the Consumer Group is inactive is a key risk-reduction practice. This avoids data races and follows Kafka’s recommended safety practice. If Consumers are still running, adjusting offsets can lead to duplicate consumption or missed messages, depending on auto.offset.reset and the Consumer’s internal state machine.

Steps:

1. Check Group state and member count: Query the precise Consumer Group state with --describe --state. Continue only when STATE is Empty or Dead and #MEMBERS is 0, meaning there are no active members.
2. Stop condition: If the result shows Stable or PreparingRebalance, or #MEMBERS is greater than 0, stop immediately. Stop the Consumer instances first or wait for the rebalance to finish; do not force a reset.

Note: --to-offset accepts only one Long integer in the Kafka CLI and applies it to every Partition in the selected Topic. If each Partition needs a different Offset, for example resetting Partition 0 to 1500 and Partition 1 to 450, --to-offset 1500,450 is not valid syntax. For per-Partition resets or precise recovery, first export a backup CSV with --export, then load the configuration with --from-file.

Decision Matrix: Scale, Repair, or Reset?

There are usually three main directions for handling Lag: horizontal scaling, code or configuration optimization, and Offset Reset. They are not mutually exclusive, but an incident requires a priority decision.

The key is not simply to eliminate Lag. It is to find the balance between business tolerance and engineering cost. If the business allows a short data delay and redeploying Consumers would take too long, an Offset Reset is often a fast path to restoring service availability. It must, however, be paired with rigorous validation and record-keeping. A reset is an incident-mitigation measure; the system’s underlying balance still depends on whether the Consumer root cause has been addressed.

Offset Reset is a trade-off that exchanges some data completeness for system availability.

• To-Latest: Skips unprocessed messages; the application layer needs to tolerate missing data.
• To-Earliest/Specific: Replays historical messages; the application layer needs idempotency.

Data in the Kafka Log is not deleted as a result.

Important limitation: Kafka supports increasing the number of partitions in a Topic, but does not support directly reducing the partition count of an existing Topic. In an emergency recovery scenario, creating a new Topic and migrating data is usually more predictable and carries lower rebalance risk than increasing partitions online.

Practical Steps for Safely Resetting Offsets

This section explains how to use Kafka CLI tools to safely reset and roll back offsets. We will follow the sequence: check state, preview the simulation, export a backup, execute the write, and validate afterward. Each step remains within a controlled boundary.

Step 1: Check Group State and Identify Target Partitions

Before resetting, use --describe --state to verify that the Consumer Group is inactive, confirming the STATE field and #MEMBERS count. Then use --describe to identify which Partitions have Lag and their current CURRENT-OFFSET.

Operation and validation:

Use kafka-consumer-groups.sh to query the state and details of Consumer Group my-consumer-group.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group my-consumer-group \
  --describe \
  --state

# 2. Query Lag and Offset details for the specified Consumer Group
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group my-consumer-group \
  --describe

Expected state output example (--describe --state):

GROUP              COORDINATOR (ID)         ASSIGNMENT-STRATEGY  STATE           #MEMBERS
my-consumer-group  10.0.1.15:9092 (1001)    -                    Empty           0

Stop condition and validation: If --describe --state returns Stable or #MEMBERS is greater than 0, members are still operating online, so stop the operation. Only when STATE is Empty or Dead and #MEMBERS is 0 should you use --describe to confirm the offset details for each Partition:

GROUP              TOPIC      PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
my-consumer-group  my-topic   0          1500            2500            1000            -               -               -
my-consumer-group  my-topic   1          450             3000            2550            -               -               -

Verify CURRENT-OFFSET against the intended target. If LAG is unusually large, confirm the Partition assignment and producer write rate again.

Step 2: Use a Dry Run to Simulate the Reset

Kafka CLI provides the --dry-run option, a key part of safe operations. It does not modify any Offset. Instead, it shows which Partition offsets would change and the values they would receive.

Dry-run example:

Assume we decide to reset my-consumer-group offsets to Latest, skipping all unprocessed older messages.

# Use --dry-run to simulate a reset to Latest
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group my-consumer-group \
  --reset-offsets \
  --to-latest \
  --topic my-topic \
  --dry-run

Expected output example:

The following offsets will be set.
Topic: my-topic Partition: 0 Offset current: 1500, new: 2500 (latest)
Topic: my-topic Partition: 1 Offset current: 450, new: 3000 (latest)

Validation: Confirm that the new offsets in the output match the intended target, such as the latest Log End Offset, and that the number of affected Partitions matches the observation from Step 1.

Step 3: Export the Original Offset Recovery File (--export)

Before executing the actual reset, export the current Offset for every Partition as a CSV recovery file that --from-file can read. This file is both an audit record and the rollback entry point. Because --to-offset accepts only one value, it cannot restore different values across multiple Partitions. Exporting a CSV file is therefore the only way to achieve precise per-Partition control.

For a single Consumer Group, the --export output contains three fields: topic,partition,offset. It does not include a Group ID field. The target Consumer Group is fixed and confirmed by the --group command-line parameter.

The recovery filename must include a unique identifier for this operation, and the target file must not already exist. Preserve all existing recovery records. Replace <operation-id> with a change-ticket number or timestamp. set -C rejects overwriting an existing file.

# Export the current committed offsets for all Topics/Partitions in my-consumer-group
backup_file="my-consumer-group-offsets-before-reset-<operation-id>.csv"

if [ -e "$backup_file" ]; then
  echo "Recovery file already exists; preserve the existing record and stop this operation: $backup_file"
  exit 1
fi

set -C
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group my-consumer-group \
  --reset-offsets \
  --all-topics \
  --to-current \
  --dry-run \
  --export > "$backup_file"
export_status=$?
set +C

if [ "$export_status" -ne 0 ]; then
  echo "Export failed; preserve the incomplete file for auditing, but do not use it as a rollback source: $backup_file"
  exit 1
fi

Validation: Compare every topic,partition,offset value in the export file with the CURRENT-OFFSET values returned by --describe in Step 1. They must match exactly. If any Partition is missing or an Offset differs, stop the reset. Preserve the file, confirm the Group state again, and export a new one. If the write fails or the comparison is incomplete, do not proceed to the next step.

Step 4: Execute the Actual Reset

The prerequisites for the actual reset are: the dry run in Step 2 matches expectations, the recovery file in Step 3 has been verified for every Partition, and the Consumer Group still has no active members.

Check the state once more before execution:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group my-consumer-group \
  --describe \
  --state

Stop condition: Do not execute the reset if the state is Stable, PreparingRebalance, or #MEMBERS is greater than 0. Stop the Consumer first or wait for the rebalance to complete, then restart from Step 1.

Execute the reset command that completed the dry run only when the state is Empty or Dead and there are no active members:

# Execute the actual reset to Latest
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group my-consumer-group \
  --reset-offsets \
  --to-latest \
  --topic my-topic \
  --execute

Before starting the Consumer, use the --describe command from Step 1 to inspect every Partition. Confirm that CURRENT-OFFSET equals the dry-run new offset. If any Partition differs, the Group state changes, or IllegalGenerationException appears, do not start the Consumer. Use the exported-file rollback process below.

Complete Rollback Process (--from-file)

When reset results do not match expectations or the business needs to reverse the change, roll back through the previously exported CSV file. The rollback follows the same five-stage safeguards.

1. State check: First confirm that the Consumer has stopped. Use --describe --state to confirm the Group is Empty or Dead and #MEMBERS is 0. Stop the rollback if it is Stable or still rebalancing.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group my-consumer-group \
  --describe \
  --state

2. Dry-run preview: Use --from-file with --group my-consumer-group to preview the offset for every Partition that will be loaded.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group my-consumer-group \
  --reset-offsets \
  --from-file "$backup_file" \
  --dry-run

3. Per-Partition verification: Compare each new offset in the --dry-run output against the CSV data in $backup_file (topic,partition,offset). Confirm that the Topic, Partition, and Offset match exactly, and that the Group specified on the command line is the target my-consumer-group.

4. Second state check and rollback execution: After verification, check --describe --state again to confirm that the Group remains Empty or Dead, then execute the rollback write:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group my-consumer-group \
  --reset-offsets \
  --from-file "$backup_file" \
  --execute

5. Post-operation validation: After the rollback completes, immediately run --describe and verify every Partition. Each CURRENT-OFFSET must exactly match the backup CSV record. This process restores only committed offsets in Kafka. Data already processed or missed downstream during the reset still needs to be handled through application-layer idempotency or a compensation path.

Common Boundary Conditions and Recovery Paths

Even with rigorous operating steps, distributed environments can still present unexpected boundary states.

Common Scenarios and Responses

1. The Consumer Group is not Empty, with active members present: If Consumer instances are running or attempting to rebalance during the reset, Offset updates can fail or produce IllegalGenerationException.

   • Stop condition: --describe --state shows Stable, PreparingRebalance, or #MEMBERS is greater than 0.
   • Recovery path: Shut down all Consumer instances, wait for the Coordinator to confirm the Group has become Empty or Dead, then run state validation again.

2. The target Offset is incorrect or has an invalid format: For example, entering --to-offset as a list for multiple Partitions, such as 1500,450, causes a syntax parsing error, or resetting to a value beyond the Log End Offset.

   • Stop condition: The CLI returns a syntax error, Lag becomes negative after reset, or the Consumer throws OffsetOutOfRangeException.
   • Recovery path: For a single specified value, use the corrected --to-offset <Long>. For differentiated values across multiple Partitions, use the CSV file exported in Step 3 and perform recovery again through --from-file with --dry-run and --execute.

3. Data-consistency boundaries and idempotent design: Resetting to Latest skips messages, while resetting to Earliest or a specific older Offset replays messages.

   • Stop condition: Downstream systems detect business-level anomalies, such as duplicate orders or missing data.
   • Recovery path: The reset CLI cannot restore business-data state. Handle duplicate messages through application-layer idempotency and use compensation jobs to supplement skipped historical data.

Validation and the Next Safe Extension

Resetting offsets is not the end of the operation. Treat the recovery as auditable only when all four success criteria pass together.

Restarting the Consumer triggers actual data processing. Before startup, the criteria are: CURRENT-OFFSET for affected Partitions matches the approved dry-run result; an available CSV recovery file has been verified; Lag and error logs can be continuously observed; and both the Consumer stop-or-isolation path and the application-layer compensation process are executable. If any condition is not met, do not start the Consumer.

Four Success Criteria

1. Per-Partition Offset consistency: Run --describe and verify that CURRENT-OFFSET for every affected Partition exactly equals the approved dry-run target value, or the CSV recovery record during rollback. A mismatch in any Partition means the operation is not complete.
2. Lag trend meets the handling objective: After restarting the Consumer, continuously monitor the Group state, per-Partition offsets, and Lag changes for Consumer Group my-consumer-group. If the goal is to clear the backlog and restore near-real-time consumption, Lag should decline steadily or remain low. If Lag begins growing again, Consumer CPU is saturated, or business validation does not pass, stop or isolate the Consumer. Preserve the current --describe output, Consumer logs, and business metrics, and do not continue processing data. If the approved consumption boundary itself is incorrect, return the Group to Empty or Dead and enter the --from-file rollback process. If downstream business state has already changed, an Offset rollback alone cannot restore consistency; keep the Consumer isolated and proceed through application-layer idempotency or data-compensation processes.
3. Error logs do not keep accumulating: Check Consumer instance logs and confirm that exceptions such as OffsetOutOfRangeException, RebalanceInProgressException, and IllegalGenerationException are not continuing to appear. If error logs keep growing, pause the Consumer immediately.
4. Business metrics meet established criteria: Compare business metrics from upstream producers and downstream consumers, such as orders processed per second and transaction-total alignment. Confirm that there are no data gaps beyond business tolerance. If metrics are abnormal, trigger the data-compensation process.

Next Safe Extension: Build Automated Safeguards

After all four criteria pass, turn this experience into preventive mechanisms:

• Create rate-based Lag alerts: In Prometheus / Grafana, configure derivative alerts for kafka_consumergroup_lag based on rate of change. Compared with a fixed threshold, this can provide earlier warning of long-tail Lag spikes caused by Consumer blocking.
• Review the auto.offset.reset configuration: Confirm the default behavior of each microservice when there is no committed Offset or an Offset has expired, earliest versus latest, to avoid unexpected automatic jumps.
• Build an operations SOP knowledge base: Archive the complete change history, including backup CSV files, --describe --state check screenshots, --dry-run comparison records, and business-validation results, as a standard operating procedure for future incidents.

When Partition offsets, Lag trends, error logs, and business metrics all meet their criteria, the Offset Reset operation is formally closed. The next safe extension is to encode these four criteria into automated monitoring and operations SOPs, gradually reducing the blast radius of manual maintenance.

Sources

• Confluent Documentation: Kafka CLI Tools — Official guidance and command reference for Kafka CLI tools
• Apache Kafka Official Guide — Core documentation for Kafka Consumer Groups and Offset management