Kafka Topic Cleanup: Retire Event Streams After Consumers Move

Kafka topic cleanup starts when old event streams keep partitions, retention, ACLs, schemas, and dashboards after producers or consumers have moved to a newer domain event. A topic can look quiet while still serving replay, reconciliation, fraud review, analytics, or a low-frequency partner integration.

The useful outcome is a topic retirement record: producers stopped, consumers migrated, schemas archived, replay requirements satisfied, and ACLs removed in the right order. Delete the topic only after the event contract is no longer needed.

Key takeaways

• Review producers, consumer groups, schema subjects, retention policy, ACLs, and lag dashboards together.
• Prove whether the topic is needed for live consumption, replay, audit, or backfill.
• Stop producers before removing consumers so no new data lands in a retired stream.
• Archive schema and ownership decisions even when the topic disappears.
• Prevent recurrence by pairing replacement topics with old-topic retirement criteria.

Topic Evidence Table

Check	What to inspect	Cleanup signal
Producer path	Services, connector jobs, transactional outbox, deploy history	No current service writes events
Consumer groups	Lag, commits, owner, offset movement, error topics	Groups are migrated, idle, or obsolete
Schema subjects	Compatibility mode, versions, generated clients	New schema owns the active contract
Replay need	Retention, audit policy, backfill jobs, warehouse loads	Historical data has another approved source

Do not confuse no lag with no usage. A healthy consumer can have no lag because it is caught up.

Example Retirement Record

topic_retirement:
  topic: orders.v1.events
  replacement: orders.v2.events
  producers_stopped: true
  consumer_groups_remaining:
    - fraud-replay-monthly
  retention_dependency: 90_day_audit_window
  next_step: migrate replay job before deleting topic

This record prevents the most common mistake: removing the stream that nobody watches daily but one batch job needs monthly.

Safe Cleanup Order

First, announce the topic as deprecated and block new producers. Then migrate consumers and replay jobs. Next, reduce retention only if audit and recovery rules allow it. Remove ACLs for obsolete producers and consumers, retire dashboards and alerts, archive schemas, and delete the topic last.

Do not rush compacted topics that store current state, topics used for restore from changelogs, or streams feeding regulatory reports. Do not rush topics with unclear schema ownership; a topic name can change while the data contract still matters.

Prevention

Require every new event stream to name its producer owner, consumer owner, retention reason, schema subject, and replacement plan. When a v2 topic launches, create the v1 cleanup issue immediately with the exact consumer migration evidence required.

FAQ

Can a topic be removed if no consumers are active?

Maybe. Check replay, audit, backfill, and state-store restore needs before deleting it.

Should retention be lowered before deletion?

Often. Lowering retention can reduce waste while leaving time to detect missed consumers, but only if retention policy allows it.