Databases
Data Lineage Cleanup: Remove Edges After Pipelines Change
Data Lineage Cleanup: Remove Edges After Pipelines Change starts with a lineage graph edge that points to an old transformation path. The lineage edge may look quiet, but quiet is not the same as unused. It can still support a rare workflow, a contract, a rollback path, or an owner who no longer sits near the team doing the cleanup.
Use this note when you need to reduce stale software surface area without turning deletion into the first real test. The useful result is a small decision record: current owner, current purpose, evidence reviewed, reversible first step, caveats, and the rule that prevents the same lineage edge from returning.
What makes this cleanup risky
The risk is not age. The risk is losing a dependency that is visible only during unusual conditions. In data lineage cleanup, the review should start by naming the exact behavior the lineage edge still enables and the exact behavior that has replaced it.
| Review area | What to inspect | Cleanup signal |
|---|---|---|
| Current owner | Team, service, data owner, or support path | Someone can approve a keep or remove decision |
| Runtime evidence | transformation code, scheduler runs, query history, dashboard dependencies, notebook execution, and catalog ownership | Recent use is absent or explained |
| Replacement path | current pipeline dependency | The new path handles the same real cases |
| Rollback or history | Backup, audit, archive, or recreation plan | A wrong decision is recoverable |
| Creation path | How new items are created | A prevention rule can stop recurrence |
A cleanup candidate with no owner should not be treated as safe. It should be treated as an ownership bug that must be resolved before the final removal.
Evidence checks that fit the subject
Collect several signals before acting:
- Inspect transformation code, scheduler runs, query history, dashboard dependencies, notebook execution, and catalog ownership.
- Confirm the replacement path, not just the absence of recent edits.
- Review the longest business, reporting, incident, or customer cycle that could still use the lineage edge.
- Ask the owner to choose keep, narrow, archive, disable, remove, or investigate.
A focused review sample can keep the conversation concrete:
SELECT user_name, COUNT(*) FROM warehouse_query_history WHERE query_text ILIKE '%legacy_customer_rollup%' GROUP BY user_name;Treat the output as a candidate list, not a deletion command. It proves one slice of behavior and must be paired with ownership, dependency review, and a rollback plan.
Prefer a reversible first move
Good cleanup usually happens in stages. First stop creating new lineage edge records or references. Then narrow the scope, disable the stale path, or archive the visible surface while watching for unexpected use. Remove only after the waiting window matches how the system is actually used.
Do not rush when the lineage edge touches security response, customer commitments, billing, compliance, incident recovery, or low-frequency operational work. Also slow down when the replacement changed semantics rather than only names; similar labels can hide different behavior.
Prevention rule
pipeline pull requests should update lineage metadata with source, target, owner, freshness, and sunset condition. Add the rule where the lineage edge is created, not only in a cleanup spreadsheet. The next cleanup should begin with owner and sunset context already attached.
Key takeaways
- Stale lineage edge cleanup needs evidence about use, ownership, replacement, and reversibility.
- Recent silence is helpful, but it is not enough by itself.
- The best first move is usually narrowing, disabling, or archiving before final removal.
- Prevention belongs in the creation path so the same stale item does not return.