Migrations & Rewrites

Most high-impact technical work is not greenfield. It is changing something that already exists, is in production, and has users on it right now. New code on a blank repository is the easy case and the rare one. The defining senior+ skill is changing a running system without breaking it — replacing the engine while the plane is in the air.

This is harder than building from scratch and less celebrated. Nobody demos a migration. But a senior who can only build new things, and freezes the moment the system has live traffic and a decade of accreted behavior, is capped at toy problems. The hard, valuable, career-defining work is here.

Why This Is The Job

• The system is load-bearing. Real users depend on the current behavior, including the bugs they've built workarounds around. You don't get to break it for a week while you finish.
• The constraints are mostly invisible. The old system encodes years of decisions, edge cases, and undocumented contracts. Half the migration is discovering what the old thing actually does, not what it was supposed to do.
• The blast radius is large. A migration touches data, dependent teams, on-call, and often customers. The cost of getting it wrong is measured in incidents and lost data, not a failed unit test.
• It rarely finishes by accident. New features have a natural endpoint — they ship. Migrations don't; the default outcome is running two systems forever. Getting to done is the actual challenge.

Closely tied to technical strategy — a migration is strategy executed over months, and it lives or dies on the same things: clear sequencing, honest risk, and the discipline to finish.

Rewrite vs Incremental

The first decision, and the one most often gotten wrong: do you rewrite, or do you change the existing system in place?

The strong default is incremental. Rewrite from scratch only when you've ruled incremental out, not the other way around.

Why rewrites fail

Joel Spolsky's "Things You Should Never Do" called the full rewrite "the single worst strategic mistake that any software company can make." The argument holds up:

• You throw away knowledge, not just code. Every weird branch in the old code is a bug someone hit in production and fixed. A clean rewrite re-introduces every one of those bugs, then re-discovers them one outage at a time.
• The second-system effect. Fred Brooks's observation: the replacement, freed of constraints, accumulates every feature and abstraction the team wished they'd had. It's over-engineered, late, and collapses under its own ambition.
• You ship nothing for a year. During the rewrite, the old system gets minimal investment and the new one isn't ready. Competitors and the business don't pause. The rewrite that takes "six months" routinely takes two years, and a meaningful fraction never ship at all.
• You chase a moving target. The old system keeps shipping features during the rewrite. The new one has to catch up to a finish line that keeps moving.

When a rewrite is justified

It does happen. Genuine cases:

• The platform is dead — the language, framework, or runtime is EOL and unsupported, with no incremental path off it.
• The architecture cannot do what the business now requires (e.g. a single-tenant design that must become multi-tenant), and bolting it on costs more than rebuilding.
• The system is small enough that a rewrite is bounded and cheap relative to the pain it removes.
• You're rewriting behind a stable interface — which is not really a "from scratch" rewrite at all; it's an incremental replacement (see below) and is the safe way to do most "rewrites."

De-risking a rewrite

If you must rewrite, make it behave like an incremental migration:

• Keep the old system running and invested in until the new one fully replaces it.
• Replace behind a stable contract, component by component, never the whole thing at once.
• Run both in parallel and diff outputs before you trust the new one.
• Cut over gradually behind flags, never big-bang.

A "rewrite" done this way is just a Strangler Fig with a fresh codebase. A rewrite done the other way — build for a year, flip the switch — is the one Spolsky warned about.

See technical decisions for framing this choice, and software architecture for the structural side.

The Strangler Fig

Named by Martin Fowler after the vine that grows around a tree, gradually replacing it until the original is gone and the new structure stands on its own. It is the backbone of safe migration: build the new system around the old one, route traffic over piece by piece, and remove the old when nothing depends on it.

The whole point is that at every step the system is fully working, fully shippable, and fully reversible. You never have a moment where the migration must succeed or everything breaks.

Expand / contract (parallel change)

The fundamental move for changing any interface — an API, a schema, a function signature — without a flag day:

Each phase ships independently and is safe on its own. The migrate phase can take weeks and the system is correct the whole time.

Dual-write

When migrating a data store or a service of record, write to both old and new for a transition period. Reads still come from the old (the source of truth) while you build confidence in the new. Dual-write plus a backfill gets the new store to parity with the old, after which you can flip reads over.

Watch the consistency trap: dual-write is not atomic. One write can succeed and the other fail. You need reconciliation — a job that detects and repairs divergence — or you'll slowly drift.

Shadow traffic

Send real production traffic to the new system in parallel, with its results discarded. The new path gets exercised under real load and real data shapes, but users never see its output. You find the crashes, the latency cliffs, and the wrong answers before anyone depends on them.

Parallel-run with diffing

The strongest verification there is: run old and new on the same input, compare the outputs, and log every mismatch. GitHub's scientist library is the canonical tool. You get a quantified answer to "is the new thing actually correct?" instead of a hopeful one. Don't cut over until the diff rate is near zero and you understand every remaining mismatch.

Gradual cutover behind flags

The final move: shift real reads/traffic from old to new incrementally — 1%, 10%, 50%, 100% — gated by a feature flag. Watch metrics at each step. If something's wrong, flip the flag back; you're instantly on the old path. This converts a terrifying cutover into a dial you can turn both directions.

Data Migrations

Data is the part that can't be rolled back. Code you can revert; a botched destructive migration that drops or corrupts rows is gone. Treat data with more fear than code.

• Backfills. Populating a new column, table, or store from existing data. Always run as a batched, throttled, resumable job — not one giant transaction that locks the table and dies at 80%.
• Online schema change. Never ALTER TABLE a large hot table directly. Use a tool (gh-ost, pt-online-schema-change, or the platform's online DDL) that builds the change on a shadow copy and swaps it in without a long lock.
• Idempotent and resumable jobs. A migration job will be interrupted — OOM, deploy, timeout. Design it so re-running is safe and it picks up where it left off (checkpoint by primary-key range). A job you're afraid to re-run is a job that will strand half-migrated data.
• Verify, don't assume. After backfilling, run a job that confirms old and new agree — counts, checksums, row-by-row sampling. "The job finished without errors" is not the same as "the data is correct."
• Irreversibility is the whole risk. Keep the old data until the new is verified and load-tested. Make destructive steps (dropping the old column) a separate, later deploy, long after reads have moved and you're sure. Never delete and migrate in the same change.

Pair every data migration with the refactoring discipline of small, verifiable steps — the same instinct applies, with higher stakes.

Zero-Downtime Principles

A few rules that, followed together, keep the system up throughout:

• Always have a rollback. Every step must be reversible cheaply and quickly. If a step can't be undone (most data deletions), it gets isolated, delayed, and triple-checked. "We'll roll forward" is not a rollback plan; it's a hope.
• Migrate in small, reversible steps. Many tiny deploys, each safe on its own, beat one big coordinated cutover. Small steps localize failure — when something breaks, you know which of the last three changes did it.
• Backward and forward compatibility at every step. During a migration, old and new code, old and new schema, run simultaneously. Each version must tolerate the other's data. This is why expand/contract exists: you never have a moment where old code meets new-only data.
• Decouple deploy from release. Ship the code dark behind a flag, then turn it on separately. Deploying and activating in one motion removes your ability to react.

The Organizational Problem

The technical pattern is the easy half. The hard half is getting N other teams to actually move off the old thing — and that's a people problem, not an engineering one.

The new system can be done, better, and faster, and migration still stalls because every other team has their own roadmap and "migrate off the thing that currently works" is nobody's priority but yours.

What actually moves teams:

• Make the new path the easy path. The single strongest lever. If the new way is less work than the old — better docs, codemods, a migration script, a paved golden path — teams migrate because it's easier, not because you asked. If the new way is more painful, no amount of asking works.
• Carrot and stick, in that order. Carrot: do the migration for people where you can, provide tooling, make the new thing genuinely better. Stick: only after the carrots — a deprecation policy, a hard sunset date with leadership backing, the old path eventually erroring.
• A real deprecation policy. Announce it, give a timeline, send reminders, escalate the loudness as the date approaches. "Deprecated" with no enforcement and no date is ignored, correctly.
• A sunset date with teeth. A date nobody enforces is a suggestion. The date must be backed by someone who can hold teams to it, and there must be a consequence for missing it (the old path breaks, or there's a leadership conversation).
• Track the long tail and go get it. The last 20% of migrations are 80% of the work — the orphaned services, the team that left, the integration nobody owns. Someone has to chase each one down by name. It does not finish itself.

Getting other teams to act without authority over them is its own skill — see scaling yourself and the influence work it links to.

Measuring Progress And Declaring Done

• Measure percent migrated, not percent of time spent. "We've worked on it for three months" tells you nothing. "82% of traffic / 41 of 60 services / 95% of rows are on the new system" tells you where you are and how far is left. Instrument the migration itself.
• Define done before you start. Done is: 100% on the new path, old path receiving zero traffic, old code and old data deleted. Not "the new thing works and most teams use it."
• Delete the old system. This is the step everyone skips, and skipping it forfeits the entire payoff. The reason to migrate was to stop paying for the old thing — the maintenance, the security surface, the cognitive load, the confusion of two ways to do everything. If the old code is still in the repo "just in case," you got the cost of two systems and the benefit of one. Set a date to delete it, and delete it.
• The migration that never finishes is the real failure. Two half-finished systems running in parallel forever is the worst outcome — more code, more confusion, more on-call, and you never collected the win. A migration that's "90% done" for a year is a failed migration, not an ongoing one.

Risk, Comms, And The Design Doc

Any non-trivial migration gets a design doc before it gets code — see RFCs and design docs. It should cover:

• The plan in reversible steps. The Strangler sequence, what ships in what order, and the rollback for each.
• The cutover and rollback mechanism. Exactly how you shift traffic and exactly how you reverse it, tested before you need it.
• Data safety. Backfill strategy, verification, and when (much later) the old data gets deleted.
• Risk and blast radius. What can go wrong at each step, the worst case, who's affected.
• The org plan. Who has to migrate, the deprecation timeline, the sunset date, who owns chasing the long tail.
• Comms. Who you tell, when, and how — affected teams, on-call, support, customers if user-visible. Migrations break trust silently; over-communicate.

Anti-Patterns

• Big-bang cutover. Flip everyone to the new system at once on a chosen night. When (not if) something's wrong, everything is wrong simultaneously and rollback is a panic. Always go incremental.
• The rewrite that never ships. A year of greenfield, no live traffic, a moving target, and a launch that keeps slipping. The classic Spolsky failure.
• No rollback. A forward-only cutover where the only recovery from a bad deploy is fix-and-roll-forward under incident pressure. Every step needs a reverse gear.
• Migrating without metrics. Cutting over on hope instead of a near-zero parallel-run diff and live dashboards. You won't know it's broken until users tell you.
• Leaving the old system alive indefinitely. Never deleting the old path. You permanently run two systems, pay double the maintenance, and never collect the benefit you migrated for.
• Dual-write without reconciliation. Writing to both stores and assuming they stay consistent. They won't; without a repair job they silently diverge.

Pre-Commit Question

Before you start a migration, ask:

What's the smallest reversible step, how will I know it worked, and what's the date I delete the old thing?

If you can't answer the first, you have a big-bang. If you can't answer the second, you're flying blind. If you can't answer the third, you're signing up to run two systems forever.