Rebuilding the Infrastructure: One Cluttered AWS Account Into Four

Where it started

Everything lived in one account, and each piece had arrived by a different route at a different time. None of it was wrong on its own. Together it meant there was no boundary anywhere.

• Both app backends ran on Elastic Beanstalk, each dragging along a load balancer neither app needed.
• One product's marketing site ran on AWS Amplify, a second and completely different deployment system.
• Deploys ran through GitHub Actions: slow, plus a permanent admin door from GitHub into AWS.
• Both apps shared one disk, so either could corrupt the other's data.

The real cost was not money. It was that the blast radius of any mistake was the whole company, and nobody could answer "what is running right now?" from anything but memory.

The four decisions

Four choices set the shape of everything that followed. Three of them held. One did not.

How the switch happened

The whole plan rested on one idea: build the new thing completely, verify it, then move the domain name. DNS is the only switch that has to flip, and it flips back.

• Build and verify the new server while the old one keeps serving every real user.
• Copy the data across twice: once early, once at the last moment to catch anything written in between.
• Point the domain at the new address, with the refresh interval set to 60 seconds beforehand so the world follows within a minute.
• Leave the old stack running for days. Rolling back is one DNS change.
• Only then delete anything, and only after both apps were confirmed working normally.

The test that mattered most came before the deletions: both databases were compared row by row: 2,617 events, 475 users, all seven billing tables identical. Deleting production data on the assumption that a copy worked is how people lose companies.

Five things that went wrong

The honest part, and the most useful one.

Round two: moving the login system

Two days later the decision to leave the login system in the shared account got revisited, and the premise it rested on collapsed on first contact with the data. Of the accounts in that pool, 21 sign in with Apple and 5 with Google. Zero use an email and password. The blocker had never existed, and one command would have shown that at the time.

What actually made the move hard was not the users. It was the iPhone app. It ships its web code compiled inside the binary, and the login system's address is compiled in with it. Changing that needs an App Store release, and the app and the server have to switch together: an old app against a new server fails, and a new app against an old server fails too. There is no ordering that avoids a window.

So the work was staged: build the new pool alongside the old and prove it, then bundle the switch into a release that was already going out. The pool was configured to be byte-identical to the original (settings, app client, both providers) and verified by diffing the two rather than by eye.

The honest version: nothing was lost, and that was luck rather than design. The handful of people who happened to sign in first had no billing records; the ones who did had not yet opened the app. Had the order been reversed, the first sign-in would have silently detached a paying customer from their subscription. No error, no alert, nothing to notice until someone complained. The margin between a clean migration and a data-integrity incident was who opened their phone first.

The repair: rebuild the mapping table from the old system's data, deriving each person's original account ID and pairing it with the login name, which is stable across both systems. Write it to the persistent disk, then point the setting at it so it can never be wiped again. Verified afterwards by proving that every billing record which had ever been reachable from a login was reachable again.

The lesson is the one worth carrying out of this whole project: verifying that a migration works is not the same as verifying it preserved anything. Sign-in succeeded for every test user while the data behind it was being detached.

Where it landed

Four days of work, measured against what was there before.

• Accounts: from 1 holding everything, to 3 in use and 1 reserved.
• Login system: from the shared account behind a cross-account door, to inside the app's own account with no door at all.
• Load balancers: from 2 to 0.
• Deployment systems: from 3 to 1, a single make deploy.
• HTTPS certificates: from paid and tied to load balancers, to free and renewed automatically.
• GitHub's access to AWS: from a permanent admin door to none.
• App data: from one shared disk to separate disks per app, with backups on.
• Signing in: from one account and switching roles by hand, to one portal covering all four.
• Cost: roughly $55 to $65 per month lower.

Once the old stack was gone, so was a long tail nobody had looked at in a year. It came to 219 stored application versions, a 665 MB bucket of old deploy files, and a 224 MB container registry from an abandoned experiment. Then eleven empty log folders, four orphaned firewall groups, three certificates for things that no longer existed, and a DNS zone for a dead side project.

Then the tidying: a single sign-in portal covering all four accounts with an authenticator app, billing and security notices pointed at a monitored address, and settings templates added to both repositories. One had none at all, so 43 production settings were undocumented. Neither repository has ever contained a real secret.

Lessons

What we would tell anyone doing this at a similar size.

• Move the domain name last. Build in parallel, verify, then flip DNS. One switch, and it flips back.
• Compare the data before deleting the source. Row counts, not optimism.
• Keep the old thing running for days. A few extra dollars beats having no way back.
• Read AWS permission errors carefully. They name the caller's account, not the resource's. That one detail cost the most time here.
• After crossing an account boundary, find every call site. One converted path proves nothing about the others.
• Check what your tool defaulted to. The console and the command line disagree, quietly.
• Check the premise before you design around it. A whole architectural compromise, and the outage it caused, rested on a constraint that one command would have disproved.
• "It works" is not "it preserved the data." Every test sign-in succeeded while the accounts behind them were being silently detached. Verify continuity, not just success.
• Find the storage nobody configured. Two of three databases were explicitly placed on permanent disk. The third defaulted onto disposable disk and was destroyed on every deploy for months without a single symptom.
• Simple beats managed at this size. A platform saves work at scale and hides things you need at small scale.

The shape now: one organization, one account per app, one small server each, one deploy command, one shared place for data, one portal to sign in. Adding a third app means following the recipe. The account is already waiting.