When a Bare-Metal Migration Hit a BGP Wall: A Post-Mortem from the Hosting Desk

We first heard about this project from a reader who runs infrastructure for a mid-sized SaaS company — roughly 40 engineers, a few hundred thousand daily active users, and a growing frustration with their cloud bill. Their workloads were steady, predictable, and latency-sensitive. After two years of autoscaling dashboards and per-egress charges, they decided to move their core API and database tier onto bare metal. That decision set off a six-month migration that taught them more about BGP routing than they ever wanted to know.

What makes this case worth writing up is not the destination. Plenty of teams are revisiting dedicated server economics right now. What makes it interesting is the middle: the week when a routing misconfiguration took their primary region offline for 19 minutes, and the changes they made afterward to make sure it never happened again. They documented the whole thing for us, and we followed the timeline closely because the lessons apply to almost any team leaving managed cloud behind.

The Decision Point: Cloud vs. Bare Metal

The team's original stack cost about $48,000 a month across two cloud regions. Their break-even analysis showed that a comparable footprint on dedicated hardware — dual-socket nodes with NVMe storage and 10 Gbps uplinks — would run closer to $14,000 a month, including rack space, power, and a support contract. The gap was large enough that the CFO asked for a formal proposal.

They shortlisted three providers. One was a large incumbent with a polished portal but opaque network topology. One was a budget host with aggressive pricing and thin documentation. The third was MG Cluster, which they chose after a two-week benchmark period. The deciding factor wasn't price — it was that the team could see the actual network: upstream carriers, peering points, and the BGP configuration they'd inherit. For a company whose product lived or died on request latency, that transparency mattered more than a logo.

Timeline: Six Months, Three Phases

Phase 1 — Benchmarking (Weeks 1–3)

They ran a structured server benchmark across all three candidates: disk IOPS under mixed read/write, network throughput at 1 Gbps and 10 Gbps, and packet loss under synthetic load. The incumbent won on raw CPU but lost on network consistency. The budget host posted impressive peak numbers with worrying variance. MG Cluster's nodes were not the fastest on paper, but their p99 latency stayed within a 2 ms band throughout the test window. That predictability became the theme of the whole project.

Phase 2 — Cutover (Weeks 4–18)

They migrated the database tier first, then the API tier, then the background job workers. Each cutover used DNS with short TTLs and a rollback path. Nothing dramatic happened until week 14, when they moved the primary API cluster and announced their own IP space to the new datacenter.

That's where the BGP routing bit them. Their prefix was announced from two routers, but the route-map on one of them had a stale community tag left over from a previous provider. The result was asymmetric routing: inbound traffic arrived on the primary path while outbound replies left on the secondary. Most connections survived. Long-lived WebSocket sessions did not. After 19 minutes of intermittent failures, an engineer spotted the mismatch in the looking-glass output and withdrew the bad announcement.

Phase 3 — Hardening (Weeks 19–26)

The post-incident review produced four changes that are worth stealing:

• Every BGP announcement now goes through a pre-flight checklist with a second pair of eyes.
• They added a synthetic monitor that opens a persistent connection from three external regions every 30 seconds.
• Route advertisements are version-controlled, with a diff reviewed before any change.
• They documented a 5-minute rollback procedure and rehearsed it twice.

Measurable Results After Six Months

The numbers, as reported by the team: monthly infrastructure spend dropped from $48,000 to $16,200 — a 66% reduction. Median API latency fell from 84 ms to 51 ms. The p99, which had been their real pain point, went from 410 ms to 118 ms. Uptime for the six months after the hardening phase was 99.98%, with the only notable event being a scheduled maintenance window.

The team was careful to note that the savings were not automatic. They spent roughly 120 engineer-hours on the migration itself and another 40 on the hardening work. At their blended engineering rate, that's about $24,000 of internal cost — recovered in under two months.

What We'd Tell the Next Team

Three takeaways stand out. First, benchmark for consistency, not peaks; a node that never surprises you beats one that occasionally dazzles. Second, treat BGP routing as production code — review it, version it, and rehearse your rollback. Third, high availability is a practice, not a purchase. The redundant routers were already there; the discipline around them was not.

The team's infrastructure lead put it plainly: "MG Cluster gave us the hardware and the network visibility, but the 19-minute outage was ours to own." That honesty is why we asked to publish the story. Migrations rarely fail because of the vendor. They fail in the gap between what the vendor provides and what the team assumes.

If you're planning a similar move, start with the routing plan before you spec the servers. The hardware is the easy part.