← All case studies · Case study · production · 3 min read

"The Firewall Says It's Allowed": Root-Causing a Multi-Day Hybrid Network Outage

AWS Transit Gateway
VPC routing
Hybrid networking

Before

An outage between a cloud application and on-premises systems had resisted triage for days, because firewall policy reported the traffic as allowed.

What I built

Traced it to a Transit Gateway attachment in a Deleted state, restored the route table associations, propagations and return routes, and turned the diagnostic order into a repeatable drill for the team.

Production work described at the level I can share publicly. Code samples are illustrative patterns, not production code; identifiers are documentation placeholders.

The problem: an outage the obvious tool couldn't explain

Traffic between a cloud application and on-premises systems stopped.
The outage resisted triage for days.
The obvious suspect, the firewall, reported the traffic as allowed, so the investigation kept circling back to firewall rules that were, in fact, correct.

The insight: "allowed" isn't the same as "reachable"

Firewall policy answers one question: is this traffic permitted? It says nothing about whether a working path exists. Policy and routing are separate layers, and a broken route looks exactly like a blocked one from the application's side.

So I stopped debugging the policy and went down to the routing layer, checking it in order:

1. Transit Gateway attachment state
2. TGW route table associations and propagations
3. VPC subnet routes toward the Transit Gateway
4. On-premises return routes

What I did

Attachment state. The Transit Gateway attachment was in a Deleted state. That was the root cause: correct firewall rules can't help traffic with no attachment to cross.
Associations and propagations. Confirmed which TGW route tables the attachment needed to be associated with and propagating into.
Subnet routes. Checked that the VPC subnets still routed the on-premises ranges to the Transit Gateway.
Return routes. Without on-premises routes back to the cloud ranges, requests leave but replies never return.

I restored the route table associations, propagations and return routes, and traffic recovered.

Illustrative: the same check order as AWS CLI commands.

# 1. Attachment state: is the path even there?
aws ec2 describe-transit-gateway-attachments \
  --filters Name=transit-gateway-id,Values=tgw-0123456789abcdef0 \
  --query 'TransitGatewayAttachments[].[TransitGatewayAttachmentId,ResourceType,State]' \
  --output table

# 2. Associations and propagations on the TGW route table
aws ec2 get-transit-gateway-route-table-associations \
  --transit-gateway-route-table-id tgw-rtb-0123456789abcdef0
aws ec2 get-transit-gateway-route-table-propagations \
  --transit-gateway-route-table-id tgw-rtb-0123456789abcdef0

# Routes the TGW actually has
aws ec2 search-transit-gateway-routes \
  --transit-gateway-route-table-id tgw-rtb-0123456789abcdef0 \
  --filters Name=state,Values=active

# 3. VPC subnet routes that point at the TGW
aws ec2 describe-route-tables \
  --filters Name=vpc-id,Values=vpc-0123456789abcdef0 \
  --query 'RouteTables[].Routes[?TransitGatewayId]'

# 4. Return routes: check on the on-premises side (router / firewall routing table).

What changed afterwards

I turned the diagnostic order into a repeatable drill for the team: attachment state first, then associations and propagations, then subnet routes, then return routes. The next hybrid connectivity incident starts from the right layer instead of from the firewall.

Things to watch

Start from state, not configuration. A resource in the wrong state makes correct configuration irrelevant. Check control-plane state before reading rules.
Associations and propagations are different things. An association decides which route table an attachment uses; a propagation decides which route tables learn its routes. Both have to be right, in both directions.
Routing is two-way. A path that works outbound but has no return route fails just as completely, and is easy to miss because the outbound side looks healthy.
Use the path tools. VPC Reachability Analyzer tests whether a path exists between two endpoints in AWS and names the hop that blocks it. That's the quickest way to answer "allowed but unreachable" questions.

What it solves

The outage was resolved after days of triage had stalled.
The team has a repeatable diagnostic order, so the next incident starts at the right layer.
A lesson that generalizes: separate "is it permitted?" from "is there a path?" in every network investigation.

What it doesn't solve

Why the attachment ended up deleted. Restoring connectivity and preventing a repeat are different jobs. Change history (CloudTrail) and infrastructure-as-code drift detection are how you stop it happening again.
Detection. A drill shortens diagnosis; it doesn't alert you. Monitoring attachment state and route-table changes catches the problem before users report it.

Checklist

[ ] Attachment state checked before any rule is examined
[ ] Associations and propagations verified in both directions
[ ] VPC subnet routes point at the Transit Gateway for the right ranges
[ ] On-premises return routes confirmed
[ ] Reachability Analyzer run between the two endpoints
[ ] Diagnostic order written down as a team drill
[ ] Alerting on attachment state and route-table changes