where your packets actually go inside AWS

Most people's first VPC comes from a wizard or a Terraform module. You get "public" and "private" subnets, a few gateways, and it works. Then one day a Lambda can't reach an API, or the NAT gateway line on the bill is bigger than the EC2 line, and it turns out nobody on the team knows where packets actually go.

The good news is that a VPC has very few moving parts. A packet's path is decided by a route table, and whether it's allowed through is decided by two kinds of firewall. Once you can trace those, most VPC problems become easy to solve.

Along the way I'll point at real templates from cloud-formation, the CloudFormation repo that runs the AWS side of my own projects. Its shared/networking/vpc.yaml builds the network in the diagram above, with two subnets of each kind instead of one.

The VPC and its subnets

A VPC is a private network with an address range you choose, like 10.0.0.0/16 (65,536 addresses). It lives in one region and spans every Availability Zone in it.

A subnet is a slice of that range, like 10.0.1.0/24, and it lives in exactly one AZ. That's why a highly available setup always has at least one subnet per AZ: if the AZ goes down, everything in its subnets goes with it.

In CloudFormation you can place each subnet in an AZ by index instead of by name. This is how the repo does it (vpc.yaml):

PrivateSubnet01:
  Type: AWS::EC2::Subnet
  Properties:
    VpcId: !Ref VPC
    CidrBlock: !Ref PrivateSubnet01CIDR
    AvailabilityZone: !Select [0, !GetAZs ""]

!GetAZs lists the region's AZs and !Select picks one. PrivateSubnet02 uses index 1, so the pair always ends up in two different AZs, and the template works in any region without hard-coding names like ap-south-1a.

AWS keeps five addresses in every subnet for itself. In 10.0.1.0/24:

So a /24 gives you 251 usable addresses, not 256. That matters more than it sounds. EKS gives every pod its own VPC address by default, and a /24 subnet fills up quickly.

Route tables

Here's the part the wizard hides: "public" and "private" aren't settings. A subnet is public only because of what's in its route table.

Every subnet is associated with one route table, and every route table starts with a local route that you can't remove:

When a packet leaves an instance, the VPC router picks the most specific route that matches its destination (longest prefix wins). A packet to 10.0.2.15 matches both rows, but /16 is more specific than /0, so it stays inside the VPC. A packet to 142.250.72.14 only matches 0.0.0.0/0, so it goes to the internet gateway.

That second row is the whole difference. A subnet whose route table sends 0.0.0.0/0 to an internet gateway is public. One that doesn't is private.

In the repo, that difference comes down to two routes with different targets (vpc.yaml):

PublicDefaultRoute:
  Type: AWS::EC2::Route
  Properties:
    RouteTableId: !Ref PublicRouteTable
    DestinationCidrBlock: 0.0.0.0/0
    GatewayId: !Ref InternetGateway
 
PrivateDefaultRoute:
  Type: AWS::EC2::Route
  Properties:
    RouteTableId: !Ref PrivateRouteTable
    DestinationCidrBlock: 0.0.0.0/0
    NatGatewayId: !Ref NatGateway

Same destination, different target. The subnets don't say whether they're public or private. That comes only from which route table each one is associated with.

Out to the internet

An internet gateway (IGW) is attached to the VPC and connects it to the internet. It scales on its own and costs nothing.

One thing surprises everyone the first time: run ip addr on an instance with a public IP and the public IP isn't there. The instance only knows its private address. The IGW translates between the two, one to one, as packets cross it. On the way out, 10.0.1.10 becomes 54.210.3.7. On the way back, the reverse.

So a public subnet needs two things for an instance to reach the internet: a route to the IGW, and a public IP (or Elastic IP) on the instance. That's what MapPublicIpOnLaunch: true on the repo's public subnets is for. With a route and no public IP, the IGW has nothing to translate to and the traffic goes nowhere.

From a private subnet

Instances in private subnets often still need to reach out: to pull images, call third-party APIs or install packages. That's what a NAT gateway is for. It sits in a public subnet, and the private subnet's route table points at it:

Now 10.0.2.15 sends its internet traffic to the NAT gateway. The NAT gateway rewrites the source to its own address and sends the packet out through the IGW, like any public instance. Replies come back to the NAT gateway, which remembers the connection and forwards them to the app. Nothing on the internet can start a connection to the app, because there's no mapping until the app makes one.

A NAT gateway lives in one AZ. If every private subnet points at a single NAT gateway, then losing that AZ cuts off all outbound traffic, and every other AZ pays cross-AZ data charges to reach it. One NAT gateway per AZ, with each AZ's private subnets routed to their own, avoids both.

The repo deliberately runs just one: a single NatGateway in PublicSubnet01, shared by both private subnets (vpc.yaml). For personal projects that's a reasonable trade, because every NAT gateway is billed by the hour whether it carries traffic or not. Moving to one per AZ takes a second Elastic IP and NAT gateway in PublicSubnet02, plus a second private route table that points at it.

The NAT gateway bill

A NAT gateway charges per hour and per gigabyte it processes (about $0.045/GB in us-east-1). That per-GB charge applies to everything that goes through it, including traffic to AWS's own services.

So the classic surprise: an app in a private subnet reads a lot from S3. Every byte goes app → NAT gateway → IGW → S3, and every byte is charged NAT processing. A job that reads 50 TB a month adds over $2,000 to the bill, just for the NAT gateway.

The fix is a gateway endpoint, which exists for S3 and DynamoDB only. It costs nothing. Creating one adds a route to the route tables you pick:

pl-63a5400a is a prefix list, AWS's list of S3's IP ranges in the region. Those ranges are more specific than 0.0.0.0/0, so longest-prefix match sends S3 traffic to the endpoint and everything else to the NAT gateway. No code changes, no DNS changes. The same SDK calls just take a different, free path.

This applies directly to the repo. The video transcoding services run on Fargate in the private subnets with AssignPublicIp: DISABLED (ECS.yaml), and they read uploads from S3 and write transcoded video back to it, so all of that goes through the NAT gateway. Video is exactly the kind of traffic where that adds up. The fix is one more resource in vpc.yaml:

S3GatewayEndpoint:
  Type: AWS::EC2::VPCEndpoint
  Properties:
    VpcId: !Ref VPC
    ServiceName: !Sub "com.amazonaws.${AWS::Region}.s3"
    VpcEndpointType: Gateway
    RouteTableIds:
      - !Ref PrivateRouteTable

Both private subnets share PrivateRouteTable, so they both get the S3 route at once.

For other services (ECR, SQS, Secrets Manager, STS and most of the rest) there are interface endpoints. They work differently: a network interface with a private IP is placed in your subnet, and the service's DNS name resolves to it. They're billed per hour per AZ and per GB, but at a fraction of the NAT rate, so they pay off for busy services like ECR image pulls.

Security groups and NACLs

Routes decide where a packet can go. Two kinds of firewall decide whether it's allowed:

The last row is the one that bites people. A security group remembers connections. If it allowed a request in on port 443, the reply is allowed out automatically, no outbound rule needed.

A NACL has no memory. It checks every packet on its own, in both directions. The reply to a request on port 443 doesn't go back to port 443. It goes to whatever random ephemeral port the client picked, somewhere between 1024 and 65535. If the NACL's outbound rules don't allow that range, the request gets in and the reply is silently dropped.

This is why most teams leave NACLs at their default (allow everything) and do all their filtering with security groups. Security groups have another advantage too: a rule can allow traffic from another security group instead of from an IP range. "The database allows port 5432 from the app's security group" keeps working as instances come and go, with no IP addresses to update.

The repo's security-groups.yaml is a chain of these. Only the load balancer accepts traffic from the internet. The app accepts ports 3000 and 3001 only from the load balancer's group, RDS Proxy accepts 5432 only from the app's, and the database accepts 5432 only from the proxy's:

DatabaseSecurityGroup:
  Type: AWS::EC2::SecurityGroup
  Properties:
    GroupDescription: Allow PostgreSQL traffic from RDS Proxy
    VpcId: !Ref VpcId
    SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: 5432
        ToPort: 5432
        SourceSecurityGroupId: !Ref ProxySecurityGroup

There isn't an IP address in the whole file, and nothing can skip a step. Even something running inside the VPC can't reach the database unless it's in the proxy's group.

Seeing it happen

When something can't connect, guessing is slow. VPC Flow Logs record a line for every flow through a network interface, subnet or the whole VPC, and you can send them to CloudWatch Logs or S3. A line in the default format looks like this:

2 123456789012 eni-0a1b2c3d 203.0.113.12 10.0.1.10 51514 443 6 10 840 1730000000 1730000060 ACCEPT OK
2 123456789012 eni-0a1b2c3d 10.0.1.10 203.0.113.12 443 51514 6 8 6120 1730000000 1730000060 REJECT OK

Reading the fields in order: version, account, interface, source address, destination address, source port, destination port, protocol (6 is TCP), packets, bytes, start and end time, the action, and the log status.

The pair above is the NACL bug from the last section, caught in two lines. The request to port 443 was accepted. The reply to port 51514 was rejected. A security group would never do that (it allows replies automatically), so this points straight at a NACL with no outbound rule for ephemeral ports.

If you can't find any line at all, the packet never reached the interface. That usually means a route problem, not a firewall problem.

Connecting VPCs

Sooner or later there's more than one VPC (per environment, per team or per account) and they need to talk.

• VPC peering connects two VPCs directly. It's simple and has no hourly charge, but it isn't transitive: if A peers with B and B peers with C, A still can't reach C through B. With many VPCs, the number of peerings grows quickly. And the address ranges can't overlap, which is why it pays to give every VPC its own range from day one.
• Transit Gateway is a hub. Each VPC attaches once, and the Transit Gateway's own route tables decide who can reach whom. It's the usual choice past a handful of VPCs, and it charges per attachment and per GB.
• PrivateLink exposes one service, not a whole network. The consumer gets an interface endpoint for your service, and the two VPCs never share routes, so overlapping ranges don't matter.

The same rule applies to all three: creating the connection isn't enough. Both sides' route tables need routes that point at it, and both sides' security groups have to allow the traffic. Most "the peering is up but nothing connects" problems are a missing route on one side.

Further reading

• cloud-formation, my CloudFormation repo with the VPC, security groups and services referenced in this post
• How Amazon VPC works, the official overview of subnets, routing and gateways
• Configure route tables, including how the most specific route is chosen
• NAT gateways, with the per-AZ setup and pricing notes
• Gateway endpoints, the free S3 and DynamoDB path
• Control traffic to resources using security groups and network ACLs, side by side
• Flow log records, every field in the format above