AWS VPC:
how one request crosses a private network

A VPC’s IPv4 range is one CIDR block, and AWS accepts any size from /16, which is 65,536 addresses, down to /28, which is 16. shop-prod takes the largest: 10.0.0.0/16, every address from 10.0.0.0 to 10.0.255.255.

That first block is the primary CIDR, and it can never be resized. When it runs out you add a secondary block, up to five per VPC by default and 50 at most, each with subnets of its own. The shop’s eight subnets, six /24s and two /28s, use 1,568 of the 65,536 addresses: 2.4% of the range.

AWS reserves five addresses in every subnet: the first four and the last. In 10.0.2.0/28 they are .0 for the network address, .1 for the VPC router, .2 for DNS, .3 for future use, and .15, the broadcast address, which AWS keeps even though VPCs do not support broadcast. The DNS resolver every subnet uses answers at 10.0.0.2, the VPC range’s base plus two.

A /24 keeps 251 of its 256 addresses, a loss under 2%. The NAT subnet is a /28, the smallest size allowed, and it keeps 11 of 16: 31% gone before anything launches. The NAT gateway takes one of the 11, 10.0.2.9.

Every subnet lives in exactly one Availability Zone, a data center or group of data centers with its own power and networking. A subnet cannot span two, so “Multi-AZ” always means at least two subnets. shop-prod repeats each of its four tiers in two zones, eight subnets in all.

The names are not fixed. AWS can map us-east-1a to a different physical zone in each account, so your us-east-1a and the fraud team’s us-east-1a may be two facilities. The AZ ID, such as use1-az1, names the same zone in every account, and it is the value two teams agree on before placing subnets side by side.

Every Region of every AWS account comes with a default VPC already built: 172.31.0.0/16, one /20 subnet of 4,096 addresses in each Availability Zone, and a route from every subnet to an internet gateway. Its subnets have MapPublicIpOnLaunch turned on, so each instance launched there gets a public IPv4 address unless you ask for none.

It exists because accounts created after 2013-12-04 have only VPC networking; the older EC2-Classic network shut down its last instance on 2023-08-15. shop-prod is built separately. None of its subnets hands out public addresses, and only the public tier has a route to the internet gateway.

The fraud-scoring team in account 444455556666 also built its VPC on 10.0.0.0/16. Two VPCs with overlapping ranges cannot be peered and cannot reach each other through a Transit Gateway, because a destination such as 10.0.10.27 would name two machines, and a route table holds one target per destination.

The only connector that works across an overlap is PrivateLink, which publishes one service through an interface in your subnet instead of joining two ranges; chapter 05 uses it. The lasting fix comes earlier: plan addresses across every account before the first VPC exists, which is what AWS IPAM pools are for.

A subnet is public for one reason: its route table has a row sending 0.0.0.0/0 to an internet gateway attached to the VPC. The subnet itself has no public setting. public-rt has that row. app-rt sends 0.0.0.0/0 to a NAT gateway instead, which chapter 03 covers, so nothing on the internet can open a connection to the app tier.

Every subnet is associated with exactly one route table. A subnet with no explicit association uses the VPC’s main route table, which is why the main table should stay private. One table can serve many subnets: public-rt serves both public subnets, because both send internet traffic to the same gateway.

The ALB node’s network interface has one IPv4 address, 10.0.0.37, and nothing behind it ever sees another. The public address 198.51.100.20 exists only in the internet gateway, as a one-to-one mapping to 10.0.0.37. On the way in, the gateway rewrites the destination to 10.0.0.37; on the way out, it rewrites the source back to 198.51.100.20.

That mapping is check 1 of the request. Without it the SYN never reaches the subnet, and no log anywhere records the drop. Since 2024-02-01 every public IPv4 address costs $0.005 an hour, $3.65 a month, whether it is attached to something running or allocated and idle.

The MTU is the largest packet a link carries in one piece. Inside a VPC, current instance types use jumbo frames with an MTU of 9,001 bytes, and any path through an internet gateway drops to 1,500. The ALB and the app server exchange 9,001-byte packets; the customer’s packets stop at 1,500, about one sixth the size.

Senders find the smaller limit through path MTU discovery: a hop that cannot forward a large packet returns an ICMP “fragmentation needed” message, and the sender shrinks its packets. A network ACL that drops all ICMP also drops those messages. Small requests still succeed, and large transfers stall, which is a hard failure to diagnose from either end.

A packet leaving a subnet is tested against every row of that subnet’s route table, and the most specific match wins. This is longest-prefix match: the row whose prefix fixes the most bits of the destination. For Postgres at 10.0.20.14, app-rt holds 10.0.0.0/16 → local and 0.0.0.0/0 → nat-0a1. Both match, and the /16 wins.

The local route covers the whole VPC range and cannot be deleted. Since August 2021 you can add a route more specific than it, such as 10.0.20.0/24 pointing at a firewall appliance’s interface. Traffic from the app subnet to the data subnet then passes through that appliance first.

Each packet is looked up in the route table of the subnet it leaves, so a request and its reply use two different tables. The customer’s SYN-ACK leaves through public-rt. The ALB’s packets to the app leave through public-rt, and the app’s replies through app-rt. The app’s packets to Postgres use app-rt, and Postgres replies through data-rt.

That is five route lookups and one address translation: 6 checks, and none of them writes a log line when it fails. Each lookup is made per packet, so nothing forces a reply to retrace the request’s path. Chapter 06 shows what such an asymmetric path does to a stateful firewall.

A security group is a set of allow rules attached to a network interface, not to a subnet. AWS considers every rule together, in no order, and drops any packet that matches none of them. Once a connection is allowed, its replies pass automatically, whatever the rules for the other direction say.

sg-alb, on the load balancer’s nodes, allows 443 in from anywhere and 8080 out to the app tier. Nothing else is written down, and nothing else gets through. A group holds up to 60 inbound and 60 outbound rules by default, and an interface carries up to 5 groups, or 16 after a quota increase.

sg-app, on the app servers, allows TCP 8080 from sg-alb. The source is a group ID instead of a CIDR, so any network interface that carries sg-alb matches, at whatever address it has right now. The load balancer adds and removes nodes as traffic changes, each with a new private address, and the rule never needs an edit.

Group references work across peered VPCs in the same Region. Since September 2024 they also work across a Transit Gateway, for inbound rules only, once the feature is turned on for both the gateway and the VPC attachment.

A network ACL is a numbered list of allow and deny rules evaluated on every packet that enters or leaves a subnet. It works like a ticket barrier: every packet presents its ticket, in both directions, and the barrier keeps no list of who already went through.

The network ACL reads its rules lowest number first, and the first match decides; nothing below it is read. A final * rule denies whatever is left. public-nacl also carries rule 90, a deny for a scraper at 192.0.2.99/32, which no security group could express. The default quota is 20 rules per direction, raisable to 40.

The customer’s browser picked source port 51234 for this connection, an ephemeral port. The load balancer’s reply therefore leaves the public subnet addressed to port 51234, and public-nacl tests it against the outbound rules like any other packet. A rule that allows only 443 out does not match it, so the reply falls through to the final deny.

Outbound rules have to open the range clients pick from. AWS lists them: 32768–61000 on Amazon Linux, 49152–65535 on Windows Server 2008 and later, 1024–65535 for NAT gateways and load balancers. Opening 1024–65535 covers all of them. Where port 51234 comes from is in the TCP chapter of the press-Enter tutorial.

The app opened its Postgres connection at 14:00:05, two minutes before the customer tapped Pay. sg-app outbound and sg-db inbound checked its first packet then, and connection tracking recorded the connection. Every later packet on it, in either direction, matches that record and passes with no security group rule evaluated.

The network ACLs keep no such record. One query and its reply cross the edges of the app and data subnets once each way: 4 evaluations. Edit a security group rule and a tracked connection carries on until it sits idle past its timeout: 432,000 seconds (5 days) on most instance types, 350 seconds by default on Nitro v6, configurable from 60 to 432,000.

Add up the first three connections. The network ACLs ran 10 evaluations: 2 for the customer’s connection, 4 between the load balancer and the app, and 4 for the database query. Security groups ran 3: 1 on the customer’s connection, 2 on the new load balancer connection, and none on the pooled database connection. Σ rises from 6 to 19 checks.

The gap is the price of each design. A stateless filter has to be told about both directions and pays one evaluation for every packet that crosses it. A stateful filter pays once per connection and nothing per packet afterwards, which is why sg-db never appears in this chapter’s ledger.

app-rt sends 0.0.0.0/0 to nat-0a1, the NAT gateway in the AZ1 NAT subnet. The NAT gateway replaces the source 10.0.10.27 with its own private address, 10.0.2.9, and picks a new source port. It keeps the mapping between the two, so the reply can be translated back to the app server.

The packet then leaves through the internet gateway, which maps 10.0.2.9 to the NAT gateway’s public address, 198.51.100.7, exactly as it mapped the load balancer’s address in chapter 01. The payment API sees that address and no other; 10.0.10.27 never leaves the VPC.

The payment provider accepts calls only from source addresses on its allowlist. Each zonal NAT gateway belongs to a single Availability Zone, so shop-prod runs nat-0a1 in AZ1 and nat-0b2 in AZ2, each with its own Elastic IP. Each AZ’s app route table points at the gateway in its own zone, and the provider allowlists both addresses.

Since 2025-11-19 a regional NAT gateway can cover several Availability Zones under one ID and needs no public subnet. shop-prod keeps two zonal ones, because chapter 06 places a firewall endpoint beside each of them.

Each NAT gateway address holds up to 55,000 simultaneous connections to one destination, meaning one IP address, port and protocol. Every connection from 10.0.2.9 to api.payments.example:443 needs a source port of its own. Once they are all in use, new connections fail, and the CloudWatch metric ErrorPortAllocation counts each one.

A NAT gateway can carry up to 8 addresses, 1 primary and 7 secondary, though only 2 Elastic IPs per gateway are allowed until you request more. Eight addresses give 440,000 connections to that same destination. Bandwidth scales on its own, from 5 Gbps to 100 Gbps.

A NAT gateway is billed on two meters. The first is time: $0.045 for every hour it exists, $32.85 a month at 730 hours, so the two zonal gateways cost $65.70 a month before a byte moves. Their two Elastic IPs add $3.65 a month each, $7.30 together.

The second meter is data. Every gigabyte the NAT gateway processes costs another $0.045, outbound or inbound, so the payment request and its response are both on the bill. That charge stacks on the normal internet data transfer rate, which is why traffic that can avoid the NAT gateway, such as the S3 upload in chapter 04, usually should.

A NAT gateway takes no security group, so the only filter in the NAT subnet is nat-nacl. It evaluates the call at four points: the request entering and leaving the subnet on its way to the API, then the response entering and leaving on its way back. sg-app checks the call once, on the first packet out of the app server.

The tally for the payment call is 6 network ACL evaluations, 3 route lookups, 2 address translations and 1 security group rule: 12 checks. Σ rises from 19 to 31 checks, and no other connection in the request has this many so far.

A gateway endpoint for S3 adds one row to app-rt. Its destination is not a CIDR block but a prefix list, pl-s3, the set of S3 address ranges in us-east-1 that AWS keeps up to date. Its target is the endpoint, vpce-s3.

Every range in that list is longer than /0, so longest-prefix match picks it over 0.0.0.0/0 → nat-0a1 for any S3 address. The receipt upload stops crossing the NAT gateway, and the app’s code does not change. Gateway endpoints exist for two services only, S3 and DynamoDB, and AWS charges nothing for them.

Chapter 03’s NAT gateway charges $0.045 for every gigabyte it processes. The shop writes 2 TB of receipt PDFs a month, so sending them through nat-0a1 costs 2,048 GB × $0.045 = $92.16 a month in processing alone.

Through the gateway endpoint the same 2,048 GB cost $0. Transfer between EC2 and S3 in one Region is free on both paths, so the NAT processing line is the whole difference: same bucket, same SDK call, one route-table row apart.

A gateway endpoint carries an endpoint policy, a JSON policy that every request through it is tested against. The default one allows every action on every resource. The policy on vpce-s3 allows two actions, s3:GetObject and s3:PutObject, on one bucket, shop-receipts, and refuses everything else by omission.

That closes a path the network checks leave open. A compromised app server holds valid role credentials and a route to all of S3, so it could copy data into a bucket in another account. Through this endpoint that write is refused, whatever the credentials allow.

The bucket policy on shop-receipts answers the reverse question: which paths may reach the bucket. It denies every request whose aws:SourceVpce key is not vpce-s3, so the same valid credentials used from a laptop are refused.

An address allowlist would not work here. aws:SourceIp is absent from the request context when the caller comes through a VPC endpoint, so a condition written on public addresses never matches this request. The key for a private source address is aws:VpcSourceIp.

Add the app role’s identity policy, and three policies decide this one PutObject. With the security group rule, two network ACL rules and one route lookup, connection C4 adds 7 checks, and Σ moves from 31 to 38.

Secrets Manager has no gateway endpoint, so it uses the other kind. An interface endpoint, built on PrivateLink, puts a network interface with a private address into one subnet per AZ: 10.0.10.41 in app-a and 10.0.11.41 in app-b. With private DNS on, the service’s public hostname resolves to those addresses instead of a public one.

Interface endpoints are billed: $0.01 per hour per AZ, which is $14.60 a month for two AZs, plus $0.01 per GB. Each carries up to 10 Gbps per AZ and scales to 100 Gbps. The app read its database password at 14:00:05, during boot, so this endpoint adds no checks to the 14:02:07 request.

A VPC peering connection routes traffic between exactly two VPCs whose ranges do not overlap. Each side adds a route to the other’s range that points at the connection. It is not transitive: A peered with B and B peered with C does not let A reach C.

Neither side can use the other’s internet gateway, NAT gateway, VPN or gateway endpoint either. A full mesh of 10 VPCs takes 10 × 9 ÷ 2 = 45 connections, and one VPC can hold 125 at most. Data between two VPCs in the same AZ over peering has been free since May 2021.

A Transit Gateway is a regional router: each VPC attaches to it once, and its own route tables decide which attachments reach which. Give prod VPCs one table and dev VPCs another, put shared services in both, and leave out the routes between prod and dev. The two environments then cannot reach each other.

It is billed per attachment: $0.05 an hour, or $36.50 a month for each VPC, so 10 VPCs cost $365 a month before $0.02 per GB processed. A VPC attachment carries up to 100 Gbps per AZ, in packets up to 8,500 bytes. Like peering, it routes between ranges, so the fraud team’s identical 10.0.0.0/16 rules it out.

PrivateLink works across overlapping ranges because nothing is routed between the two VPCs. The fraud team puts a Network Load Balancer behind an endpoint service, vpce-svc-0fraudscore. On the consumer side, shop-prod creates an interface endpoint whose network interface takes 10.0.10.200 in app-a. The app connects to that local address, and the endpoint forwards the connection to the load balancer.

Connections open one way only, consumer to provider, and the provider accepts each endpoint before traffic flows. The scorer sees the load balancer’s private address as the source, not 10.0.10.27. If the provider turns on Proxy Protocol v2, each connection also carries the consumer’s endpoint ID.

The endpoint interface sits in app-a, the same subnet as the app server, so C6 crosses no network ACL: those rules apply only where traffic enters or leaves a subnet. Your account evaluates 4 checks: sg-app outbound, sg-vpce inbound, and one app-rt lookup in each direction.

The fraud team’s account evaluates the other 4: the load balancer’s security group, the route lookup to the scorer, the scorer’s security group, and the reply’s route lookup. Network Load Balancers gained security groups in August 2023, with an option not to enforce them on PrivateLink traffic. Σ reaches 46.

An on-premises network joins the Transit Gateway as one more attachment. A Site-to-Site VPN connection is two encrypted tunnels over the internet, and a standard tunnel carries up to 1.25 Gbps. Since 2025-11-12 a large tunnel carries up to 5 Gbps, on Transit Gateway or Cloud WAN only. Direct Connect is a dedicated private circuit, joined through a Direct Connect gateway.

Names need a path too. A Route 53 Resolver inbound endpoint lets on-premises servers resolve names inside the VPC, and an outbound endpoint forwards the VPC’s queries for the data center’s domains to its DNS servers. None of this sits on the checkout’s path, so it adds 0 checks.

Every DNS query from shop-prod goes to the Route 53 VPC Resolver at 10.0.0.2. Security groups don’t filter traffic to that address, and AWS’s documentation says a network ACL can’t block DNS requests to or from the resolver either. So a compromised server can hide stolen data in the query name itself, and the resolver passes the name on to whoever runs that domain. The label aGVsbG8 is the word “hello” in base64.

Route 53 Resolver DNS Firewall closes that path from inside the VPC. A rule group associated with the VPC checks every query against domain lists, and each rule allows, alerts on or blocks a match. A blocked query gets the reply you choose, here NXDOMAIN, so the name never leaves AWS. This request’s three first lookups after boot are three checks, numbers 47 to 49.

The resolver has one more limit: 1,024 packets per second from each network interface. Past that, queries are throttled whatever the rules say, which is the first thing to check when DNS fails only under load.

The instance metadata service answers at 169.254.169.254 on every instance, and one of its paths returns the instance role’s temporary credentials. Network ACLs can’t block it. Any bug that makes the server fetch a URL an attacker chose, a server-side request forgery (SSRF), can be pointed at that path.

AWS’s post introducing IMDSv2, on 2019-11-19, names that threat: SSRF through open firewalls and reverse proxies. IMDSv2 answers it in two ways. Credentials need a session token, and the token comes only from a PUT, which a forwarded GET can’t become. The token response also leaves with a hop limit, 1 by default and 2 on AMIs marked v2.0, so a proxy one network hop further away never receives it.

This request adds 0 checks here. The SDK fetched these credentials at 14:00:05, during boot, and reuses them until they expire, so Σ stays at 49.

AWS Network Firewall is a managed stateful firewall. It runs as one endpoint per Availability Zone, each in a small subnet of its own: 10.0.3.0/28 in AZ1 and 10.0.3.16/28 in AZ2. Traffic reaches an endpoint only because a route table sends it there, which is why the next step is all routes.

The shop-prod policy allows outbound TLS to a list of server names and drops the rest. The firewall reads the name from the SNI field of the TLS ClientHello, which the client sends before any encryption starts. api.payments.example matches rule 1 and passes: check 51. A connection to any other host matches rule 2 and is dropped.

The price is per endpoint-hour: $0.395, which is $288.35 a month for one AZ and $576.70 for two, before $0.065 per GB processed. By the hour, that is almost nine times what chapter 03’s two NAT gateways cost.

Inserting the firewall takes three route changes. nat-rt now sends 0.0.0.0/0 to the firewall endpoint vpce-fw instead of the internet gateway, and fw-rt, the firewall subnet’s table, sends it on to the internet gateway. Outbound, the payment call now runs NAT gateway, firewall, internet gateway.

The reply is the hard part. It reaches the internet gateway addressed to the NAT gateway’s 10.0.2.9, and the VPC’s local route would deliver it straight there, past the firewall. Ingress routing closes the gap: a route table associated with the internet gateway itself, called an edge association, with one row sending 10.0.2.0/28 to vpce-fw. Without that row the firewall sees only the outbound half.

Count what the payment call gained: 4 network ACL checks on fw-nacl, 3 route lookups and 1 firewall rule, 8 in all. With the 3 DNS lookups from step 1 that makes 11, and Σ reaches 57.

Larger networks often run their firewalls in one central inspection VPC and send every VPC’s traffic there through a Transit Gateway. By default a transit gateway keeps traffic in the Availability Zone it came from. When the client sits in AZ1 and the server in AZ2, the request crosses the firewall in AZ1 and the reply comes back through the firewall in AZ2.

A stateful firewall that never saw the SYN treats the reply as a stray packet and drops it. Appliance mode, a setting on the transit gateway’s attachment to the inspection VPC, fixes this. It picks one AZ from the flow’s addresses and ports and keeps both directions there for the flow’s lifetime.

Third-party appliances sit behind a Gateway Load Balancer instead, which carries packets to them wrapped in GENEVE on UDP port 6081. None of this is on the checkout request’s path, so this step adds 0 checks and Σ stays at 57.

A flow log record describes one flow on one network interface during one aggregation window. The ALB node’s interface, eni-0alba, records C1 as a single line: version 2, the account, the interface, both addresses and ports, protocol 6 for TCP, 14 packets, 6,230 bytes, the window’s start and end in Unix seconds, then ACCEPT and OK.

The window is 10 minutes by default, and you can set it to 1 minute when you create the flow log. On Nitro instances it is always 1 minute or less, whatever you set. A record is written only after its window closes, so a failed connection can take minutes to show up.

Read the fields again for what is missing. There is no security group name or rule number anywhere, and not one byte of payload. A flow log records what was decided at the interface, never which rule decided it.

A flow log doesn’t name the filter that dropped a packet, but the pattern of records does. A security group is stateful: once it accepts a request, it lets the reply out without checking any rule. A rejected reply therefore needs something that checks the reply again, and in a VPC that is a network ACL.

Remove check 9, the public-nacl outbound rule for ports 1024–65535, and C1 produces exactly this pair. The customer’s request to port 443 is an inbound ACCEPT. The load balancer’s reply to port 51234 is an outbound REJECT. The customer’s browser retries, so both records count three packets.

The opposite pattern, an inbound REJECT with nothing after it, fits either filter. The scanner’s attempt on port 22 at the top of the list is one.

Flow logs skip whole categories of traffic by design. Three are services every instance uses: the Amazon DNS server, the metadata service at 169.254.169.254 and Time Sync at 169.254.169.123. The rest are DHCP, Windows license activation, the default router’s reserved address, ARP, mirrored traffic, and traffic between an endpoint interface and a Network Load Balancer interface.

So none of this request’s three lookups appears in a flow log, and neither does the credential fetch at 14:00:05. A DNS exfiltration attempt like chapter 06’s leaves no flow log line at all. Resolver query logging records every query instead, with its name, type, response code and the instance that sent it.

Map each of the 57 checks to the place its failure would appear. Your own flow logs cover 29: every network ACL check and every security group check on your side. The provider’s flow logs cover 2, in the fraud team’s account. Resolver query logs cover the 3 DNS Firewall checks, and CloudTrail or the S3 AccessDenied message covers the 3 policies.

The Network Firewall alert log covers check 51, and the CloudWatch metric ErrorPortAllocation covers check 24, the NAT gateway’s port allocation. That leaves 18: the 16 route lookups and the 2 internet-gateway translations. When one of those fails, nothing writes a log line. The only clue is an ACCEPT with no reply.

Reachability Analyzer answers a question about one path, such as “can 10.0.10.27 reach the internet on TCP 443?” It reads the route tables, network ACLs, security groups and gateways along the path, sends no packet, and reports the first component that blocks it. Each analysis costs $0.10.

Network Access Analyzer checks a whole property instead, such as “no data subnet is reachable from an internet gateway”, and lists every path that breaks it, at $0.002 per network interface analyzed. Both read configuration rather than traffic, which makes them the tools for the 18 checks that log nothing.