1 LAPTOP · 1 DAY · 21 REQUESTS · 0 OPEN PORTS

AWS access:
how one laptop reaches a private database

ONE LAPTOP · 2026-11-09· ledger-prod · us-east-1 ·
requests the laptop sent to AWS·····································21
HOW EACH ONE PROVED WHO SENT IT
signed with a long-term key·····································7
signed with a temporary key·····································7
an SSO bearer token·····································3
a Session Manager stream token·····································2
no AWS credential at all·····································2
AWS secret keys the laptop sent·····································0

Every request to AWS has to prove who sent it. This tutorial follows one laptop through one working day, from an access key pasted in at 09:00 to a SQL query against a private database at 14:10, and counts every request it sends to AWS. There are 21, and none of them carries your secret access key.

The scenes play themselves, so watch or just scroll. are tappable. Every account ID, key and hostname is an AWS documentation placeholder, and every price is us-east-1, November 2026.

Two strings, and the account they belong to

Your first AWS credential is 60 characters long: a 20-character access key ID and a 40-character secret access key. A teammate created it for the IAM user new-engineer, and at 09:00 you paste it into your laptop. This tutorial counts every request the laptop sends to AWS that day. The count starts at 0, pasting the key adds nothing, and by the end of this chapter it is at 2.

STEP 01 / 04

TWENTY CHARACTERS YOU MAY SHOW, FORTY YOU MAY NOT

IAM user new-engineer · long-term access key
AKIAIOSFODNN7EXAMPLEaccess key ID · 20 chars · sent with every request
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYsecret access key · 40 chars · shown once, never sent
noneExpiration
Fig 0.1The accent string is the one AWS shows exactly once. The key ID next to it travels with every request.

An belongs to one and has two parts. The access key ID, AKIAIOSFODNN7EXAMPLE, identifies the key the way a username identifies a person. It travels with every request and appears in every CloudTrail event the key causes, so it is not a secret.

The is 40 characters long, and AWS shows it only once, in the response that creates the key. No API returns it later. Lose it, and the only fix is to delete the key and create a new one.

Each IAM user can have at most two access keys at a time. The second slot exists so a rotation can overlap: create the new key, move every program to it, then delete the old one.

STEP 02 / 04

FOUR LETTERS SAY WHAT KIND OF KEY

AKIA…long-term access key~/.aws/credentials
ASIA…temporary access key, from STSAWS_ACCESS_KEY_ID after assume-role
AIDA…IAM userUserId from get-caller-identity
AROA…IAM roleRoleId, and an assumed role’s UserId
ABIA…STS service bearer tokena few service logins
ANPA…managed policyPolicyId from iam get-policy
Fig 0.2Read the first four letters of each ID. The two outlined cards are this tutorial’s two kinds of access key.

Every IAM identifier starts with a four-letter prefix that names its type. AKIA marks a long-term access key, like the one in your credentials file. ASIA marks a temporary access key issued by STS, the subject of chapter 02. IAM’s documentation adds that an ASIA key ID is unique only together with its secret and its session token.

The other prefixes name identities rather than keys: AIDA is an IAM user, AROA a role, ANPA a managed policy, ABIA an STS service bearer token. The reference page for STS GetAccessKeyInfo states the same rule for keys: AKIA means long-term, and ASIA means temporary.

STEP 03 / 04

A PLAIN-TEXT FILE WITH NO EXPIRY DATE

~/.aws/credentials
[default]aws_access_key_id = AKIAIOSFODNN7EXAMPLEaws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
plain text · readable by any process running as you
~/.aws/config
[default]region = us-east-1output = json
expiry date in either file: none
Fig 0.3Compare the two files: the accent line is the secret, in plain text, with no expiry date anywhere near it.

aws configure asks four questions and writes the answers to two files. The key ID and secret go under [default] in ~/.aws/credentials, and the Region and output format go in ~/.aws/config. Both are plain-text INI files. Any process running as you can read them, and so can backups, dotfile repositories and container images built from your home directory.

The command sends nothing to AWS. It doesn’t check that the key exists or works, so a mistyped secret only shows up on the first real request, as SignatureDoesNotMatch. Nothing in either file says when the key expires, because an AKIA key never does.

STEP 04 / 04

THE ONE CALL NOTHING CAN DENY

  1. #1$ aws sts get-caller-identitysts:GetCallerIdentity✓ ok09:02:10AKIAArn arn:aws:iam::111122223333:user/new-engineer
  2. #2$ aws sts get-access-key-infosts:GetAccessKeyInfo✓ ok09:03:40AKIAAccount 111122223333
Σ requests the laptop sent2
Fig 0.4Rows #1 and #2 are the laptop’s first requests. The total under them is the counter this tutorial keeps.

At 09:02:10 you run aws sts get-caller-identity, request #1. It returns a UserId starting AIDA, the Account 111122223333 and the Arn of user new-engineer. The call needs no IAM permission and works even under an explicit deny, because the same details come back in any access-denied error.

At 09:03:40, aws sts get-access-key-info with --access-key-id AKIAIOSFODNN7EXAMPLE is request #2. It returns the account that owns the key, from the key ID alone, and doesn’t say whether the key is still active. Pasting the key in sent nothing, so Σ requests the laptop sent is now 2.

REQUESTS #1 AND #2: WHO AM I, AND WHOSE KEY
# aws configure sent nothing; these are requests #1 and #2
aws sts get-caller-identity
# {
#     "UserId": "AIDACKCEVSQ6C2EXAMPLE",
#     "Account": "111122223333",
#     "Arn": "arn:aws:iam::111122223333:user/new-engineer"
# }

aws sts get-access-key-info --access-key-id AKIAIOSFODNN7EXAMPLE
# {
#     "Account": "111122223333"
# }

WHAT YOU NOW KNOW

  • An access key is a 20-character ID that travels with every request and a 40-character secret that AWS shows only once.
  • AKIA keys belong to IAM users and never expire; ASIA keys come from STS and always do.
  • aws configure sent nothing. get-caller-identity and get-access-key-info were the laptop’s first 2 requests.
deep dive: rotating keys, last-used dates, and the root user’s keys

Rotation uses the two-key limit. Create a second key, deploy it everywhere the first one is used, then mark the first key Inactive with aws iam update-access-key. An inactive key is refused but can be switched back on, so leave it inactive for a while before deleting it.

aws iam get-access-key-last-used returns the date, Region and service of a key’s most recent request. The IAM credential report lists every user’s keys with their ages and last-used dates. Together they show which keys nobody uses, and those are the cheapest ones to delete.

The root user can have access keys too, and AWS’s advice is to have none. The pair in this tutorial, AKIAIOSFODNN7EXAMPLE and its secret, is AWS’s own documentation placeholder, which is why it appears in so many READMEs.

Named profiles let one laptop hold several keys: aws configure --profile staging writes a [staging] section next to [default], and aws configure set changes one value without the prompts.

ERRORS THIS CHAPTER EXPLAINS

  • Unable to locate credentials → no source the CLI checks had a key → run aws configure, or see chapter 03
  • InvalidClientTokenId → the key ID doesn’t exist, or was deleted → check the key ID; chapter 05
  • SignatureDoesNotMatch → the secret is wrong, often with a pasted trailing space → paste the secret again; chapter 01

The secret signs the request and stays home

Both requests counted so far carried a header that begins Authorization: AWS4-HMAC-SHA256. That header is Signature Version 4, SigV4, and it lets a request prove who sent it while the secret stays on the laptop. This chapter takes one request apart, #3, an aws s3 ls at 09:05, and Σ requests the laptop sent goes from 2 to 3.

STEP 01 / 05

WHAT ACTUALLY CROSSES THE WIRE

REQUEST #3 · 09:05:12 · aws s3 ls s3://ledger-invoices/2026/11/
GET /?list-type=2&prefix=2026%2F11%2F&delimiter=%2F&encoding-type=url
Hostledger-invoices.s3.us-east-1.amazonaws.com
X-Amz-Date20261109T090512Z
X-Amz-Content-Sha256e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
AuthorizationAWS4-HMAC-SHA256
Credential=AKIAIOSFODNN7EXAMPLE/20261109/us-east-1/s3/aws4_request
SignedHeaders=host;x-amz-content-sha256;x-amz-date
Signature=7600dfd183917886590d5704260e7ab452dde7f822f9b5bc38720d04f847146b
characters of the secret access key in this request0 of 40
Fig 1.1Look for the secret access key in this request: it isn’t there. The accent Signature line is what stands in for it.

At 09:05:12, aws s3 ls s3://ledger-invoices/2026/11/ sends request #3, a GET to the bucket with the prefix in the query string. Its headers carry the host, an X-Amz-Date timestamp and X-Amz-Content-Sha256, the SHA-256 of the body, which for a GET is the hash of zero bytes.

The last header is Authorization, in form: the algorithm, a Credential= naming the key ID with a date, Region and service, the list of SignedHeaders, and a Signature= of 64 hex characters, one HMAC-SHA256. None of the secret’s 40 characters appears in the request. Σ requests the laptop sent is now 3.

STEP 02 / 05

SIX LINES THAT DESCRIBE THE REQUEST

canonical request · 6 parts
methodGET
path/
query, sorteddelimiter=%2F&encoding-type=url&list-type=2&prefix=2026%2F11%2F
headers, lowercased and sortedhost:ledger-invoices.s3.us-east-1.amazonaws.comx-amz-content-sha256:e3b0c442…7852b855x-amz-date:20261109T090512Z
signed headershost;x-amz-content-sha256;x-amz-date
body hashe3b0c442…7852b855 · SHA-256 of an empty body
string to sign · 4 lines
algorithmAWS4-HMAC-SHA256
timestamp20261109T090512Z
credential scope20261109/us-east-1/s3/aws4_request
hash of the canonical request20b8044d…fc0f88f2
Fig 1.2The accent block is rebuilt from the request in Fig 1.1. Its hash goes into the four lines below, which get signed.

Before signing, the CLI writes a canonical request: the method, the path, the query string sorted by name, the headers lowercased and sorted, the list of signed header names, and the SHA-256 of the body. Canonical means there is exactly one way to write it, so two machines that see the same request produce the same bytes.

The string to sign is four lines: AWS4-HMAC-SHA256, the timestamp, the credential scope, and the SHA-256 of the canonical request. AWS rebuilds both from what it received and signs them with its own copy of the secret. If one byte of a signed part changed on the way, the two signatures differ and the answer is SignatureDoesNotMatch.

STEP 03 / 05

FOUR HMACS BEFORE THE FIRST SIGNATURE

  1. secret"AWS4" + the 40-character secretnever leaves the laptop
  2. kDateHMAC(secret, "20261109")2026-11-09 only
  3. kRegionHMAC(kDate, "us-east-1")+ us-east-1 only
  4. kServiceHMAC(kRegion, "s3")+ s3 only
  5. kSigningHMAC(kService, "aws4_request")signs today’s S3 requests in us-east-1
Fig 1.3Follow the chain down: each rung keys the next HMAC, and the accent kSigning is the only key that signs.

The secret never signs a request directly. It goes through four HMACs first. kDate is the HMAC of the date 20261109 under the key “AWS4” plus the secret. kRegion is the HMAC of us-east-1 under kDate, kService adds s3, and kSigning adds the fixed string aws4_request.

The signature is the hex HMAC of the string to sign under kSigning. AWS derives the same key from its own copy of the secret, so both sides agree without the secret crossing the network. The derived key is good for one date, one Region and one service. The same signature sent to S3 in us-west-2, or to STS, is refused.

THE SIGNATURE ON REQUEST #3, BY HAND
import hashlib
import hmac

secret = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"  # 40 chars, never sent

def hmac_sha256(key: bytes, msg: str) -> bytes:
    return hmac.new(key, msg.encode(), hashlib.sha256).digest()

k_date = hmac_sha256(("AWS4" + secret).encode(), "20261109")
k_region = hmac_sha256(k_date, "us-east-1")
k_service = hmac_sha256(k_region, "s3")
k_signing = hmac_sha256(k_service, "aws4_request")  # valid for 2026-11-09, us-east-1, s3 only

canonical = build_canonical_request(request)  # the canonical request from Fig 1.2
string_to_sign = "\n".join([
    "AWS4-HMAC-SHA256",
    "20261109T090512Z",
    "20261109/us-east-1/s3/aws4_request",
    hashlib.sha256(canonical.encode()).hexdigest(),
])
signature = hmac.new(k_signing, string_to_sign.encode(), hashlib.sha256).hexdigest()  # 64 hex chars
STEP 04 / 05

FIVE MINUTES TO ARRIVE

X-Amz-Date
✗ replay
09:0509:0709:0909:11
5 minutes after X-Amz-Date · accepted until 09:10:12
  • 09:05:12X-Amz-Date, signed into request #3
  • 09:05:12.3#3 arrives at S3✓ accepted
  • 09:11:00the same bytes, replayed✗ refused, signature still correct
Fig 1.4The accent bar is the five minutes after #3’s timestamp. The replay at 09:11:00 falls outside it and is refused.

X-Amz-Date is one of the signed headers, so the timestamp can’t be changed without breaking the signature. In most cases a request must reach AWS within five minutes of that timestamp, or AWS refuses it.

That rule limits replay: a request copied off the network stops working a few minutes later, although its signature is still correct. It also refuses honest requests from a laptop whose clock has drifted, which is why a Signature expired error is fixed with NTP, not with new keys.

STEP 05 / 05

A SIGNATURE YOU CAN HAND TO SOMEONE ELSE

aws s3 presign · 09:08:30 · computed on the laptop, nothing sent
https://ledger-invoices.s3.us-east-1.amazonaws.com/2026/11/INV-0042.pdfthe object, a GET with no Authorization header
?X-Amz-AlgorithmAWS4-HMAC-SHA256the same algorithm as Fig 1.1
&X-Amz-CredentialAKIAIOSFODNN7EXAMPLE/20261109/us-east-1/s3/aws4_requestkey ID and credential scope (URL-encoded in the real link)
&X-Amz-Date20261109T090830Zwhen it was signed
&X-Amz-Expires3600seconds this link works
&X-Amz-SignedHeadershostonly the Host header is signed
&X-Amz-Signature64 hex charactersanyone holding the URL can make this one GET as you
Fig 1.5The accent parameter is a 64-character signature like Fig 1.1’s, moved out of a header and into the URL.

aws s3 presign s3://ledger-invoices/2026/11/INV-0042.pdf --expires-in 3600 moves the same kind of signature from the header into the query string, as six X-Amz parameters. The CLI computes all of it on the laptop and sends nothing, so Σ requests the laptop sent stays at 3.

The result is a . Whoever holds it can make that one GET as you, with no credentials of their own, until it expires. Signed with an IAM user’s key, it can last up to 7 days. Signed with temporary credentials, it stops working when they expire, whatever X-Amz-Expires says.

A PRESIGNED URL: SIGNED ON THE LAPTOP
aws s3 presign s3://ledger-invoices/2026/11/INV-0042.pdf --expires-in 3600

# https://ledger-invoices.s3.us-east-1.amazonaws.com/2026/11/INV-0042.pdf
#   ?X-Amz-Algorithm=AWS4-HMAC-SHA256
#   &X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20261109%2Fus-east-1%2Fs3%2Faws4_request
#   &X-Amz-Date=20261109T090830Z
#   &X-Amz-Expires=3600
#   &X-Amz-SignedHeaders=host
#   &X-Amz-Signature=<64 hex characters>

# no request was sent: the URL was computed on the laptop

WHAT YOU NOW KNOW

  • SigV4 sends the key ID, a timestamp and a 64-character signature. The 40-character secret never leaves the laptop.
  • The signature covers a canonical copy of the request, signed with a key that is valid for one date, one Region and one service.
  • aws s3 ls was request #3. The presigned URL added nothing to the count, because the signing happened on the laptop.
deep dive: SigV4a, aws --debug, and signatures inside URLs

SigV4a, algorithm AWS4-ECDSA-P256-SHA256, derives an ECDSA key pair from the same secret and signs with the private half. S3 Multi-Region Access Points use it, because one signature has to be valid in several Regions. The CLI and SDKs switch to it on their own when an endpoint needs it.

aws s3 ls s3://ledger-invoices/2026/11/ --debug logs the canonical request and the string to sign that the CLI built. Comparing them with what a proxy forwarded is the quickest way to find the byte that changed.

Large S3 uploads can skip hashing the body up front. UNSIGNED-PAYLOAD in X-Amz-Content-Sha256 leaves the body out of the signature, and chunked signing signs each chunk as it is sent. The RDS database token in chapter 07 is this chapter’s presigned URL again, with a database as the target.

A presigned URL for an object encrypted with SSE-KMS also needs kms:Decrypt for whoever signed it. AWS KMS: how one invoice stays encrypted for seven years follows that second check.

ERRORS THIS CHAPTER EXPLAINS

  • SignatureDoesNotMatch → the secret is wrong, or a proxy rewrote a signed header → paste the secret again, or compare the --debug output
  • Signature expired, or S3’s RequestTimeTooSkewed → the laptop’s clock is more than five minutes off → turn on NTP
  • AuthorizationHeaderMalformed → the credential scope names a different Region from the bucket’s → set the bucket’s Region with --region

Why some keys need a third string

The onboarding doc for ledger-prod tells you to export three variables, not two: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. The third one comes from STS, and an hour after you export it, the same command stops working. This chapter follows the three values from the request that issues them to the request that fails, and takes the count from 3 to 6.

STEP 01 / 05

A ROLE HANDS OUT KEYS THAT EXPIRE

response to #4 · sts:AssumeRole · role ops-readonly
ASIAIOSFODNN7EXAMPLEaccess key ID · 20 chars
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYsecret access key · 40 chars · never sent
IQoJb3JpZ2luX2VjE…EXAMPLETOKEN==session token · length varies
2026-11-09T10:20:31ZExpiration · in 3,600 s
DurationSeconds asked for3,600 (the default)
allowed range for this role900–43,200
Fig 2.1Four fields came back. The accent Expiration is the one field a long-term key never has.

The IAM user new-engineer can do almost nothing in ledger-prod on its own. What it may do is assume the ops-readonly. At 09:20:31, aws sts assume-role sends request #4 to , signed with the AKIA key from chapter 00.

STS returns four fields: an AccessKeyId starting ASIA, a SecretAccessKey, a SessionToken and an Expiration of 2026-11-09T10:20:31Z. Together they are . Temporary credentials work like a hotel key card: encoded for one room and one checkout time, and useless after checkout without anyone having to collect them.

DurationSeconds sets the lifetime. It defaults to 3,600 seconds, and it can range from 900 up to the maximum configured on the role, which is at most 43,200 seconds (12 hours).

REQUEST #4, AND THE THREE VARIABLES THE ONBOARDING DOC ASKS FOR
aws sts assume-role \
  --role-arn arn:aws:iam::111122223333:role/ops-readonly \
  --role-session-name onboarding \
  --serial-number arn:aws:iam::111122223333:mfa/new-engineer \
  --token-code 123456                      # request #4, signed with the AKIA key
# "Credentials": {
#     "AccessKeyId": "ASIAIOSFODNN7EXAMPLE",
#     "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
#     "SessionToken": "IQoJb3JpZ2luX2VjEXAMPLETOKEN==",
#     "Expiration": "2026-11-09T10:20:31+00:00"     # 3,600 s from now
# }

export AWS_ACCESS_KEY_ID=ASIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEXAMPLETOKEN==
STEP 02 / 05

WHY THE THIRD STRING EXISTS

AKIAIOSFODNN7EXAMPLElong-term key ID
IAM user new-engineera record stored in IAM, listed by list-access-keys
ASIAIOSFODNN7EXAMPLEtemporary key ID
no IAM recordno user owns it; list-access-keys never shows it
session token · IQoJb3JpZ2luX2VjE…EXAMPLETOKEN==
role
ops-readonly
session name
onboarding
expires
2026-11-09T10:20:31Z
MFA used when created
yes
session policy
none
Fig 2.2Compare the rows: the AKIA key points to a record in IAM; everything about the ASIA key travels in the accent token.

IAM has no record of an ASIA key. No user owns it, and aws iam list-access-keys never lists it. IAM’s documentation says an ASIA key ID is unique only in combination with its secret access key and its session token.

The is an opaque string from STS. It tells the service that receives a request which role session the key belongs to, when that session expires, and which session policies and tags apply. Those policies and tags take up room inside the token, so AssumeRole reports how much of the maximum they used, in PackedPolicySize.

STEP 03 / 05

THE TOKEN RIDES IN A SIGNED HEADER

REQUEST #5 · s3:ListObjectsV2 · compared with #3
GET /?list-type=2&prefix=2026%2F11%2F HTTP/1.1
Hostledger-invoices.s3.us-east-1.amazonaws.com
X-Amz-Date20261109T092105Z
X-Amz-Content-Sha256e3b0c44298fc1c149afbf4c8996fb924…7852b855
+ X-Amz-Security-TokenIQoJb3JpZ2luX2VjE…EXAMPLETOKEN==
AuthorizationAWS4-HMAC-SHA256
Credential=ASIAIOSFODNN7EXAMPLE/20261109/us-east-1/s3/aws4_request
SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token
Signature=64 hex characters, made with the temporary secret
+ marks the one header request #3 did not have
Fig 2.3Compare with Fig 1.1: the accent X-Amz-Security-Token header is new, and the SignedHeaders list now names it.

At 09:21:05, with the three values exported, aws s3 ls sends request #5. Compared with #3 in chapter 01, one header is new: X-Amz-Security-Token, which carries the session token. Its name appears in SignedHeaders, so the token is covered by the signature, just as the timestamp is.

The Credential= field now starts with ASIA, and the signature is computed exactly as in chapter 01, with the temporary secret in place of the long-term one. An ASIA key sent without its token is refused with InvalidClientTokenId, even when the signature is correct.

STEP 04 / 05

A SIX-DIGIT CODE, CHECKED ONCE

trust policy · role ops-readonly
Effect
Allow
Principal
arn:aws:iam::111122223333:root
Action
sts:AssumeRole
Condition
Bool · aws:MultiFactorAuthPresent = true
checked once, when #4 created the session
SESSION LIFETIMES
AssumeRole15 min–12 h
default 1 h
GetSessionToken, IAM user15 min–36 h
default 12 h
GetSessionToken, root user15 min–1 h
default 1 h
role chainingat most 1 h
whatever the role allows
Fig 2.4The accent Condition is why #4 needed a six-digit code. The table beside it gives each kind of session’s lifetime range.

The on ops-readonly lets the account assume the role only when aws:MultiFactorAuthPresent is true. That is why request #4 carried --serial-number, the ARN of new-engineer’s device, and --token-code with a 6-digit code.

STS checks the code when it creates the session and records that MFA was used. Requests signed with that session then count as MFA-authenticated until its Expiration, and the code is never sent again.

GetSessionToken is the other way to get MFA-backed credentials, with no role involved: 12 hours by default and up to 36 for an IAM user, and at most 1 hour for the root user. A role assumed from another role session, called role chaining, is capped at 1 hour.

THE TRUST POLICY ON OPS-READONLY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
      "Action": "sts:AssumeRole",
      "Condition": {
        "Bool": { "aws:MultiFactorAuthPresent": "true" }
      }
    }
  ]
}
STEP 05 / 05

AN HOUR LATER, THE SAME COMMAND FAILS

#4
#6 ✗
09:0009:3010:0010:30
credentials from #4 · valid until Expiration 10:20:31
  1. #4sts:AssumeRole✓ ok · expires 10:20:3109:20:31AKIA
  2. #5s3:ListObjectsV2✓ ok09:21:05ASIA + token
  3. #6s3:ListObjectsV2✗ ExpiredToken10:24:50ASIA + token
Σ requests the laptop sent6
Fig 2.5Rows #5 and #6 are the same command 63 minutes apart. The accent bar ends at the Expiration #4 returned.

At 10:24:50, with the same three variables still exported, aws s3 ls sends request #6, and S3 answers ExpiredToken. Nothing on the laptop changed. The Expiration inside the token passed at 10:20:31, four minutes earlier.

Exported variables are plain strings. Nothing watches their expiry time or fetches a replacement, so every command after 10:20:31 sends the same expired credentials. A profile can renew itself, which is chapter 03. This chapter adds 3 requests, and Σ requests the laptop sent is now 6.

WHAT YOU NOW KNOW

  • Temporary credentials are an ASIA key ID, a secret and a session token, and they stop working at Expiration without anyone cleaning them up.
  • The session token carries the session’s role, expiry and policies. Every request sends it in the signed X-Amz-Security-Token header.
  • assume-role (#4), one good read (#5) and one ExpiredToken (#6) bring the count to 6.
deep dive: GetSessionToken, web identity, and role chaining

STS has three operations for callers that already hold AWS credentials. AssumeRole returns credentials for a role, for up to the role’s maximum of 12 hours. GetSessionToken returns credentials for the same IAM user, usually to add MFA, for up to 36 hours. GetFederationToken, called with an IAM user’s long-term key, returns credentials for a federated user, also for up to 36 hours.

AssumeRoleWithWebIdentity and AssumeRoleWithSAML are called with no AWS credentials at all. The identity token from an OpenID Connect or SAML provider is the proof, the same idea chapter 04 uses for IAM Identity Center. GitHub Actions workflows and EKS service accounts reach AWS this way, and neither stores an access key.

Role chaining means using one role session to assume another role, and STS caps the second session at 1 hour whatever the role’s maximum says. Session policies and session tags passed to AssumeRole are packed into the token, which is one reason its size isn’t fixed.

STS has a global endpoint, sts.amazonaws.com, and regional ones such as sts.us-east-1.amazonaws.com. By default the global endpoint issues version 1 tokens, which don’t work in Regions you enabled by hand, such as Asia Pacific (Hong Kong). aws iam set-security-token-service-preferences --global-endpoint-token-version v2Token switches to version 2 tokens, which are longer and valid everywhere. AWS recommends the regional endpoints in any case.

Some services refuse a call with an encoded authorization message. aws sts decode-authorization-message turns it into readable JSON that names the policy and statement that refused, for a caller allowed sts:DecodeAuthorizationMessage.

ERRORS THIS CHAPTER EXPLAINS

  • ExpiredToken → the session passed its Expiration → get new credentials, or use a profile that renews them (chapter 03)
  • InvalidClientTokenId with an ASIA key → the session token is missing or truncated → export all three values, not two
  • AccessDenied … is not authorized to perform: sts:AssumeRole → the trust policy refused, often because no MFA code was sent → pass --serial-number and --token-code
  • The requested DurationSeconds exceeds the 1 hour session limit for roles assumed by role chaining → a role session asked for more than an hour → ask for 3,600 seconds or less

Ten places the CLI looks, in a fixed order

Environment variables are second in the AWS CLI’s list of places to look for credentials; your ~/.aws/credentials file is sixth. The variables exported in chapter 02 are still set, and they expired at 10:20:31. This chapter shows what that order costs you, how a profile avoids it, and takes the count from 6 to 9.

STEP 01 / 04

THE FIRST SOURCE THAT ANSWERS WINS

  1. 1command-line options--profile, --region
  2. 2environment variablesAWS_ACCESS_KEY_ID · AWS_SECRET_ACCESS_KEY · AWS_SESSION_TOKEN
rows 3–8 · read from the profile AWS_PROFILE names
  1. 3assume rolerole_arn + source_profile in ~/.aws/config
  2. 4assume role with web identityweb_identity_token_file
  3. 5IAM Identity Centersso_session in ~/.aws/config
  4. 6credentials file~/.aws/credentials
  5. 7custom processcredential_process
  6. 8config filekeys written in ~/.aws/config
  1. 9container credentialsECS task role endpoint
  2. 10EC2 instance profileinstance metadata service
↓ checked in this order; the first source that returns credentials is used
Fig 3.1Read from the top: the accent row is where the search stopped for #7, before any of the profile rows below it.

The is the fixed list of sources the AWS CLI v2 checks before it signs a request. In order: command-line options, environment variables, assume-role settings, web identity, IAM Identity Center, the credentials file, a custom process, the config file, ECS container credentials and the EC2 instance profile.

The CLI stops at the first source that returns credentials and never reads the rest. Sources are never merged, so a key ID from one place is never paired with a secret from another. The environment-variable source doesn’t check whether its keys have expired either: it hands them over as they are.

STEP 02 / 04

THE VARIABLE THAT OUTRANKS YOUR PROFILE

$ aws configure list
NAME
profile
VALUE
ops-readonly
TYPE
env
LOCATION
['AWS_PROFILE', 'AWS_DEFAULT_PROFILE']
NAME
access_key
VALUE
****************MPLE
TYPE
env
LOCATION
NAME
secret_key
VALUE
****************EKEY
TYPE
env
LOCATION
NAME
region
VALUE
us-east-1
TYPE
config-file
LOCATION
~/.aws/config
  1. #7$ AWS_PROFILE=ops-readonly aws s3 ls …s3:ListObjectsV2✗ ExpiredToken10:26:15ASIA + token
Fig 3.2The accent TYPE cell says env: the expired keys from chapter 02 answered before the profile was read.

At 10:26:15 you run export AWS_PROFILE=ops-readonly and then aws s3 ls again. Request #7 gets ExpiredToken, the same answer as #6. AWS_PROFILE only chooses which profile rows 3 to 8 read, and row 2 answered first, with the keys that expired at 10:20:31.

--profile ops-readonly on the command line would have worked. When a profile is named that way, botocore, the Python library under the CLI, leaves the environment-variable source out of the chain. aws configure list shows where each value came from, and here its TYPE column says env for both keys.

WHICH SOURCE ANSWERED, AND THE FIX
export AWS_PROFILE=ops-readonly
aws configure list
# NAME       : VALUE                : TYPE : LOCATION
# profile    : ops-readonly         : env  : ['AWS_PROFILE', 'AWS_DEFAULT_PROFILE']
# access_key : ****************MPLE : env  :
# secret_key : ****************EKEY : env  :

# aws s3 ls ... --profile ops-readonly would also work: --profile skips the env keys
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN

aws configure list
# access_key : ****************MPLE : assume-role :
aws s3 ls s3://ledger-invoices/2026/11/   # MFA prompt, then #8 AssumeRole and #9
STEP 03 / 04

A PROFILE THAT FETCHES ITS OWN KEYS

~/.aws/config
[profile ops-readonly]role_arn = arn:aws:iam::111122223333:role/ops-readonlysource_profile = defaultmfa_serial = arn:aws:iam::111122223333:mfa/new-engineerregion = us-east-1
no key or secret in this section
  1. #8sts:AssumeRole✓ ok · expires 11:27:4010:27:40AKIA
  2. #9s3:ListObjectsV2✓ ok10:27:41ASIA + token
Σ requests the laptop sent9
~/.aws/cli/cache/<sha1>.json
"Credentials": { "AccessKeyId": "ASIAIOSFODNN7EXAMPLE", "SecretAccessKey": "wJalrXUtnFEMI/…", "SessionToken": "IQoJb3JpZ2luX2VjE…", "Expiration": "2026-11-09T11:27:40+00:00"}
the response to #8, reused until it expires
Fig 3.3The accent role_arn line turns one aws s3 ls into two requests: #8 to STS and #9 to S3.

Unset the three variables, and profile ops-readonly takes over. Its section in ~/.aws/config holds no key at all: role_arn names the role, source_profile = default names the key that may assume it, and mfa_serial names the device to ask a code for.

At 10:27:40 the CLI prompts for a code and calls AssumeRole itself as request #8, signed with the default profile’s AKIA key. It writes the response to ~/.aws/cli/cache/ as JSON, then signs #9, the listing, with the new ASIA key. When those credentials expire at 11:27:40, the next command calls AssumeRole again, so there is no exported copy to go stale.

This chapter adds 3 requests, #7 to #9, and Σ requests the laptop sent is now 9.

STEP 04 / 04

SERVERS NEVER GET A KEY FILE

EC2 instance
chain row that answers
row 10 · instance profile
where the keys come from
instance metadata service, 169.254.169.254
key prefix
ASIA
ECS task
chain row that answers
row 9 · container credentials
where the keys come from
task role, from the container credentials endpoint
key prefix
ASIA
EKS pod
chain row that answers
row 4 · web identity, or row 9
where the keys come from
a service account token, or the EKS Pod Identity agent at 169.254.170.23
key prefix
ASIA
Lambda function
chain row that answers
row 2 · environment variables
where the keys come from
execution role, as AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN
key prefix
ASIA
Fig 3.4Each accent line is the chain row one kind of AWS compute answers from. Every row hands out ASIA keys.

On AWS compute the same chain ends in its lower rows, and every source there hands out ASIA keys. An EC2 instance gets its role’s credentials from the instance metadata service at 169.254.169.254, row 10. IMDSv2, the token-protected version of that service, is covered in the VPC tutorial.

An ECS task gets its task role from the container credentials endpoint, row 9. An EKS pod uses web identity, row 4, or EKS Pod Identity, whose agent on the node at 169.254.170.23 also answers through row 9. Lambda sets AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN as environment variables, row 2.

The same code therefore runs on the laptop and on the server without a change. Only the row that answers is different.

WHAT YOU NOW KNOW

  • The CLI checks 10 sources in a fixed order and uses the first one that has credentials, even expired ones.
  • AWS_PROFILE loses to exported keys. --profile on the command line, or unsetting the variables, is what fixed #7.
  • A role profile turned one aws s3 ls into #8 (AssumeRole) and #9, so the count is now 9.
deep dive: credential_process, export-credentials, and how SDK chains differ

credential_process in a profile names a program that prints credentials as JSON. Password managers and tools such as aws-vault use it to keep the long-term key out of ~/.aws/credentials, often handing the CLI short-lived session credentials instead.

aws configure export-credentials --format env prints whatever the chain resolved, as export lines. It is the safe way to hand credentials to a tool that only reads environment variables, because it includes the session token and the current, renewed values.

SDK chains are close to the CLI’s but not identical. boto3, the JavaScript SDK v3 and the Java SDK v2 differ in where they place the IAM Identity Center and process sources, so check the SDK’s own reference before relying on an order. The Region has a separate lookup: AWS_REGION or AWS_DEFAULT_REGION, then region = in the profile.

Inside a container on EC2, an IMDSv2 hop limit of 1 stops the metadata response one network hop short, so row 10 returns nothing and the chain ends empty. The VPC tutorial covers the setting.

ERRORS THIS CHAPTER EXPLAINS

  • Unable to locate credentials → no source in the chain returned any → run aws configure sso, or name a profile that exists
  • The config profile (x) could not be found → a typo in AWS_PROFILE or --profile → check the section names in ~/.aws/config
  • ExpiredToken while the same command works with --profile → stale exported variables answered first → unset AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN

Sign in once, get the same three strings

https://my-sso-portal.awsapps.com/start replaces the access key in your onboarding doc. IAM Identity Center (formerly AWS SSO) still ends in the same place: every request to an AWS service is signed with SigV4, using a key ID and a secret. What it changes is where that key comes from and how long it lives. This chapter takes Σ requests the laptop sent from 9 to 15.

STEP 01 / 05

A NEW NAME FOR THE SAME SERVICE

oncall@example.comIdentity Center user · not an IAM user · no access key
DatabaseOpspermission set · policies + session duration 4 h
ledger-prod · 111122223333the AWS account it is assigned to
AWSReservedSSO_DatabaseOps_a1b2c3d4e5f60718IAM role · your requests to ledger-prod are made as this role
Fig 4.1Follow one assignment down: the accent permission set turns into a role inside ledger-prod, and that role is what your requests use.

AWS Single Sign-On was renamed on 26 July 2022. The CLI kept the old name: its commands are still aws sso and its config keys still start with sso_. Identity Center users live in its own directory or in an external one such as Okta or Microsoft Entra ID. None of them is an IAM user, so none of them has an access key.

An administrator assigns a to a user for one account, and Identity Center creates a matching IAM role in that account, named AWSReservedSSO_DatabaseOps_ plus a random suffix. When you sign in and pick DatabaseOps, every request you make to ledger-prod is made as that role.

STEP 02 / 05

TWO REQUESTS WITH NO AWS SIGNATURE

  1. AWS CLI on the laptoplocalhost listener
  2. browsersign-in page
  3. IAM Identity Centeroidc.us-east-1
RegisterClient (#10): the CLI registers itself as an OAuth client
  1. 1CLI → Identity CenterRegisterClient #10 · returns clientId and clientSecret
  2. 2CLI → browseropens the sign-in URL with a PKCE challenge
  3. 3browser → Identity Centeryou sign in and pass MFA (browser traffic, not counted)
  4. 4browser → CLI on localhostan authorization code
  5. 5CLI → Identity CenterCreateToken #11 · code + PKCE verifier · returns the access token
~/.aws/sso/cache/<sha1>.json
"accessToken": "aoaAAAAAG…EXAMPLE""refreshToken": "Atrt|IwEB…EXAMPLE""clientId": "_Z4bW…EXAMPLE""clientSecret": "eyJr…EXAMPLE"
written after hop 5 · a bearer token, not an AWS key
Fig 4.2Watch the dots move between the CLI, the browser and the sign-in service. None of them carries an Authorization header.

At 11:00:05 you run aws configure sso. The person signing in has no AWS key yet, which is the reason for signing in, so the first requests can’t be signed with SigV4. RegisterClient (#10) registers the CLI as an OAuth client (OAuth is the web’s standard for letting an app act for a signed-in user) and returns a clientId and a clientSecret.

The CLI then opens your browser at the sign-in page. Since CLI 2.22.0 the default flow is PKCE: the CLI sends a hash of a random secret first and reveals the secret itself at the end, which proves the same program started and finished the sign-in. After sign-in and MFA, the browser hands an authorization code to a listener the CLI runs on localhost.

CreateToken (#11) trades that code and the PKCE secret for an access token and a refresh token. botocore’s service model marks RegisterClient and CreateToken authtype: none. With --use-device-code, the CLI uses the older device-code flow instead: it prints a short code, and you type it into a browser anywhere.

STEP 03 / 05

A TOKEN THAT BUYS KEYS

  1. #10sso-oidc:RegisterClient✓ ok11:00:05none
  2. #11sso-oidc:CreateToken✓ ok11:00:38none
  3. #12sso:ListAccounts✓ ok11:00:39bearerheader x-amz-sso_bearer_token: aoaAAAAAG…
  4. #13sso:ListAccountRoles✓ ok11:00:44bearer
  5. #14sso:GetRoleCredentials✓ ok · expires 15:01:1011:01:10bearer↓ returns the role credentials below
  6. #15sts:GetCallerIdentity✓ ok11:01:11ASIA + tokensigned with SigV4, using the key #14 returned
Σ requests the laptop sent15
what GetRoleCredentials (#14) returned
ASIAIOSFODNN7EXAMPLEaccessKeyId · 20 chars
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYsecretAccessKey · 40 chars
IQoJb3JpZ2luX2VjE…EXAMPLETOKEN==sessionToken · length varies
2026-11-09T15:01:10Zexpiration · in 4 h
Fig 4.3Rows #12 to #14 carry the accent bearer header. Row #15 is SigV4 again, signed with the key #14 returned.

~/.aws/sso/cache/ now holds a JSON file with accessToken, expiresAt and refreshToken. The access token is a : AWS accepts it from whoever sends it, with no signature. The CLI uses it to list your accounts (#12, ListAccounts) and your roles in ledger-prod (#13, ListAccountRoles), sending it in an x-amz-sso_bearer_token header.

At 11:01:10, aws sts get-caller-identity --profile ledger-prod needs a key, so the CLI first calls GetRoleCredentials (#14) with the bearer token. It gets back accessKeyId (ASIA…), secretAccessKey, sessionToken and expiration. Then it signs GetCallerIdentity (#15) with SigV4, and the Arn that comes back ends in oncall@example.com: your email address is the role session name.

Signing in and asking one question took 6 requests, and Σ requests the laptop sent is now 15.

REQUESTS #10 TO #15: SIGNING IN, THEN ONE QUESTION
aws configure sso
# SSO session name: ledger
# SSO start URL: https://my-sso-portal.awsapps.com/start
# SSO region: us-east-1
#   (the browser opens; sign in, MFA, approve: requests #10 to #13)
# Profile name: ledger-prod

# ~/.aws/config now holds:
#   [profile ledger-prod]
#   sso_session = ledger
#   sso_account_id = 111122223333
#   sso_role_name = DatabaseOps
#
#   [sso-session ledger]
#   sso_start_url = https://my-sso-portal.awsapps.com/start
#   sso_region = us-east-1

aws sts get-caller-identity --profile ledger-prod   # #14 GetRoleCredentials, then #15
# "Arn": "arn:aws:sts::111122223333:assumed-role/AWSReservedSSO_DatabaseOps_a1b2c3d4e5f60718/oncall@example.com"
STEP 04 / 05

TWO CLOCKS, NOT ONE

sign-in
expiry
11:0013:0015:0017:0019:00
access portal session · 8 h · until about 19:00, then sign in again
role credentials from #14 · 4 h · until 15:01:10
next batch · fetched on first use after 15:01, no prompt
portal session: 15 min to 90 days, default 8 h · permission set: 1 to 12 h, default 1 h
Fig 4.4The accent bar is the portal session. The bars under it are role credentials the CLI fetches inside it, 4 hours each.

Two durations apply, and an administrator sets them in different places. The access portal session decides how often you sign in through the browser: 8 hours by default, configurable from 15 minutes to 90 days. The permission set’s session duration decides how long each batch of role credentials lasts: 1 hour by default, up to 12. DatabaseOps is set to 4 hours.

While the portal session is valid, the CLI fetches new role credentials without asking you anything. You signed in at 11:00, and the credentials from #14 last until 15:01:10. That covers both Session Manager commands in chapters 06 and 07, so neither of them needs another GetRoleCredentials.

STEP 05 / 05

KEYS AND SSO, SIDE BY SIDE

stored on the laptopaccess key (IAM user)AKIA key ID + secret in ~/.aws/credentialsIAM Identity Centera token file in ~/.aws/sso/cache + cached ASIA credentials
lifetimeaccess key (IAM user)until someone deletes itIAM Identity Center4 h role credentials · 8 h portal session
MFA checkedaccess key (IAM user)only on AssumeRole with --token-codeIAM Identity Centerat every sign-in, in the browser
offboardingaccess key (IAM user)find and delete keys in every accountIAM Identity Centerdisable one user (with a catch, 05.3)
name in CloudTrailaccess key (IAM user)user/new-engineerIAM Identity Centeroncall@example.com, as the role session name
rotationaccess key (IAM user)by hand, using the second key slotIAM Identity Centernone needed: every key expires
request to the serviceaccess key (IAM user)SigV4, signed with the keyIAM Identity CenterSigV4, signed with the key
Fig 4.5Every row differs except the accent one: how the request that reaches the AWS service is signed.

An access key and an Identity Center sign-in differ in what sits on the laptop and how long it lasts. The key is two strings in ~/.aws/credentials that work until someone deletes them. Identity Center leaves a token file and cached ASIA credentials that expire within hours. MFA happens at every sign-in, and CloudTrail names you by email instead of by IAM user.

Offboarding differs too. A departing engineer’s access keys have to be found and deleted in every account that has them, while an Identity Center user is disabled once. That has one catch, covered in chapter 05: role sessions already issued keep working. The last row is identical on both sides, because the request that reaches S3 or STS is ordinary SigV4.

WHAT YOU NOW KNOW

  • IAM Identity Center signs you in through a browser and gives the CLI a bearer token, not an AWS key.
  • That token buys ASIA credentials for a permission set’s role, and from there every request is ordinary SigV4.
  • Signing in and fetching credentials took 6 requests, #10 to #15, and 2 of them carried no credential at all.
deep dive: device codes, external identity providers, and logging out

The device-code flow, aws sso login --use-device-code, prints a URL and a short code instead of opening a browser. It works on a machine with no browser, because you can approve the code on a phone. That same property makes it the riskier flow: an attacker can start a device-code sign-in and persuade you to approve their code, which hands them your token.

The [sso-session] section is the current config format. The legacy format, with sso_start_url inside the profile itself, still works but can’t refresh its token, so you sign in again whenever it expires. aws sso logout deletes the cached tokens from the laptop and ends the sign-in session; role credentials already fetched keep working until they expire.

With an external identity provider, sign-in goes through that provider over SAML, and users are copied into Identity Center with SCIM. Identity Center still issues the token the CLI uses. Opening the console from the access portal uses the same AWSReservedSSO_ role. Those roles can’t be edited in IAM: change the permission set, and Identity Center updates the role in every account it is assigned to.

ERRORS THIS CHAPTER EXPLAINS

  • Error loading SSO Token: Token for ledger does not exist → nobody has signed in to the ledger sso-session on this laptop; run aws sso login --profile ledger-prod
  • The SSO session associated with this profile has expired or is otherwise invalid → the portal session ended, or its token was revoked; run aws sso login again
  • ForbiddenException “No access” on GetRoleCredentials → no assignment gives you that role in that account; check the profile’s sso_account_id and sso_role_name, or ask for the assignment

Deleting a key, and the credentials deletion misses

The access key from chapter 00 has signed zero requests since 10:27:40, when it signed AssumeRole #8, and everything since 11:01 has used Identity Center. So at 13:00 you delete the key, and you use the key itself to do it. That takes 2 requests, and it is the easy half: temporary credentials can’t be deleted at all. Σ requests the laptop sent goes from 15 to 17.

STEP 01 / 04

THE LAST REQUEST A KEY EVER SIGNS

access key of new-engineer · deleted by #16 at 13:00:02
AKIAIOSFODNN7EXAMPLEaccess key ID · 20 chars · sent with every request
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYsecret access key · 40 chars · shown once, never sent
noneExpiration
  1. #16$ aws iam delete-access-key …iam:DeleteAccessKey✓ ok13:00:02AKIAsigned by AKIAIOSFODNN7EXAMPLE, the key it deletes
  2. #17$ aws sts get-caller-identitysts:GetCallerIdentity✗ InvalidClientTokenId13:00:30AKIAthe same key, 28 seconds later: “The security token included in the request is invalid.”
Σ requests the laptop sent17
Fig 5.1Row #16 is signed by the key it deletes. Row #17, 28 seconds later, is the same key refused.

At 13:00:02 you run aws iam delete-access-key --user-name new-engineer --access-key-id AKIAIOSFODNN7EXAMPLE with the default profile. Request #16 is signed by the key it deletes, and IAM accepts it, because the key still existed when the request arrived. At 13:00:30, aws sts get-caller-identity (#17) is refused with InvalidClientTokenId: “The security token included in the request is invalid.”

The message says “security token” about a key that has no token: AWS uses that wording for any key ID it doesn’t recognize. In an incident, mark the key Inactive with aws iam update-access-key first. An inactive key is refused too, and unlike a deleted one it can be switched back on. Σ requests the laptop sent is now 17.

STEP 02 / 04

TEN SECONDS ON A PUBLIC REPOSITORY

seconds after 18:50:00 UTC, 2025-12-19
1
2
3
0 s10 s20 s
  1. 118:50:05access key pushed to a public GitHub repository
  2. 218:50:15AWS attaches AWSCompromisedKeyQuarantineV3 to the user (AttachUserPolicy)
  3. 318:50:16GitHub’s secret-scanning alert
key status after all three · still Active
Fig 5.2Three timestamps within 11 seconds: the push, the accent AttachUserPolicy made by AWS, and GitHub’s alert.

In a controlled test on 19 December 2025, Palo Alto Networks Unit 42 pushed a working access key to a public GitHub repository at 18:50:05 UTC. At 18:50:15, ten seconds later, AWS attached the managed policy AWSCompromisedKeyQuarantineV3 to the key’s IAM user. GitHub’s own alert arrived at 18:50:16.

The quarantine policy denies a list of commonly abused actions, such as creating users and access keys or launching instances. It doesn’t deactivate the key, so every action the list leaves out still works. CloudTrail records the attachment as an AttachUserPolicy call, as if a user had made it.

STEP 03 / 04

ISSUED CREDENTIALS CANNOT BE RECALLED

inline policy on the role · AWSRevokeOlderSessions
Effect
Deny
Action
*
Resource
*
Condition
DateLessThan aws:TokenIssueTime 2026-11-09T13:05:00Z
long-term keys carry no issue time, so this policy doesn’t touch them
cut 13:05
09:0011:0013:0015:00
session issued 09:20 · ✗ refused from 13:05
session issued 11:01 · ✗ refused from 13:05
session issued 13:10 · ✓ issued after the cut, allowed
Fig 5.3The accent Condition compares each request’s aws:TokenIssueTime with one instant and refuses everything issued before it.

No API deletes a temporary credential. Deleting an AKIA key doesn’t end the sessions already created from it either, so credentials it got from GetSessionToken keep working for up to 36 hours. What you can do is refuse them: a Deny on every action when aws:TokenIssueTime is earlier than a chosen instant refuses every session issued before that instant.

The IAM console’s “Revoke active sessions” button on a role does exactly this. It attaches an inline policy named AWSRevokeOlderSessions holding the current time, and it doesn’t affect long-term keys, which carry no issue time. Disabling an Identity Center user blocks new sign-ins, but the role sessions that user already holds keep working for up to 12 hours.

REFUSE EVERY SESSION ISSUED BEFORE 13:05
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "DateLessThan": { "aws:TokenIssueTime": "2026-11-09T13:05:00Z" }
      }
    }
  ]
}
STEP 04 / 04

THE SHORTER THE LIFETIME, THE SMALLER THE LEAK

1 min1 h12 h36 h90 days
access key (AKIA) · until someone deletes it
→ no end
AssumeRole credentials · 1 h here, 12 h at most
GetSessionToken credentials · 12 h by default, 36 h at most
SSO role credentials · 4 h here, 12 h at most
SSO token file · 8 h portal session here, 90 days at most
Session Manager TokenValue · long enough to open one connection
solid: today’s lifetime · dashed: the maximum
Fig 5.4All bars share one log-scale ruler. The accent bar is the only credential with no right-hand end.

Rank each credential the laptop held today by how long it would work if it leaked. The AKIA key had no end of its own and worked until #16 deleted it. AssumeRole credentials lasted 1 hour here and can last 12. GetSessionToken credentials last 12 hours by default and 36 at most. The SSO role credentials last 4 hours.

The SSO token file is the long one. While the portal session lasts, 8 hours here and up to 90 days if an administrator allows it, a copy of ~/.aws/sso/cache can buy fresh role credentials on any machine. A Session Manager TokenValue is the short one: it is valid only long enough to open its connection.

WHAT YOU NOW KNOW

  • A deleted access key fails on its very next request: #16 deleted it, and #17 was refused 28 seconds later.
  • Temporary credentials can’t be deleted. A Deny on aws:TokenIssueTime refuses them until they expire.
  • Disabling an Identity Center user stops new sign-ins, but role sessions they already have keep working for up to 12 hours.
deep dive: a leak runbook, and banning access keys across an organization

An incident order that works: deactivate the key, then search CloudTrail for its userIdentity.accessKeyId to see everything it did. Revoke the sessions issued from it, and only then delete it. AWS also emails the account owner and opens an AWS Health event when it finds an exposed key.

Organizations that have moved to Identity Center often deny iam:CreateAccessKey with a service control policy, so new long-term keys can’t appear in any member account. IAM Access Analyzer’s unused-access findings and the IAM credential report list the keys nobody uses, which are the cheapest ones to delete.

For an Identity Center user, AWS’s advice is a Deny on that user’s identitystore:UserId in the permission sets they had, kept in place for at least 12 hours, the longest a role session can last. Disabling the user alone stops new sign-ins and nothing else.

ERRORS THIS CHAPTER EXPLAINS

  • InvalidClientTokenId “The security token included in the request is invalid.” → the key was deleted or deactivated, or never existed; get working credentials
  • AccessDenied “… with an explicit deny in an identity-based policy” after a revoke → the session was issued before the cut; get new credentials, which carry a later issue time

A shell on a machine with no open ports

Port 22 is closed on db-bastion, and so is every other port: its security group has 0 inbound rules, and the instance has no public IP address. Two requests from the laptop still open a shell on it, with sudo. They take Σ requests the laptop sent from 17 to 19.

STEP 01 / 04

THE INSTANCE CALLS OUT FIRST

db-bastioni-1234567890abcdef0 · private subnet · public IP: nonelistens on: nothing
its security group
inbound
0 rules
outbound
443 to the endpoints · 5432 to ledger-db
  • ssmregistration, StartSession · interface endpointlistens on: 443
  • ssmmessagessession data channels · interface endpointlistens on: 443
  • ec2messagesolder agents only · interface endpointlistens on: 443
Fig 6.1Every accent arrow starts at the instance and points out. The security group’s inbound column below it is empty.

, part of AWS Systems Manager, reaches an instance through a connection the instance opened itself. , preinstalled on Amazon Linux 2023 and many other AWS AMIs, uses the instance profile’s role, with the managed policy AmazonSSMManagedInstanceCore, to open outbound HTTPS on port 443 to the ssm and ssmmessages endpoints. The role’s keys come from row 10 of chapter 03’s chain. Agents from version 3.3.40.0 prefer ssmmessages over the older ec2messages.

db-bastion’s subnet has no NAT gateway, so it reaches those services through interface endpoints, one per service. Three endpoints in 2 Availability Zones at $0.01 an hour each, for 730 hours, cost $43.80 a month. Default Host Management Configuration can supply the role without an instance profile, but only on instances that require IMDSv2.

STEP 02 / 04

ONE SIGNED CALL, THEN A WEBSOCKET

  1. ssmlistens on 443
  2. laptoplistens on nothing
  3. ssmmessageslistens on 443
  4. db-bastionlistens on nothing
SSM Agent on db-bastion has kept a connection out to ssmmessages open since boot
  1. db-bastion → ssmmessagescontrol channel, opened at boot
  2. laptop → ssmStartSession #18, SigV4
  3. db-bastion → ssmmessagesdata channel for this session
  4. laptop → ssmmessagesWebSocket #19, TokenValue
both ends connect out; nothing connects in
Fig 6.2Watch the order: the instance’s connection is already open before yours, and your keystrokes travel over two outbound connections.

At 14:00:00, aws ssm start-session --target i-1234567890abcdef0 --profile ledger-prod sends StartSession (#18), signed with SigV4 using the SSO role credentials from chapter 04. The response has three fields: a SessionId, a StreamUrl of the form wss://ssmmessages.us-east-1.amazonaws.com/v1/data-channel/…, and a TokenValue of at most 300 characters.

The Session Manager plugin, a separate program the CLI starts, opens that WebSocket (#19) and presents the TokenValue, which is good only for opening this one connection. SSM Agent on db-bastion opens its own outbound data channel to ssmmessages, and the service joins the two connections. Σ requests the laptop sent is now 19.

STEP 03 / 04

TWO THINGS CALLED A SESSION

session token · from #4, #14
issued by
STS (AssumeRole) or Identity Center (GetRoleCredentials)
used for
signing requests, in X-Amz-Security-Token
lifetime
until its Expiration: 1 h and 4 h today
keep it secret?
yes: it is part of a credential
SessionId · from #18
issued by
Session Manager (StartSession)
used for
naming one connection: describe-sessions, terminate-session
lifetime
the connection it names
keep it secret?
no: it signs nothing
TokenValue · from #18
issued by
Session Manager (StartSession)
used for
opening one WebSocket, #19
lifetime
until that WebSocket opens
keep it secret?
until it has been used
Fig 6.3Three strings, three jobs: only the accent card can sign an AWS request.

A session token, from chapter 02, is part of a credential. It travels with an ASIA key, is signed into every request, and expires with that key. Requests #4 and #14 each returned one, and whoever copies it can call AWS as you until its Expiration.

A Session Manager SessionId names one shell connection. aws ssm describe-sessions lists it, aws ssm terminate-session ends it, and it begins with the caller’s name: the API reference’s example is John-Doe-0dc5b7af96EXAMPLE. It signs nothing, so printing it in a log does no harm.

The TokenValue that came back with it in #18 opens one WebSocket and is useless afterwards. When an onboarding doc asks for a “session ID” next to an access key, it means the session token, the only one of the three that can sign an AWS request.

STEP 04 / 04

IAM DECIDES WHO GETS A SHELL

DatabaseOps · one statement
Effect
Allow
Action
ssm:StartSession
Resource
arn:aws:ec2:us-east-1:111122223333:instance/*
Condition
StringEquals ssm:resourceTag/team = ledger-ops
a second statement names the session documents
db-bastion
tag
team = ledger-ops
OS user for the shell
ssm-user, in /etc/sudoers.d
SSH keys on disk
none
Linux passwords set
none
Fig 6.4The accent Condition is the whole access list: any instance tagged team=ledger-ops, and no SSH key anywhere.

Session Manager checks one IAM permission instead of a Linux account: ssm:StartSession on the instance and on the session document. Here the instance must carry the tag team=ledger-ops, and the DatabaseOps permission set grants it. There is no authorized_keys file and no Linux password anywhere in the path.

The shell runs as ssm-user, an account SSM Agent creates and adds to sudoers. Run As can map IAM principals to other OS users, though never to root. CloudTrail records each StartSession under your SSO identity, and the shell’s input and output can be logged to S3 or CloudWatch Logs.

WHO MAY OPEN A SESSION, AND WITH WHICH DOCUMENTS
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ssm:StartSession",
      "Resource": "arn:aws:ec2:us-east-1:111122223333:instance/*",
      "Condition": {
        "StringEquals": { "ssm:resourceTag/team": "ledger-ops" }
      }
    },
    {
      "Effect": "Allow",
      "Action": "ssm:StartSession",
      "Resource": [
        "arn:aws:ssm:us-east-1:111122223333:document/SSM-SessionManagerRunShell",
        "arn:aws:ssm:us-east-1::document/AWS-StartPortForwardingSessionToRemoteHost"
      ]
    }
  ]
}

WHAT YOU NOW KNOW

  • SSM Agent opens outbound HTTPS to ssm and ssmmessages, so the instance needs no inbound rule and no public IP.
  • StartSession (#18) is SigV4; the shell is a WebSocket (#19), opened with a short-lived TokenValue. The count is now 19.
  • An IAM policy on ssm:StartSession decides who gets a shell, and CloudTrail records each one by name.
deep dive: hybrid nodes, encrypted sessions, and Run As

Servers outside AWS can run SSM Agent too. They register with an activation code and appear as hybrid nodes with IDs starting mi-. Session Manager is free for EC2 instances, and since 30 September 2026 it costs $0.05 per session on hybrid nodes.

Session data is already inside TLS, and Session Manager preferences can add a second layer with a KMS key, which adds KMS calls to every session (see the AWS KMS tutorial). The interface endpoints for ssm and ssmmessages work through private DNS, which the AWS VPC tutorial’s chapter 04 explains.

Run As picks the OS user from the IAM principal’s SSMSessionRunAs tag. ECS Exec uses the same agent inside a container, so a Fargate task gets a shell the same way. When the agent itself is broken, the EC2 serial console is the fallback.

ERRORS THIS CHAPTER EXPLAINS

  • TargetNotConnected → the instance isn’t registered with Systems Manager: no role, no route to ssmmessages, a stopped agent, or another account or Region → fix the role or the route, then check that Fleet Manager lists the instance
  • SessionManagerPlugin is not found. → the CLI found no plugin on the laptop → install the Session Manager plugin
  • AccessDeniedException on StartSession → the instance’s tag or the session document isn’t in your policy → add it to the permission set

The bastion stays. Port 22 goes.

ledger-db.cexample1234.us-east-1.rds.amazonaws.com has no agent, no shell and no public address: it is a managed PostgreSQL database in a private subnet. Something inside the VPC has to carry bytes from the laptop to its port 5432, and this chapter asks whether that hop needs SSH. Two more requests open the path, and Σ requests the laptop sent ends at 21.

STEP 01 / 05

THE OLD DESIGN NEEDED AN OPEN PORT

  • public IPv4SSH bastion · yes · $3.65 a monthSession Manager · none
  • inbound ruleSSH bastion · 22 from 203.0.113.0/24Session Manager · none
  • credential on the laptopSSH bastion · SSH private key · no expirySession Manager · SSO role credentials · 4 h
  • who can revoke itSSH bastion · whoever can edit authorized_keysSession Manager · IAM: one policy change
  • CloudTrail recordSSH bastion · noneSession Manager · StartSession, by name
Fig 7.1Count the accent rows: a public address, an inbound rule and a key file, which the new design drops.

A traditional sits in a public subnet with a public IPv4 address, which costs $0.005 an hour, or $3.65 a month. Its security group allows inbound port 22 from an office range, and every engineer’s public key goes into authorized_keys. To reach the database you run ssh -L 5432:ledger-db…:5432 ec2-user@bastion.

Each of those SSH keys is a long-term credential, like an AKIA key, with three differences. IAM can’t see it, CloudTrail doesn’t log its use, and removing it means editing a file on the instance. Offboarding one engineer means finding their line in authorized_keys on every bastion they could reach.

STEP 02 / 05

A TUNNEL WITH NO LISTENING PORT ON THE INSTANCE

  1. psqllistens on nothing
  2. pluginlistens on localhost:5432
  3. ssmmessageslistens on 443
  4. SSM Agent on db-bastionlistens on nothing
  5. ledger-dblistens on 5432
psql sends SELECT to localhost:5432, where the plugin is listening
  1. psql → localhost:5432TCP on the laptop
  2. plugin → ssmmessages:443WebSocket #21
  3. SSM Agent → ssmmessages:443data channel, outbound
  4. SSM Agent → ledger-db:5432TCP inside the VPC
Fig 7.2Follow one query from psql to ledger-db and back. The accent hop is the one connection the database ever sees.

through Session Manager uses a different session document, not a different service. At 14:10:00, aws ssm start-session with --document-name AWS-StartPortForwardingSessionToRemoteHost, host set to ledger-db’s endpoint and both port numbers set to 5432, sends StartSession (#20). The plugin then opens WebSocket #21, as in chapter 06.

The plugin listens on localhost:5432 on the laptop. For each connection psql makes, SSM Agent on db-bastion opens a TCP connection to ledger-db:5432, which ledger-db’s security group allows from db-bastion’s. The database needs no agent; the feature needs SSM Agent 3.1.1374.0 or later, from May 2022. Σ requests the laptop sent is now 21.

REQUESTS #20 AND #21: THE TUNNEL TO LEDGER-DB
aws ssm start-session \
  --target i-1234567890abcdef0 \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["ledger-db.cexample1234.us-east-1.rds.amazonaws.com"],"portNumber":["5432"],"localPortNumber":["5432"]}' \
  --profile ledger-prod             # #20 StartSession, then #21 the WebSocket

# Starting session with SessionId: <session-id>
# Port 5432 opened for sessionId <session-id>.
# Waiting for connections...
# Connection accepted for session [<session-id>]     (when psql connects)
STEP 03 / 05

A DATABASE PASSWORD THAT IS A SIGNATURE

PGPASSWORD · printed by aws rds generate-db-auth-token
  1. ledger-db.cexample1234.us-east-1.rds.amazonaws.com:5432/host and port, signed
  2. ?Action=connectthe only action this token allows
  3. &DBUser=ops_readonlythe database user, signed
  4. &X-Amz-Algorithm=AWS4-HMAC-SHA256SigV4
  5. &X-Amz-Credential=ASIAIOSFODNN7EXAMPLE/20261109/us-east-1/rds-db/aws4_requesttemporary key, scoped to rds-db
  6. &X-Amz-Date=20261109T141005Zwhen it was signed
  7. &X-Amz-Expires=900seconds it works: 15 minutes
  8. &X-Amz-Security-Token=IQoJb3JpZ2luX2VjE…the session token rides along
  9. &X-Amz-SignedHeaders=hostheaders covered by the signature
  10. &X-Amz-Signature=5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d764 hex chars, computed on the laptop
Fig 7.3The accent part is an X-Amz-Signature, as in Fig 1.5; the database user and host are signed into it.

RDS IAM authentication replaces the database password with a SigV4 signature. aws rds generate-db-auth-token with the hostname, port 5432 and username ops_readonly builds a presigned request on the laptop, the same construction as chapter 01’s presigned URL, and sends nothing. The token it prints is valid for 15 minutes.

psql sends that token as the PostgreSQL password over the tunnel, with sslmode=require, because the server’s certificate names ledger-db and not localhost. The database user has the rds_iam role, and DatabaseOps grants rds-db:connect. No database password is stored anywhere, and Σ requests the laptop sent stays at 21.

A 15-MINUTE PASSWORD, COMPUTED ON THE LAPTOP
TOKEN=$(aws rds generate-db-auth-token \
  --hostname ledger-db.cexample1234.us-east-1.rds.amazonaws.com \
  --port 5432 \
  --username ops_readonly \
  --region us-east-1 \
  --profile ledger-prod)        # computed on the laptop, valid 15 minutes

# sslmode=require, not verify-full: the certificate names ledger-db, not localhost
PGPASSWORD="$TOKEN" psql \
  "host=localhost port=5432 dbname=ledger user=ops_readonly sslmode=require"
STEP 04 / 05

WHEN YOU STILL WANT SSH

  • shell (06)inbound port on the instance · noneSSH key needed · nosession data logged · yes, every keystroke
  • port forwarding (07.2)inbound port on the instance · noneSSH key needed · nosession data logged · no
  • SSH over Session Managerinbound port on the instance · none in the security groupSSH key needed · yessession data logged · no
  • EC2 Instance Connect Endpointinbound port on the instance · 22, from the endpoint onlySSH key needed · yessession data logged · no
Fig 7.4The accent column is what Session Manager can record: everything for a shell, nothing for the other three rows.

SSH over Session Manager runs the SSH protocol inside a session, for tools such as scp, rsync and remote IDEs. A ProxyCommand in ~/.ssh/config runs aws ssm start-session with the document AWS-StartSSHSession. sshd must run on the instance, but inbound port 22 stays closed. You still need an SSH key, from a key pair or from EC2 Instance Connect, which pushes a public key that lasts 60 seconds.

The cost is visibility. Session Manager can record every keystroke of a shell session, but it carries SSH and port-forwarding sessions as bytes it can’t read, so it logs nothing inside them. EC2 Instance Connect Endpoint, launched on 13 June 2023, is the other option: real SSH from an endpoint inside the VPC, with no agent on the instance.

SCP THROUGH SESSION MANAGER
cat >> ~/.ssh/config <<'EOF'
Host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
    User ec2-user
EOF

# inbound port 22 stays closed; sshd and an SSH key are still needed
scp report.csv i-1234567890abcdef0:/tmp/
STEP 05 / 05

WHICH PATH FOR WHICH JOB

  • shell on an instance with SSM Agentaws ssm start-sessionno bastion
  • database or other private endpointport forward via a private instancea bastion with no SSH
  • copy files with scp or rsyncSSH over Session Managersshd and a key; not logged
  • instance with no agentEC2 Instance Connect Endpointan SSH key; 22 open to the endpoint
Fig 7.5Start from the job in the left column. The accent path is the one this chapter used to reach ledger-db.

Session Manager alone is enough when the target runs SSM Agent: start-session reaches it directly, and no bastion is needed. A target that can’t run an agent, such as RDS, ElastiCache or a network appliance, needs a hop inside the VPC. Session Manager turns that hop into a private instance with no SSH, so the bastion stays and port 22 goes.

To copy files, use SSH over Session Manager and accept that the session isn’t logged. For an instance with no agent at all, use EC2 Instance Connect Endpoint. In every row, no instance in ledger-prod needs a public IP address or an inbound rule from the internet.

WHAT YOU NOW KNOW

  • Session Manager alone is enough when the target runs SSM Agent. A database needs a forwarding hop inside the VPC.
  • That hop, db-bastion, has no public IP and no inbound rule; StartSession #20 and WebSocket #21 opened the tunnel.
  • The database password was a 15-minute SigV4 token computed on the laptop, so Σ requests the laptop sent ends at 21.
deep dive: tunnel-only permissions, ECS Exec, and Instance Connect Endpoint

A permission set can allow tunnels and refuse shells. Limit the document in the StartSession statement to arn:aws:ssm:us-east-1::document/AWS-StartPortForwardingSessionToRemoteHost and add the condition ssm:SessionDocumentAccessCheck set to true. Without that condition, a request that names no document still gets the default shell. The sibling document AWS-StartPortForwardingSession forwards to a port on the instance itself instead of a remote host.

Remote-host forwarding works the same way for ElastiCache, OpenSearch or any private HTTP endpoint the bastion can reach. A forwarding-only bastion copies bytes and runs nothing else, so the smallest instance size that runs SSM Agent is enough. ECS Exec gives the same shell inside a Fargate task, with no instance at all.

EC2 Instance Connect Endpoint caps each TCP connection at 1 hour, so a long psql session through it drops on the hour. A Session Manager session ends instead after the idle timeout set in Session Manager preferences, 20 minutes by default.

ERRORS THIS CHAPTER EXPLAINS

  • Connection to destination port failed, check SSM Agent logs. → the agent on db-bastion can’t reach host:port → allow db-bastion’s security group into ledger-db’s on 5432
  • PAM authentication failed for user "ops_readonly" → the token is more than 15 minutes old, or the user lacks rds_iam → generate a new token, or grant rds_iam to the user
  • TargetNotConnected → the agent isn’t registered → see chapter 06

Twenty-one requests, and how each one proved who sent it

One working day of one laptop, counted once more. The board sorts the 21 requests by how each one proved who sent it, how AWS answered, and what crossed the network. The proof block is the answer to “SSO or keys”: both end in the same kind of signed request.

laptop · 2026-11-09 · 09:02 → 14:10 · ledger-prod
09:0010:0011:0012:0013:0014:0015:00
one tick per request; the 3 refused requests in purple
HOW EACH REQUEST PROVED WHO SENT IT
signed with a long-term key (AKIA)7 requests
signed with a temporary key and session token (ASIA)7 requests
an SSO bearer token3 requests
no AWS credential (signing in)2 requests
a Session Manager stream token2 requests
BY OUTCOME
succeeded18 requests
ExpiredToken2 requests
InvalidClientTokenId1 request
WHAT CROSSED THE NETWORK
AWS secret access keys the laptop sent0
responses that sent a secret key to the laptop3 responses
commands that signed something and sent nothing2 commands
WHAT THE DATABASE PATH NEEDED
inbound rules on db-bastion0 rules
public IP addresses on db-bastion0
SSH keys involved0
Σ requests the laptop sent to AWS21
Read next. AWS VPC: how one request crosses a private network for the interface endpoints and IMDSv2 this tutorial leaned on.
AWS KMS: how one invoice stays encrypted for seven years for what happens when an object in ledger-invoices is actually read.
Exiting session with sessionId: <session-id>.