1 SUPPORT CHAT · 4 CALLS TO AI MODELS · 3 EU COUNTRIES · $5,386 A MONTH

AWS Bedrock and SageMaker:
how one support chat gets answered

Amazon Bedrock lets you call AI models that AWS runs for you, such as Anthropic’s Claude, and pay for each piece of text they read and write. Amazon SageMaker AI rents you machines with GPUs and runs your own model on them, behind an address your code can call.

WHAT THIS TUTORIAL COUNTS

A customer in Lyon asks an online shop’s support chat where a late parcel is. Answering takes four calls to AI models. The tutorial prices that design for a month of 300,000 chats, in US dollars.

dollars a month for 300,000 chats$5,386.41
  • Claude, writing the answers$2,739.00
    a model AWS runs for the shop, paid per token: a piece of text about four characters long
  • two GPU servers, sorting chats$2,231.02
    machines the shop rents by the hour to run its own smaller model
  • nightly summaries$371.25
    a cheaper Claude reads every chat overnight, at half price for waiting
  • private network links$35.20
    keep the calls to the models off the public internet
  • prompt logs$9.94
    a copy of every prompt and answer, kept 30 days in Frankfurt
AND FOR KEEPING THE CHATS IN THE EU
  • EU-only routing for Claude$249.00 of the $2,739.00
    Claude costs 10% more when it may only run in EU countries
shop-prod · eu-central-1 Frankfurt · AWS list prices, November 2026

This tutorial follows one chat at the online shop from the AWS VPC tutorial, which now sells in the EU. A customer in Lyon writes in French that a parcel is late. The shop’s own model sorts the message, Claude asks for the order, the shop’s code looks it up, and Claude writes the reply; overnight, another model summarizes the chat. The shop has promised its EU business customers that their chats are processed only in the EU, a requirement called . Every chapter adds what its part of the design costs to the : everything AWS charges in one 730-hour month to answer 300,000 chats this way. It ends at $5,386.41, and the largest surprise in it is not Claude.

You need to know what an HTTP request, a JSON policy and a VPC are; no machine-learning or Bedrock experience is assumed. The scenes play themselves, so watch or just scroll. are tappable. Prices are AWS list prices in Frankfurt on 14 November 2026, and which Region answered each call is one illustrative run. Each chapter’s deep dive ends with the scenario cues used by the AWS Solutions Architect Professional (SAP-C03, live from 17 November 2026) and Generative AI Developer Professional (AIP-C01) exams.

Two calls to Claude for one answer

The shop’s chat server has the customer’s message and the label its triage model gave it, and no way to write a reply. It sends the conversation to Amazon Bedrock, AWS’s service for calling models that AWS runs. At 14:00:01 the first version of the chat code calls Claude Sonnet 5.5 twice for one answer, and the monthly bill goes from $0 to $2,490.00.

STEP 01 / 06

ONE MODEL, THREE ADDRESSES

THE SHOP CALLS THIS ONE
bedrock-runtime.eu-central-1.amazonaws.com
Converse, InvokeModel, Messages, OpenAI-compatible
run byAWS (Bedrock)
EU inference profileyes
invocation logsyes
HOSTNAME
bedrock-mantle.eu-central-1.api.aws
Messages, Responses, Chat Completions
run byAWS (Bedrock)
EU inference profileno
invocation logsno
HOSTNAME
aws-external-anthropic.eu-central-1.api.aws
the full Claude API
run byAnthropic (Claude Platform on AWS)
EU inference profileno
invocation logsno
Fig 0.1Three hostnames serve Claude to AWS accounts. The accent card is the only one that offers EU routing and invocation logs.

A is a model a provider trained that a cloud service runs for many customers; Claude is Anthropic’s. Claude reaches an AWS account at three hostnames. The one AWS recommends is bedrock-runtime.eu-central-1.amazonaws.com, which serves the Converse and InvokeModel APIs plus Messages and OpenAI-compatible paths. It is the only one with cross-Region inference profiles and invocation logging.

bedrock-mantle.eu-central-1.api.aws serves the Messages and Responses APIs. AWS says both Bedrock hostnames run on the same inference engine at the same price per token. The third, aws-external-anthropic.eu-central-1.api.aws, is Claude Platform on AWS, which Anthropic runs itself; chapter 02 returns to it. The shop’s chat server calls bedrock-runtime.

STEP 02 / 06

A CONVERSE CALL IS ONE HTTPS POST

POST bedrock-runtime.eu-central-1.amazonaws.com/model/{modelId}/conversecall #2
REQUEST
  • modelIdglobal.anthropic.claude-sonnet-5-5
    which model, and where it may run
  • system4,000 tokens
    support policy and tool rules, then a cache point
  • messagesthe customer’s message
    120 tokens, in French
  • toolConfigget_order, issue_refund
  • inferenceConfigmaxTokens 1,024
RESPONSE
  • stopReasontool_use
    Claude asks for data, no answer yet
  • outputtoolUse · get_order · order_id 48213
  • usage4,000 cached · 400 in · 150 out
    tokens, which set the price
signed with SigV4 by the role shop-support-chat · IAM checks bedrock:InvokeModel
Fig 0.2The accent field, modelId, is the only thing chapter 01 changes. Everything else in the request stays the same.

Converse is Bedrock’s one request shape for every chat model. Call #2 is a single POST /model/{modelId}/converse to bedrock-runtime.eu-central-1.amazonaws.com, signed with SigV4, AWS’s request-signing scheme, using the chat server’s IAM role shop-support-chat. The AWS access tutorial shows how that signature is made.

The body carries system, messages, toolConfig and inferenceConfig.maxTokens; the response carries output.message, stopReason, usage and metrics. IAM has no Converse action: it checks bedrock:InvokeModel. The first version of the code names global.anthropic.claude-sonnet-5-5, copied from the model card.

THE FIRST VERSION OF THE CHAT CODE, AND WHAT CALL #2 RETURNS
import boto3

brt = boto3.client("bedrock-runtime", region_name="eu-central-1")

resp = brt.converse(
    modelId="global.anthropic.claude-sonnet-5-5",  # chapter 01 changes this prefix to eu.
    system=[{"text": SUPPORT_POLICY}, {"cachePoint": {"type": "default", "ttl": "1h"}}],
    messages=[{"role": "user", "content": [{"text": customer_message}]}],
    toolConfig={"tools": SHOP_TOOLS},              # get_order, issue_refund
    inferenceConfig={"maxTokens": 1024},
)
print(resp["stopReason"])                          # tool_use
print(resp["output"]["message"]["content"][-1]["toolUse"])
# {'toolUseId': 'tooluse_kq9WbXsYQ0', 'name': 'get_order', 'input': {'order_id': '48213'}}
STEP 03 / 06

THE ANSWER NEEDS A SECOND TRIP

  1. orders databasethe shop’s own, no model in it
  2. get_order 48213 · no model ↕
  3. chat server10.8.10.27
  4. call #2, then call #3 ↕
  5. Bedrock · Claudebedrock-runtime.eu-central-1
call #2 · the conversation goes to Claude
#2 · tool_use · get_order#3 · end_turn · reply in French
Fig 0.3Follow the two trips. The accent hop is the shop’s own database lookup: no model is involved in it.

Call #2 comes back with stopReason: tool_use and a toolUse block asking for get_order with order 48213. With tool use, the model can ask the caller to run a function the request described, instead of answering. Here Claude has asked for data and written no reply.

The chat server runs get_order against its own orders database: the parcel is held at the Lyon depot. Then it sends call #3, the whole conversation again plus a toolResult. Bedrock keeps no conversation between calls, so call #3 resends everything call #2 sent. It returns stopReason: end_turn and the reply, in French.

STEP 04 / 06

8,000 OF 9,750 TOKENS ARE A REREAD

tokens in one chat, both calls9,750
  • read from the cache8,000 tokens · 82.1%$0.10 per million
  • fresh input1,250 tokens · 12.8%$2.00 per million
  • written by Claude500 tokens · 5.1%$10.00 per million
the same 4,000-token start in both calls: support policy and two tool definitions
Fig 0.4The bar is drawn to scale in tokens. The accent segment is read from the cache, at a twentieth of the price.

A is the unit a model reads and writes, about four characters of text. Both calls start with the same 4,000 tokens: the shop’s support policy and two tool definitions. A cache point after the system prompt tells Bedrock to keep that start in a , and later calls read it at $0.10 per million tokens instead of $2.00.

The cached start must be at least 512 tokens for Claude Sonnet 5.5, and it lives 5 minutes or an hour after its last use; the shop asks for an hour. Per chat, Claude reads 8,000 tokens from the cache and 1,250 fresh, and writes 500.

CALL #3’S USAGE: 4,000 OF ITS INPUT TOKENS CAME FROM THE CACHE
{
  "stopReason": "end_turn",
  "usage": {
    "inputTokens": 850,
    "cacheReadInputTokens": 4000,
    "cacheWriteInputTokens": 0,
    "outputTokens": 350,
    "totalTokens": 5200
  }
}
STEP 05 / 06

500 TOKENS OUT COST MORE THAN 9,250 IN

ONE CHAT · CLAUDE SONNET 5.5 · GLOBAL RATES
calls #2 and #3 together
read from the cache$0.0008
8,000 tokens × $0.10 per million
fresh input$0.0025
1,250 tokens × $2.00 per million
written by Claude$0.0050
500 tokens × $10.00 per million
one chat$0.0083
shop-prod · eu-central-1 · one month
730 hours · 300,000 chats
Claude Sonnet 5.5, global rates$2,490.00
$0.0083 a chat × 300,000 chats
monthly bill so far$2,490.00
Fig 0.5Three lines, one per kind of token. The accent line, output, is the smallest count and the largest cost.

At global rates Claude Sonnet 5.5 costs $2.00 per million input tokens, $10.00 per million output tokens and $0.10 per million read from the cache. One chat comes to $0.0008 for the cache, $0.0025 for fresh input and $0.0050 for output: $0.0083.

Output is 5% of the tokens and 60% of the price. At 300,000 chats a month that is $2,490.00, the first line of the .

STEP 06 / 06

THE QUOTA COUNTS TOKENS YOU HAVEN’T USED YET

  • held back when call #3 starts, maxTokens 1,0245,874 tokens
    4,850 input + 1,024 maxTokens · the shop’s setting
  • the same call with maxTokens 64,00068,850 tokens
    4,850 input + 64,000 maxTokens · 12 times as much held back
when the call ends, the hold settles to what was used, with output tokens counted several times overthe limit both draw from: 6,000,000 tokens a minute for Claude Sonnet 5.5 through a cross-Region profile · over it, HTTP 429 ThrottlingException
Fig 0.6Two bars for call #3, to scale. The accent bar is what Bedrock holds back with the shop’s maxTokens of 1,024.

Bedrock limits each account by tokens per minute, per model and Region; Claude Sonnet 5.5 through a cross-Region profile starts at 6,000,000. When a request starts, Bedrock holds back its input tokens plus its maxTokens, the most it may write. When it ends, the hold settles to what was used, with output counted several times over for most Claude models.

AWS publishes that multiplier per model: 5 for Claude models up to version 4.7, 10 for Claude Opus 5.5, 15 for Claude Opus 4.8. Past the limit Bedrock answers HTTP 429, ThrottlingException. The shop sets maxTokens to 1,024, not 64,000. Separately, each account may use 700,000,000 tokens a day in a Region, a limit that can’t be raised.

WHAT YOU NOW KNOW

  • Bedrock serves Claude at bedrock-runtime and bedrock-mantle on one engine at one price; only bedrock-runtime offers EU routing and invocation logs.
  • One answer took two Converse calls, because Claude asked for the order through a tool that the shop’s own code ran.
  • Per chat: 8,000 cached, 1,250 fresh and 500 output tokens cost $0.0083, and the monthly bill is $2,490.00.
deep dive: Converse, InvokeModel and Messages, model access, errors, and what the bill leaves out

Three request shapes. InvokeModel takes the provider’s own body; for Claude that means anthropic_version: "bedrock-2023-05-31" and Anthropic’s Messages fields. Converse takes one shape for every model. The Messages API, on bedrock-runtime under /anthropic and on bedrock-mantle, takes Anthropic’s shape directly. ConverseStream and InvokeModelWithResponseStream need the separate permission bedrock:InvokeModelWithResponseStream. Bedrock caps one request at 20 MB.

API keys. Since 2025-07-08 Bedrock also accepts bearer tokens. Short-term keys last up to 12 hours and work only in the Region that issued them. Long-term keys create an IAM user, and AWS recommends them only for exploration. Policies can deny them with bedrock:CallWithBearerToken and the condition key bedrock:bearerTokenType.

Model access, 2025 onward. Since 2025-10-15 every serverless model is enabled by default in commercial Regions, and the Model access page survives only in GovCloud. Anthropic’s first-time use form is no longer needed from Claude Opus 4.7 on. Older models, including the Claude Haiku 4.5 of chapter 05, still need it once per account or once at the organization’s management account. The first call to a Marketplace-listed model subscribes the account automatically, which can take 15 minutes; denying aws-marketplace:Subscribe does not block that call, and denying bedrock:InvokeModel does.

Errors worth recognising. ThrottlingException 429, ModelNotReadyException 429, ModelTimeoutException 408, ModelErrorException 424, ValidationException 400 (including the bare-model-ID error in chapter 01) and AccessDeniedException 403. Stop reasons: end_turn, tool_use, max_tokens, stop_sequence, guardrail_intervened, content_filtered and model_context_window_exceeded.

What the monthly bill leaves out. Data transfer between Availability Zones, CloudTrail’s default trail (free for the first copy of management events), the orders database, the chat server itself, and S3 storage for the model and batch files, which is under $1. Cache writes are left out too: with a one-hour cache and traffic that rarely pauses for an hour, the start is written about once a day, $0.53 a month.

On the exam

  • “several teams need the same model through one API, with per-team cost reports” (SAP 1.2.1, 3.3.4)Converse plus application inference profiles with cost allocation tags (chapter 01)
  • “cut the cost of a chatbot that resends a long system prompt on every turn” (SAP 3.1.4)prompt caching, with a cache point after the system prompt
  • “requests fail with HTTP 429 at peak”check the tokens-per-minute quota and maxTokens, use a cross-Region profile for its higher quota, retry with backoff
  • “stop developers using long-lived Bedrock API keys”deny bedrock:CallWithBearerToken unless bedrock:bearerTokenType is SHORT_TERM

Six Regions behind one model ID

The first version of the chat code named global.anthropic.claude-sonnet-5-5, which lets Bedrock compute each answer in any commercial AWS Region on earth. The shop’s contracts with its EU business customers say their chats are processed in the EU. One prefix changes, the two calls land in two EU countries, and the monthly bill goes from $2,490.00 to $2,739.00.

STEP 01 / 06

A MODEL ID THAT IS A LIST OF REGIONS

model IDeu.anthropic.claude-sonnet-5-5
  • Ireland
    eu-west-1 · Ireland
  • London
    eu-west-2 · UKnot a destination from Frankfurt
  • Stockholm
    eu-north-1 · Sweden
  • Paris
    eu-west-3 · France
  • Frankfurt
    eu-central-1 · Germanycalled from here
  • Zurich
    eu-central-2 · Switzerlandnot a destination from Frankfurt
  • Spain
    eu-south-2 · Spain
  • Milan
    eu-south-1 · Italy
lit: 6 Regions that may compute the answer · dashed: in Europe, outside this list
Fig 1.1The outlined tile is where the chat server calls from. The six lit tiles are where Claude may answer.

An is an ID that names one model and a fixed list of Regions, and Bedrock sends each request to a Region in that list with capacity. AWS calls this . It works like a restaurant group’s booking line: you call one number, and you get a table at whichever of its restaurants is free, always in the city you chose.

Geographic profile IDs start us., eu., jp., au. or ca. (apac. on older models); global ones start global.. Called from Frankfurt, eu.anthropic.claude-sonnet-5-5 may run in six Regions: Frankfurt, Stockholm, Milan, Spain, Ireland and Paris. AWS says a geographic profile’s list never changes; the global list grows as Regions are added.

In Frankfurt, Claude Sonnet 5.5 can’t be called by its bare ID, anthropic.claude-sonnet-5-5. Bedrock answers HTTP 400, saying on-demand throughput isn’t supported, and asks for an inference profile. From Frankfurt the only choices are eu. and global..

THE SIX REGIONS BEHIND ONE PROFILE ID, ONE MODEL ARN EACH
aws bedrock get-inference-profile \
  --region eu-central-1 \
  --inference-profile-identifier eu.anthropic.claude-sonnet-5-5 \
  --query "models[].modelArn" --output text
# arn:aws:bedrock:eu-central-1::foundation-model/anthropic.claude-sonnet-5-5
# arn:aws:bedrock:eu-north-1::foundation-model/anthropic.claude-sonnet-5-5
# arn:aws:bedrock:eu-south-1::foundation-model/anthropic.claude-sonnet-5-5
# arn:aws:bedrock:eu-south-2::foundation-model/anthropic.claude-sonnet-5-5
# arn:aws:bedrock:eu-west-1::foundation-model/anthropic.claude-sonnet-5-5
# arn:aws:bedrock:eu-west-3::foundation-model/anthropic.claude-sonnet-5-5
STEP 02 / 06

CALL FROM LONDON AND LONDON JOINS IN

CALLED FROM
Frankfurt · eu-central-1profile prefix eu.
MAY BE ANSWERED IN
FrankfurtStockholmMilanSpainIrelandParis
the shop’s chat server: six EU Regions
CALLED FROM
London · eu-west-2profile prefix eu.
MAY BE ANSWERED IN
FrankfurtStockholmMilanSpainIrelandParisLondon
the same six, plus London because the call came from there
CALLED FROM
Canada Central · ca-central-1profile prefix us.
MAY BE ANSWERED IN
Canada CentralN. VirginiaOhioOregon
a call from Canada that can be computed in the US
Fig 1.2Three callers, three destination lists. The accent row shows a call leaving its own country for another one.

The Region you call is the source Region, and the Regions that may compute the answer are destination Regions. The EU profile accepts calls from eight European Regions. It adds London (eu-west-2) or Zurich (eu-central-2) to its destinations only when the call comes from there.

Cross-Region inference can also route to Regions your account hasn’t enabled. And geography belongs to the profile, not to the caller’s country: a Canadian app that calls the us. profile from ca-central-1 can be answered in Canada, Virginia, Ohio or Oregon.

STEP 03 / 06

TWO CALLS, TWO COUNTRIES

  • Ireland
    eu-west-1 · Ireland
  • London
    eu-west-2 · UK
  • Stockholm
    eu-north-1 · Sweden
  • Paris
    eu-west-3 · France
  • Frankfurt
    eu-central-1 · Germanycalled from here
  • Zurich
    eu-central-2 · Switzerland
  • Spain
    eu-south-2 · Spain
  • Milan
    eu-south-1 · Italy
call #2 leaves Frankfurt and is computed in Paris
CloudTrail, in eu-central-1: call #2 inferenceRegion eu-west-3 · call #3 inferenceRegion eu-north-1one illustrative run · Bedrock picks the Region per request
Fig 1.3Follow the two dots out of Frankfurt. Each lands where CloudTrail’s inferenceRegion field says that call was computed.

In this run call #2 was computed in Paris and call #3 in Stockholm, about two seconds apart: Bedrock picks the Region per request. CloudTrail records both in the source Region, Frankfurt, and names the Region that computed each in additionalEventData.inferenceRegion.

The traffic stays on the AWS network, encrypted, and routing costs nothing extra: the price is the calling Region’s. Data at rest, such as logs, configuration and batch output, stays in the source Region. If a model keeps prompts for abuse detection, they are stored in the destination Region.

CALL #2 IN CLOUDTRAIL: LOGGED IN FRANKFURT, COMPUTED IN PARIS
{
  "eventSource": "bedrock.amazonaws.com",
  "eventName": "Converse",
  "awsRegion": "eu-central-1",
  "userIdentity": {
    "arn": "arn:aws:sts::111122223333:assumed-role/shop-support-chat/i-0a1b2c3d4e5f60718"
  },
  "requestParameters": { "modelId": "eu.anthropic.claude-sonnet-5-5" },
  "additionalEventData": { "inferenceRegion": "eu-west-3" },
  "eventCategory": "Management"
}
STEP 04 / 06

ONE PERMISSION, SEVEN RESOURCES

THE POLICY MUST ALLOW bedrock:InvokeModel ON ALL SEVEN
  1. inference-profile/eu.anthropic.claude-sonnet-5-5eu-central-1 · the Region the shop calls
  2. foundation-model/anthropic.claude-sonnet-5-5eu-central-1 · a destination
  3. foundation-model/anthropic.claude-sonnet-5-5eu-north-1 · a destination
  4. foundation-model/anthropic.claude-sonnet-5-5eu-south-1 · a destination
  5. foundation-model/anthropic.claude-sonnet-5-5eu-south-2 · a destination
  6. foundation-model/anthropic.claude-sonnet-5-5eu-west-1 · a destination
  7. foundation-model/anthropic.claude-sonnet-5-5eu-west-3 · a destination
AND FOR A GLOBAL PROFILE, ONE MORE
  1. foundation-model/anthropic.claude-sonnet-5-5no Region, no account · aws:RequestedRegion = unspecified
Fig 1.4Seven ticked rows, one per resource the policy must allow. The accent row is extra, for global profiles only.

For a geographic profile, IAM must allow bedrock:InvokeModel on the profile’s ARN in the source Region and on the model’s ARN in every destination: 7 resources here. The usual condition, bedrock:InferenceProfileArn, lets the model be reached only through the profile.

A service control policy that denies every Region outside a list breaks these calls unless it exempts inference profiles; chapter 06 writes one. A global profile needs one more resource, arn:aws:bedrock:::foundation-model/anthropic.claude-sonnet-5-5, with no Region or account, and policies see its Region as the literal string unspecified.

THE CHAT SERVER’S POLICY: THE PROFILE, THEN THE MODEL IN ALL SIX DESTINATIONS
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "bedrock:InvokeModel",
      "Resource": "arn:aws:bedrock:eu-central-1:111122223333:inference-profile/eu.anthropic.claude-sonnet-5-5" },
    { "Effect": "Allow", "Action": "bedrock:InvokeModel",
      "Resource": [
        "arn:aws:bedrock:eu-central-1::foundation-model/anthropic.claude-sonnet-5-5",
        "arn:aws:bedrock:eu-north-1::foundation-model/anthropic.claude-sonnet-5-5",
        "arn:aws:bedrock:eu-south-1::foundation-model/anthropic.claude-sonnet-5-5",
        "arn:aws:bedrock:eu-south-2::foundation-model/anthropic.claude-sonnet-5-5",
        "arn:aws:bedrock:eu-west-1::foundation-model/anthropic.claude-sonnet-5-5",
        "arn:aws:bedrock:eu-west-3::foundation-model/anthropic.claude-sonnet-5-5" ],
      "Condition": { "StringEquals": { "bedrock:InferenceProfileArn":
        "arn:aws:bedrock:eu-central-1:111122223333:inference-profile/eu.anthropic.claude-sonnet-5-5" } } }
  ]
}
STEP 05 / 06

TEN PERCENT FOR STAYING IN EUROPE

GLOBAL PROFILE · ANY COMMERCIAL REGION
300,000 chats
read from the cache$240.00
8,000 tokens × $0.10 per million
fresh input$750.00
1,250 tokens × $2.00 per million
written by Claude$1,500.00
500 tokens × $10.00 per million
Claude, a month$2,490.00
EU PROFILE · SIX EU REGIONS
300,000 chats
read from the cache$264.00
8,000 tokens × $0.11 per million
fresh input$825.00
1,250 tokens × $2.20 per million
written by Claude$1,650.00
500 tokens × $11.00 per million
Claude, a month$2,739.00
shop-prod · eu-central-1 · one month
730 hours · 300,000 chats
Claude Sonnet 5.5, global rates$2,490.00
EU routing premium$249.00
the difference between the two receipts above
monthly bill so far$2,739.00
Fig 1.5Same tokens, two price lists. The accent line is the premium for keeping every chat in EU Regions.

From Claude Sonnet 4.5 on, a geographic or single-Region route costs 10% more than global. Through the EU profile, Claude Sonnet 5.5 is $2.20 per million input tokens, $11.00 output and $0.11 cached, against $2.00, $10.00 and $0.10. The same tokens cost $0.0091 a chat, and $2,739.00 a month.

The premium is $249.00. GDPR doesn’t require EU processing: it restricts transfers out of the European Economic Area and allows them with safeguards. The requirement here comes from the shop’s contracts, which is how usually arrives.

STEP 06 / 06

A PROFILE YOU CAN PUT A PRICE TAG ON

APPLICATION INFERENCE PROFILE · THE SHOP’S OWN
support-chat-euarn:aws:bedrock:eu-central-1:111122223333:application-inference-profile/a1b2c3d4e5f6
team = supportcost-centre = cx-eu
tags reach the AWS bill and Cost Explorer
SYSTEM-DEFINED INFERENCE PROFILE · AWS’S
eu.anthropic.claude-sonnet-5-5no tags: its spend shows up as the model
Frankfurt · eu-central-1Stockholm · eu-north-1Milan · eu-south-1Spain · eu-south-2Ireland · eu-west-1Paris · eu-west-3
Fig 1.6The accent layer is the shop’s own profile. Its tags reach the bill; the system profile underneath has none.

A system-defined profile such as eu.anthropic.claude-sonnet-5-5 can’t carry tags, so its spend shows up on the bill as the model, not as the support team. An application inference profile is one the shop creates by copying a system profile with CreateInferenceProfile, and then tags. Calls through its ARN are billed under those tags, and its CloudWatch metrics count tokens per team.

An account can hold 1,000 of them per Region. One 2026 limit: a Responses API call on bedrock-runtime that names an application profile is rejected with HTTP 400.

ONE PROFILE PER TEAM, TAGGED FOR THE BILL
aws bedrock create-inference-profile \
  --region eu-central-1 \
  --inference-profile-name support-chat-eu \
  --model-source copyFrom=arn:aws:bedrock:eu-central-1:111122223333:inference-profile/eu.anthropic.claude-sonnet-5-5 \
  --tags key=team,value=support key=cost-centre,value=cx-eu

# The chat code then calls the new ARN instead of eu.anthropic.claude-sonnet-5-5:
#   arn:aws:bedrock:eu-central-1:111122223333:application-inference-profile/a1b2c3d4e5f6
# Activate team and cost-centre as cost allocation tags to see them in Cost Explorer.

WHAT YOU NOW KNOW

  • Called from Frankfurt, eu.anthropic.claude-sonnet-5-5 may run in 6 EU Regions; CloudTrail’s inferenceRegion shows which one computed each call.
  • IAM must allow the profile plus the model in every destination; SCPs see a global profile’s Region as “unspecified”.
  • EU-only routing costs 10% more: the monthly bill goes from $2,490.00 to $2,739.00.
deep dive: in-Region inference, the Regions each geography covers, and turning global off

In-Region, geographic, global. Some models also take the bare ID in a single Region, which AWS calls in-Region inference. For the newest Claude models most Regions offer profiles only, and the bare ID returns HTTP 400: “Invocation of model ID anthropic.claude-sonnet-4-5-20250929-v1:0 with on-demand throughput isn’t supported. Retry your request with the ID or ARN of an inference profile that contains this model.” Several Regions offer only a limited model selection, and the destination list differs per model, so read each model card.

Quotas and capacity. Cross-Region quotas are about twice the in-Region ones: Claude Sonnet 4.6 gets 6,000,000 tokens a minute through a profile against 3,000,000 in one Region. Global routing reaches the most capacity, and is the default AWS recommends when residency doesn’t matter.

What stays where. CloudWatch and CloudTrail entries, invocation logs, guardrail configuration and batch output stay in the source Region. Guardrails have their own cross-Region “guardrail profile”, which stays inside the geography and costs nothing extra.

Turning global off for a whole organization. One SCP statement on aws:RequestedRegion equal to unspecified and a profile ARN matching global.* does it; chapter 06 shows it.

On the exam

  • “EU customer data must be processed only in the EU; minimize cost otherwise”the eu. geographic profile, not global, and accept the 10% premium
  • “cut data transfer cost for a multi-Region AI workload” (SAP 3.2.5)cross-Region inference has no routing charge; the price is the source Region’s
  • “calls fail with AccessDenied after an SCP restricted Regions”exempt bedrock:InferenceProfileArn or allow the destinations; for global, aws:RequestedRegion is unspecified
  • “show Bedrock token spend per business unit” (SAP 3.3.4)application inference profiles with cost allocation tags
  • “prove which Region processed a request” (SAP 2.4.3)CloudTrail additionalEventData.inferenceRegion

Whose computers run Claude

The customer typed an order number and a complaint into a chat box and never asked whose computer would read it. The shop has to know: Anthropic built the model, the machines are AWS’s, and the shop’s contracts name who may see the text. This chapter follows call #2 into the account that runs Claude, and the one copy of it the shop chooses to keep. The monthly bill goes from $2,739.00 to $2,748.94.

STEP 01 / 06

ONE ACCOUNT PER MODEL PROVIDER, PER REGION

shop-prodthe shop
  • Claude’s weightsno access
  • the machinesno access
  • the customer’s messagesends it, reads the answer
  • logs of the callonly if switched on
Bedrock serviceAWS
  • Claude’s weightscopies them in
  • the machinesruns them
  • the customer’s messagepasses it through
  • logs of the callits own service logs
model deployment accountAWS
  • Claude’s weightsholds them
  • the machinesholds them
  • the customer’s messagecomputes on it
  • logs of the callnone for the shop
Anthropicbuilt Claude
  • Claude’s weightsno access to this copy
  • the machinesno access
  • the customer’s messageno access
  • logs of the callno access
zero operator access · Anthropic personnel have no access to the inference machines
Fig 2.1Four columns, four parties. The accent row is the customer’s message: only the shop and AWS’s service can touch it.

In every Region where Bedrock runs, it keeps one per model provider, owned and operated by the Bedrock service team. Bedrock makes a deep copy of the provider’s inference software into that account. The copy of Claude that computed call #2 in Paris runs in the deployment account for Anthropic’s models in eu-west-3.

AWS’s documentation says model providers have no access to those accounts, to Bedrock’s logs, or to customer prompts and completions. AWS calls this zero operator access, and Anthropic’s own docs say its staff have no access to the inference machines. The shop’s account sends the request and receives the answer; it never sees the weights either.

STEP 02 / 06

MOST OF IT RUNS ON TRAINIUM

  • the shop’s triage endpoint2 GPUs
    two NVIDIA L4s, one per instance (chapters 03 and 04)
  • one Trn2 UltraServer64 Trainium2 chips
    four Trn2 instances joined into one machine
  • Project Rainier~500,000 Trainium2 chips
    US data centres, including Indiana · runs Claude for training and inference
chips, log scale: each tick is ten times the last
Fig 2.2Bars on a log scale, in chips. The accent bar is Project Rainier; the shop’s own two GPUs make the shortest.

No field in the Converse request, the response or any log says which chip computed the answer. Amazon says Bedrock “runs most of its inference on Trainium today”. Trainium is AWS’s own , a chip built for the arithmetic models do, in the way an NVIDIA GPU is.

Project Rainier, a cluster of nearly 500,000 Trainium2 chips across US data centres including one in Indiana, runs Claude for training and inference. In April 2026 Anthropic committed to spend more than $100 billion on AWS over ten years, for up to 5 gigawatts of Trainium capacity. Amazon has invested $13 billion in Anthropic, with up to $20 billion more agreed.

STEP 03 / 06

HOW LONG A PROMPT IS KEPT IS A SETTING

RETENTION MODES · LEAST KEPT FIRST
  • nonenothing written to durable storage
  • defaultthe model’s own default
  • aws_reviewkept for abuse checks, may be reviewed by AWS
  • provider_data_sharelegacy · nothing is shared today
  • Claude Opus 4.8what the retention page saysreleased before Claude Fable 5: no changeresultallows none, so nothing is kept
  • Claude Sonnet 5.5what the retention page saysnot named on the pageresultthe shop sets its account to none
  • Claude Fable 5, 5.1, 5.2what the retention page saysneeds aws_reviewresultall traffic kept up to 30 days
set per Region, account or project · Bedrock docs, November 2026
Fig 2.3Rows are Claude models and what the retention page says about each. The accent row is the shop’s own setting.

Bedrock’s data-retention setting decides whether prompts and answers are written to durable storage at all. Its modes, from least kept to most, are none, default, aws_review and the legacy provider_data_share, set per Region, account or project. AWS says no mode shares content with model providers today.

The page names one family with a rule of its own. The Claude Fable 5 models need aws_review: all their traffic is kept up to 30 days for automated abuse detection, the checks for misuse, and flagged items may be reviewed by AWS staff. Models released before Fable 5 kept their old behaviour; Claude Opus 4.8 writes nothing to durable storage under none.

Claude Sonnet 5.5 isn’t named on that page, so the shop sets its account’s mode to none, the strictest. One sentence quoted widely in 2024, “Amazon Bedrock doesn’t store or log your prompts and completions”, is no longer on the data-protection page.

STEP 04 / 06

THE COPY YOU CONTROL IS THE ONE YOU SWITCH ON

one invocation log recordcall #3
LOG RECORD
  • timestamp2026-11-14T14:00:03Z
  • regioneu-central-1
  • operationConverse
  • modelIdeu.anthropic.claude-sonnet-5-5
  • identity.arnarn:aws:sts::111122223333:assumed-role/shop-support-chat
  • input.inputTokenCount850
  • output.outputTokenCount350
  • input body21 KB
    the system prompt, the chat and the order record, in full
shop-prod · eu-central-1 · one month
730 hours · 300,000 chats
monthly bill before logging$2,739.00
prompt logs in Frankfurt$9.94
2 records a chat × 25 KB × 300,000 = 15 GB · $0.63 per GB in + $0.0324 per GB-month kept
monthly bill so far$2,748.94
at standard CloudWatch Logs rates in Frankfurt; storage before compression, an upper bound
Fig 2.4One log record for call #3, then the receipt. The accent line pays for the one lasting copy the shop controls.

is off by default. Switched on, it writes every request and response sent through bedrock-runtime to CloudWatch Logs or S3, in the same account and Region as the call. Bodies over 100 KB go to S3 as separate objects. Calls made through bedrock-mantle are not captured at all.

A record carries the time, account, Region, request ID, operation, model ID, the caller’s ARN, both bodies and the token counts. The shop logs to /aws/bedrock/invocations/support-eu in Frankfurt and keeps 30 days. Two records a chat at about 25 KB each come to 15 GB a month.

AWS lists no separate price for these logs. At Frankfurt’s standard CloudWatch Logs rates, $0.63 per GB ingested and $0.0324 per GB-month stored, they cost $9.94. The monthly bill goes from $2,739.00 to $2,748.94.

ONE CALL SWITCHES LOGGING ON, A SECOND KEEPS 30 DAYS
aws bedrock put-model-invocation-logging-configuration \
  --region eu-central-1 \
  --logging-config '{
    "cloudWatchConfig": {
      "logGroupName": "/aws/bedrock/invocations/support-eu",
      "roleArn": "arn:aws:iam::111122223333:role/bedrock-invocation-logging",
      "largeDataDeliveryS3Config": { "bucketName": "shop-eu-bedrock-logs", "keyPrefix": "large/" }
    },
    "textDataDeliveryEnabled": true,
    "imageDataDeliveryEnabled": false,
    "embeddingDataDeliveryEnabled": false
  }'

aws logs put-retention-policy --region eu-central-1 \
  --log-group-name /aws/bedrock/invocations/support-eu --retention-in-days 30
STEP 05 / 06

A SECOND WAY IN, RUN BY ANTHROPIC

Amazon Bedrock
  • runs the modelAWS
  • processes your promptsAWS only
  • model IDeu.anthropic.claude-sonnet-5-5
  • where it can runone geography per profile: eu., us., jp., au.
  • new featureson Bedrock’s release schedule
  • billed asan AWS service
Claude Platform on AWS
  • runs the modelAnthropic
  • processes your promptsAnthropic
  • model IDclaude-sonnet-5-5
  • where it can runinference_geo: us (1.1× price) or global
  • new featuresthe same day as Anthropic’s API
  • billed asAWS Marketplace, in Claude Consumption Units
Fig 2.5Same model, two operators. The accent row is the one the shop’s EU promise decides: who processes the prompt.

Claude Platform on AWS, generally available since 11 May 2026, is Anthropic’s own platform reached through an AWS account. AWS provides SigV4 sign-in, IAM permissions and billing through AWS Marketplace, in Claude Consumption Units. Anthropic runs the inference and is the data processor for prompts and answers.

Its model IDs carry no prefix (claude-sonnet-5-5), and features arrive the same day as on Anthropic’s own API. The AWS Region you pick decides only where IAM, CloudTrail and billing live. A request field, inference_geo, decides where inference runs: us, at 1.1 times the price, or global.

Anthropic’s docs recommend Bedrock when AWS must be the only data processor, or for FedRAMP High, IL4/5 or HIPAA. The shop’s promise is about EU processing, and Bedrock’s eu. profile is the route that keeps it, so the shop stays on Bedrock.

STEP 06 / 06

RESIDENCY IS A PLACE, SOVEREIGNTY IS CONTROL

GDPR
personal data may leave the EEA with safeguards: adequacy decisions, standard contractual clauses
data residency
processed and stored inside a chosen boundary · the shop’s contracts ask for this
eu. profileClaude Sonnet 5.5 ✓
data sovereignty
location plus control of access, keys and operations
AWS European Sovereign Cloudeusc-de-east-1Nova, open-weight modelsno Claude
Fig 2.6Three rings, widest first. The accent ring is where the shop’s promise sits; the innermost offers no Claude.

AWS uses for a requirement that content is processed and stored inside a chosen boundary. goes further: control over who can access the data, who holds its keys and who operates the systems, as well as where it is.

GDPR, the EU’s data-protection law, restricts transfers outside the European Economic Area in its Chapter V and allows them with adequacy decisions or standard contractual clauses, which AWS’s service terms include. It doesn’t require the shop’s EU-only promise; the shop’s contracts do.

The AWS European Sovereign Cloud, eusc-de-east-1 in Brandenburg, opened on 14 January 2026 and offers Bedrock, but its model list has Amazon Nova and open-weight models and no Claude. In AWS GovCloud (US), Claude models are in scope for FedRAMP High.

WHAT YOU NOW KNOW

  • Claude on Bedrock runs in AWS-owned model deployment accounts that Anthropic can’t access; Amazon says most Bedrock inference runs on Trainium.
  • Prompt retention is a per-model setting, and invocation logs exist only if switched on, in the shop’s own account and Region.
  • Logging 15 GB a month in Frankfurt adds $9.94: the monthly bill is $2,748.94.
deep dive: abuse detection, encryption, CloudTrail event types, and three other ways a proprietary model reaches your account

Abuse detection. Automated checks run on inputs and outputs. With cross-Region inference, any input and output that is retained is stored in the destination Region, not the source. Apparent child sexual abuse material in images is blocked, stored and reported to NCMEC. Customers eligible for Enterprise Frontier Safeguards get zero retention through at least 30 June 2027.

Encryption. Bedrock requires TLS 1.2 and recommends 1.3. Customer managed KMS keys can encrypt custom models, agents, knowledge bases, evaluation jobs and guardrails. Invocation logs are encrypted by the log group’s or the bucket’s own KMS key. Key policies and grants are covered in the AWS KMS tutorial.

CloudTrail event types. InvokeModel, InvokeModelWithResponseStream, Converse and ConverseStream are management events, so the default trail already has them. InvokeAgent, Retrieve, RetrieveAndGenerate, InvokeFlow, ApplyGuardrail and the asynchronous APIs are data events, which a trail records only if you enable them.

Three other ways a proprietary model reaches your account. Bedrock Marketplace deploys one of over 100 models to a SageMaker-managed endpoint whose instance type you choose, and you call it through InvokeModel or Converse; network isolation must stay on. Bedrock Custom Model Import serves your own Llama, Mistral, Qwen or GPT-OSS weights from S3 without instances, billed per Custom Model Unit in 5-minute windows (chapter 05). AWS Marketplace model packages on SageMaker run the seller’s container in the buyer’s account with no network access: the buyer can’t pull the image or weights, and the seller can’t see the buyer’s data.

On the exam

  • “a regulator asks who can access prompts sent to a third-party model” (SAP 2.x)model deployment accounts, zero operator access, no provider access
  • “all prompts and responses must be retained for audit” (SAP 2.3.2, 2.4.3)model invocation logging to CloudWatch Logs or S3, KMS-encrypted, same Region
  • “audit who called which model”CloudTrail management events, no extra setup
  • “use a proprietary third-party model without the vendor seeing data”an AWS Marketplace model package on SageMaker with network isolation, or Bedrock Marketplace
  • “must stay on EU-operated infrastructure, sovereign requirements”the European Sovereign Cloud, after checking which models it offers

Which chip holds 8.2 billion numbers

Before Claude sees the chat, the shop’s own model has already sorted it into one of twelve labels. That model is a file of 8.2 billion numbers in S3, and Bedrock’s Claude can’t run it. The shop has to rent a machine with enough of the right memory. This chapter works out which one, and the monthly bill goes from $2,748.94 to $3,864.45.

STEP 01 / 06

A MODEL THAT ONLY SORTS

the customer, in Lyon
“Bonjour, ma commande 48213 devait arriver mardi et je n’ai toujours rien. Je peux être remboursée ?”
with its instructions, about 1,500 tokens
call #1
shop-triageQwen3-8B, fine-tuned on 40,000 chats
late_delivery_refundlanguage · frabout 8 tokens written
THE TWELVE LABELS IT CHOOSES FROM
  • late_delivery_refund
  • where_is_my_order
  • return_request
  • damaged_item
  • wrong_item
  • payment_problem
  • change_address
  • cancel_order
  • invoice_request
  • product_question
  • account_access
  • other
Fig 3.1One chat in, one label out. The accent label is all the triage model writes: about 8 tokens.

The triage model is Qwen3-8B, an open-weight model: one whose numbers anyone can download and run. The shop fine-tuned it with LoRA on 40,000 labelled past chats and merged the result into its weights (how LoRA works). It reads the chat with its instructions, about 1,500 tokens, and writes a label of about 8 tokens: late_delivery_refund|fr.

The label routes the chat, chooses which instructions Claude gets, and feeds the shop’s reporting, which was built on these twelve labels. Call #1 happens at 14:00:00, before either Claude call.

STEP 02 / 06

16.4 GB BEFORE THE FIRST CHAT

NVIDIA L4 · GPU memory24 GB
  • weights 16.4 GB
  • runtime (assumed) 1.5 GB
  • room for chats 6.1 GB
weights: 8.2 billion parameters × 2 bytes (BF16)
one 1,500-token chat · 144 KiB a token = 221 MB
6.1 GB ÷ 221 MB a chat ≈ 27 chats at once
Fig 3.2A 24 GB bar, drawn to scale. The accent segment is what is left for chats after the weights load.

A model’s weights must sit in while it runs, and they take parameters × bytes per parameter. In BF16, a 16-bit number format, 8.2 billion parameters × 2 bytes is 16.4 GB. FP8 halves that to 8.2 GB and 4-bit quarters it to 4.1 GB; AWS’s own sizing guidance uses the same rule.

Each token the model holds in context also needs memory: 2 (keys and values) × 36 layers × 8 key-value heads × 128 numbers × 2 bytes = 147,456 bytes, 144 KiB, per token. A 1,500-token chat needs 221 MB on top of the weights.

On a 24 GB L4, after the weights and about 1.5 GB assumed for the runtime, 6.1 GB is left: room for about 27 chats at once.

WEIGHTS PLUS KV CACHE, IN SIX LINES OF ARITHMETIC
params = 8.2e9                                   # Qwen3-8B
weights_gb = params * 2 / 1e9                    # BF16, 2 bytes each: 16.4 GB

layers, kv_heads, head_dim = 36, 8, 128          # from the model config.json
kv_bytes = 2 * layers * kv_heads * head_dim * 2  # keys and values, BF16
print(kv_bytes)                                  # 147456 bytes (144 KiB) per token

chat_gb = kv_bytes * 1500 / 1e9                  # a 1,500-token chat: 0.22 GB
room_gb = 24 - weights_gb - 1.5                  # L4 memory minus weights and ~1.5 GB runtime
print(int(room_gb / chat_gb))                    # 27 chats in memory at once
STEP 03 / 06

THE 16 GB GPU MISSES BY 0.4 GB

NVIDIA T416 GB
ml.g4dn.xlarge · $0.921 an hour in Frankfurt · misses by 0.4 GB · no BF16 support
NVIDIA L424 GB
  • weights 16.4 GB
  • runtime 1.5 GB
ml.g6.xlarge · $1.409 an hour in Frankfurt · fits, with room for chats
AWS Inferentia232 GB
  • weights 16.4 GB
  • runtime 1.5 GB
ml.inf2.xlarge · $1.308 an hour in Frankfurt · fits once compiled with the Neuron SDK
NVIDIA L40S48 GB
  • weights 16.4 GB
  • runtime 1.5 GB
ml.g6e.xlarge · $3.13 an hour in Frankfurt · fits, room for long chats
RTX PRO 6000 Blackwell96 GB
  • weights 16.4 GB
  • runtime 1.5 GB
ml.g7e.2xlarge · $7.1489 an hour in Frankfurt · fits five times over
purple tick: the model’s weights, 16.4 GB, on every bar
Fig 3.3Five bars, one scale. The tick is the model’s 16.4 GB; the accent bar is the smallest that clears it with room.

ml.g4dn.xlarge carries an NVIDIA T4 with 16 GB at $0.921 an hour in Frankfurt: 0.4 GB short of the weights, and the T4 has no BF16 support. ml.g6.xlarge carries an NVIDIA L4 with 24 GB at $1.409 and fits, with 6.1 GB to spare.

ml.inf2.xlarge carries one AWS Inferentia2, AWS’s own chip for inference, with 32 GB at $1.308, but the model must first be compiled with AWS’s Neuron SDK. ml.g6e.xlarge carries an L40S, 48 GB at $3.13. ml.g7e.2xlarge carries an RTX PRO 6000 Blackwell, 96 GB at $7.15, enough for this model five times over.

The shop picks the L4: the smallest memory that holds the model with room for chats, on the CUDA software stack it already uses.

STEP 04 / 06

SAME GPU, TWICE THE HOST MEMORY

ml.g6.xlarge
GPU1 × L4, 24 GB
vCPUs4
host memory16 GiB
per hour, Frankfurt$1.409
ml.g6.2xlarge
GPU1 × L4, 24 GB
vCPUs8
host memory32 GiB
per hour, Frankfurt$1.5281
the shop’s choice
shop-prod · eu-central-1 · one month
730 hours · 300,000 chats
monthly bill before the GPU$2,748.94
one ml.g6.2xlarge$1,115.51
730 h × $1.5281 an hour
monthly bill so far$3,864.45
Fig 3.4Two instance cards with one identical GPU. The accent row, host memory, is the difference that matters for this model.

ml.g6.xlarge and ml.g6.2xlarge carry the same single L4. The xlarge has 4 vCPUs and 16 GiB of host memory, the ordinary memory the server uses while it reads the 16.4 GB model file and moves it onto the GPU; the 2xlarge has 8 vCPUs and 32 GiB. AWS’s own Qwen3-8B example on SageMaker uses ml.g6.2xlarge.

In Frankfurt the 2xlarge costs $1.5281 an hour against $1.409: $0.12 more, 8.5%. The shop takes the larger host for headroom. One ml.g6.2xlarge for 730 hours is $1,115.51, and the monthly bill goes from $2,748.94 to $3,864.45.

STEP 05 / 06

EVERY TOKEN REREADS ALL 16.4 GB

the triage model’s weights16.4 GB
one full read of the weights for every token, at 300 GB a second
tokens written0 of 8
each token≥ 55 ms
the whole label≥ 0.44 s
Fig 3.5Watch the sweep: one pass over the weights for each token written. Eight passes make the label.

To write one token, the model reads every one of its weights once. The L4 reads its memory at 300 GB a second, so writing from a 16.4 GB model takes at least 55 ms a token: at most about 18 tokens a second for one request. The 8-token label takes at least 0.44 s of writing.

Reading, not arithmetic, sets this limit; the AI numbers tutorial shows why. Claude writes 500 tokens a chat on far larger hardware the shop never sees, which is one reason the shop pays Bedrock per token for it instead of renting that hardware.

STEP 06 / 06

FROM ONE GPU TO 144 CHIPS

  • NVIDIA T4 · g4dn16 GB
    inference
  • NVIDIA L4 · g624 GB
    inference · the shop’s triage model
  • AWS Inferentia2 · inf232 GB
    inference
  • NVIDIA L40S · g6e48 GB
    inference
  • RTX PRO 6000 Blackwell · g7e96 GB
    inference
  • 8 × H100 · p5.48xlarge640 GB
    inference and training
  • 8 × B200 · p6-b200.48xlarge1,432 GB
    inference and training
  • 72 × GB200 · P6e UltraServer13.3 TB
    inference and training
  • 144 × Trainium3 · Trn3 UltraServer20.7 TB
    inference and training
accelerator memory per machine, log scale from 10 GB to 30 TB
Fig 3.6A ladder of memory per machine, log scale. The accent rung is the shop’s L4; the top rung is one UltraServer.

AWS groups accelerated instances into families. G instances (L4, L40S, RTX PRO Blackwell) run single-GPU work and small models. P instances (A100, H100, H200, B200, B300) run large models and training: p5.48xlarge has 8 H100s with 640 GB, at $55.04 an hour on EC2 in us-east-1 after a June 2025 cut of up to 45%.

A P6e-GB200 UltraServer joins 72 GPUs with 13.3 TB. Inf2 and Trn2/Trn3 carry AWS’s own chips, Inferentia for inference and Trainium for both; a Trn3 UltraServer joins 144 Trainium3 chips with 20.7 TB.

A new account can run none of them at first. The default quota for P, G, Inf and Trn instances is 0 vCPUs, and SageMaker’s endpoint quota for each GPU instance type, ml.g6.2xlarge included, starts at 0 too.

THE ENDPOINT QUOTA STARTS AT 0; THE FIRST GPU NEEDS A REQUEST
aws service-quotas list-service-quotas \
  --region eu-central-1 --service-code sagemaker \
  --query "Quotas[?QuotaName=='ml.g6.2xlarge for endpoint usage'].[QuotaName,Value]" \
  --output text
# ml.g6.2xlarge for endpoint usage    0.0     (the documented default)

aws service-quotas request-service-quota-increase \
  --region eu-central-1 --service-code sagemaker \
  --quota-code "$G6_2XL_ENDPOINT_QUOTA_CODE" --desired-value 4

WHAT YOU NOW KNOW

  • Weights need parameters × bytes: 8.2 billion in BF16 is 16.4 GB, plus 144 KiB of KV cache for every token in context.
  • A 16 GB T4 can’t hold it; a 24 GB L4 can, with room for about 27 chats at once.
  • One ml.g6.2xlarge for 730 hours costs $1,115.51: the monthly bill is $3,864.45.
deep dive: the accelerated instance families, getting capacity, and what SAP-C03 does and doesn’t ask

The families. One row per family: chip, memory on the largest size, and a representative price or status. SageMaker hosts the G, P and Inf2 families under ml. names at a premium; ml.trn2 has no hosting price at all.

  • G5 · A10G8 × 24 GB · g5.48xlarge $16.29/h
  • G6 · L48 × 24 GB · g6.xlarge $0.80/h
  • G6e · L40S8 × 48 GB · g6e.xlarge $1.86/h
  • G7 · RTX PRO 4500 Blackwell8 × 32 GB · GA June 2026
  • G7e · RTX PRO 6000 Blackwell8 × 96 GB · GA January 2026
  • P4d / P4de · A1008 × 40 or 80 GB · p4d.24xlarge $21.96/h
  • P5 · H1008 × 80 GB · p5.48xlarge $55.04/h
  • P5e / P5en · H2008 × 141 GB · p5en.48xlarge $63.30/h
  • P6-B200 · B2008 × ~179 GB · p6-b200.48xlarge $113.93/h
  • P6-B300 · B3008 × ~268 GB · p6-b300.48xlarge $142.42/h
  • P6e-GB200 · GB20072 GPUs, 13.3 TB · Capacity Blocks only
  • Inf2 · Inferentia212 × 32 GB · inf2.xlarge $0.76/h
  • Trn1 · Trainium16 × 32 GB · trn1.32xlarge $21.50/h
  • Trn2 · Trainium216 × 96 GiB · Capacity Blocks
  • Trn3 UltraServer · Trainium3144 × 144 GB · GA December 2025

Getting capacity. EC2 Capacity Blocks for ML reserve GPUs or Trainium for 1 to 14 days in 1-day steps, then 7-day steps up to 182 days, starting up to 8 weeks ahead, paid upfront, with no cancellation and no Savings Plans discount. SageMaker training plans can target inference endpoints. On-Demand Capacity Reservations have no term and no discount; future-dated ones exclude P and Trn. Savings Plans save up to 66% (Compute), 72% (EC2 Instance) or 64% (SageMaker, which covers real-time inference and batch transform). Spot saves up to 90%, but not for real-time endpoints.

Shrinking a model. FP8 halves the weights and usually keeps accuracy; 4-bit quarters them with a measurable loss. MIG partitions, set with AcceleratorPartitionType, slice one large GPU into several smaller ones for several small models.

What SAP-C03 asks. No task in the guide names an instance family. Expect the consequence instead: the model doesn’t fit, the GPUs are idle, capacity isn’t there on launch day. The Generative AI Developer exam’s Operational Efficiency domain goes further into the hardware.

On the exam

  • “a fine-tuned 8B model must be hosted at lowest cost with real-time latency” (SAP 1.2.3, 3.1.4)the smallest GPU whose memory holds the weights plus KV cache, or Inferentia2 if compiling is acceptable
  • “guaranteed GPU capacity for a two-week launch”EC2 Capacity Blocks for ML, or a SageMaker training plan that targets the endpoint
  • “a steady 24/7 inference fleet, reduce cost” (SAP 3.1.1)a SageMaker Savings Plan
  • “creating an endpoint fails with a quota error for a new GPU type”request the per-instance endpoint quota in Service Quotas

From a model file to an address

The shop now has a model in an S3 bucket and an instance type that can hold it. The chat server still has nothing to send a request to. SageMaker AI builds an address from three objects, the server inside it is vLLM, and a second instance in a second data centre takes the monthly bill from $3,864.45 to $4,979.96.

STEP 01 / 06

THREE OBJECTS MAKE ONE ENDPOINT

MODEL
shop-triage-qwen3-8b-v3
what to run
  • container image
    the AWS LMI container
  • model artefacts
    s3://shop-eu-models/triage/v3/
  • execution role
    shop-sagemaker-triage
  • network
    the app subnets in 2 zones
ENDPOINT CONFIG
shop-triage-ec-v3
on what, how many
  • production variant
    main
  • instance type
    ml.g6.2xlarge
  • instance count
    × 2 · can’t be edited later
  • traffic weight
    1.0, all requests
ENDPOINT
shop-triage
the address the chat server calls
  • status
    Creating → InService, several minutes
  • config in use
    shop-triage-ec-v3
  • address
    runtime.sagemaker.eu-central-1.amazonaws.com
Fig 4.1Read left to right: a Model, a config naming the instances, then the Endpoint. The accent field is the instance count.

A is an HTTPS address in front of one or more instances running a model server. SageMaker AI builds it from three objects. A Model names the container image, the model files in S3, the IAM role the container runs as and, optionally, the subnets it uses. An EndpointConfig lists production variants, each an instance type, a count and a share of traffic. The Endpoint puts one config into service.

Creating the endpoint takes several minutes, while SageMaker starts the instances and copies 16.4 GB of weights to each. An EndpointConfig can’t be edited: every change is a new config and an UpdateEndpoint call that swaps it in. SageMaker was renamed SageMaker AI on 3 December 2024; the API, the CLI and IAM policies still say sagemaker.

A MODEL, A CONFIG FOR TWO ML.G6.2XLARGE INSTANCES, THEN THE ENDPOINT
import boto3

sm = boto3.client("sagemaker", region_name="eu-central-1")
sm.create_model(
    ModelName="shop-triage-qwen3-8b-v3",
    ExecutionRoleArn="arn:aws:iam::111122223333:role/shop-sagemaker-triage",
    PrimaryContainer={
        "Image": LMI_IMAGE_URI,                  # the AWS LMI container with vLLM
        "ModelDataSource": {"S3DataSource": {"S3Uri": "s3://shop-eu-models/triage/v3/",
                            "S3DataType": "S3Prefix", "CompressionType": "None"}},
        "Environment": {"HF_MODEL_ID": "/opt/ml/model", "TENSOR_PARALLEL_DEGREE": "1"},
    },
    VpcConfig={"Subnets": APP_SUBNETS, "SecurityGroupIds": ["sg-0c3d4e5f6a7b8c9d0"]},
)
sm.create_endpoint_config(EndpointConfigName="shop-triage-ec-v3", ProductionVariants=[{
    "VariantName": "main", "ModelName": "shop-triage-qwen3-8b-v3",
    "InstanceType": "ml.g6.2xlarge", "InitialInstanceCount": 2}])
sm.create_endpoint(EndpointName="shop-triage", EndpointConfigName="shop-triage-ec-v3")
STEP 02 / 06

THE SERVER INSIDE IS vLLM

  1. Qwen3-8B triage weights16.4 GB, read from S3 when an instance starts
  2. vLLM 0.23.0continuous batching: new chats join while others are mid-answer · about 27 at once
  3. DJL Servinganswers SageMaker’s HTTP calls and passes them to vLLM
  4. LMI container (AWS)Large Model Inference image v27, 2026-07-16
  5. ml.g6.2xlarge1 × NVIDIA L4 · 24 GB of GPU memory
Fig 4.2Five layers from weights down to the chip. The accent layer, vLLM, decides how many chats share the GPU at once.

The container is AWS’s (Large Model Inference), version 27 from 16 July 2026. Inside it, DJL Serving answers SageMaker’s HTTP calls and hands them to vLLM 0.23.0, the open-source model server that runs the model on the GPU. It is configured with environment variables such as HF_MODEL_ID and TENSOR_PARALLEL_DEGREE, which the previous step’s sample sets.

vLLM batches continuously: a new chat joins the running batch while others are mid-answer, instead of waiting for the batch to finish. That is how about 27 chats share one L4, each with its own 221 MB of KV cache. Hugging Face’s TGI, long the default LLM server on SageMaker, went into maintenance mode and was archived on 21 March 2026.

STEP 03 / 06

TWO INSTANCES, TWO DATA CENTRES

shop-triage
spreads requests across both zones
eu-central-1a · Availability Zone
subnet 10.8.10.0/24
ml.g6.2xlarge
1 × L4 · the first copy, chapter 03
eu-central-1b · Availability Zone
subnet 10.8.11.0/24
ml.g6.2xlarge
the second copy · added in this step
shop-prod · eu-central-1 · one month
730 hours · 300,000 chats
up to chapter 03$3,864.45
second ml.g6.2xlarge, second zone$1,115.51
730 h × $1.5281 an hour
monthly bill so far$4,979.96
Fig 4.3Two zones, one instance each. The accent instance is the second copy, and the receipt line it adds.

An Availability Zone is one or more separate data centres inside a Region. When an endpoint has two or more instances and its subnets sit in two zones, SageMaker places the instances in different zones and spreads requests across them. AWS recommends at least two instances for any production endpoint.

The shop’s EndpointConfig says 2: one instance in eu-central-1a and one in eu-central-1b. The second ml.g6.2xlarge costs the same as the first, $1,115.51 for 730 hours. The monthly bill goes from $3,864.45 to $4,979.96.

STEP 04 / 06

6 MB IN, 60 SECONDS TO ANSWER

InvokeEndpoint · sagemaker-runtimecall #1
REQUEST
  • EndpointNameshop-triage
  • ContentTypeapplication/json
  • Bodyabout 6 KB
    the chat and its instructions, 1,500 tokens
RESPONSE
  • Bodylate_delivery_refund|fr
    about 8 tokens
request body6 KB of 6 MB
the API’s hard limit; parts of the guide say 25 MB
time to answer≈ 0.5 s of 60 s
past 60 s the call fails · streaming may run 8 minutes · estimate
Fig 4.4Call #1 against the limits. The accent gauge is the 60-second deadline, which the 8-token label clears easily.

Call #1 is InvokeEndpoint on the SageMaker runtime, sagemaker-runtime in boto3. The request body may be at most 6 MB: that is the API reference’s hard limit, although parts of the developer guide say 25 MB. The container must answer within 60 seconds, so AWS suggests a 70-second socket timeout in the SDK. Streaming responses may run for up to 8 minutes.

The triage request is about 6 KB, the chat plus its instructions, and the answer is a label of about 8 tokens. A request that is larger, or that takes minutes, belongs on an asynchronous endpoint, which chapter 05 compares with the other ways to run a model.

CALL #1: THE CHAT IN, ONE LABEL OUT
import json
import boto3

smr = boto3.client("sagemaker-runtime", region_name="eu-central-1")

resp = smr.invoke_endpoint(
    EndpointName="shop-triage",
    ContentType="application/json",
    Body=json.dumps({"inputs": triage_prompt, "parameters": {"max_new_tokens": 8}}),
)                                    # Body at most 6 MB; the container has 60 seconds
print(json.loads(resp["Body"].read()))
# {'generated_text': 'late_delivery_refund|fr'}
STEP 05 / 06

MANY MODELS, ONE INSTANCE

ONE INSTANCE · ml.g6.2xlarge · 1 × L4 24 GB
  • base: Qwen3-8Bthe 16.4 GB of weightsreserves 1 GPU · 18 GB of memorycopies · 1 · scales on its own
  • adapter: triage-dea LoRA adapter on the base, a few MBborrows the base’s GPU and weightscopies · 1 · scales on its own
  • adapter: triage-esa LoRA adapter on the base, a few MBborrows the base’s GPU and weightscopies · 1 · scales on its own
illustrative: the shop runs one model today and needs none of this yet
Fig 4.5One instance, split three ways. The accent block is a LoRA adapter that borrows the base model’s weights.

An reserves part of an endpoint’s instances for one model: a number of accelerators, CPU cores and memory (NumberOfAcceleratorDevicesRequired, MinMemoryRequiredInMb), a number of copies, and its own scaling. Several components can share one instance.

A component can also be a LoRA adapter on another one (BaseInferenceComponentName). One GPU then serves a base model and several adapters, each scaled separately; the LoRA tutorial shows why an adapter can be swapped per request. Two older options remain: multi-model endpoints share one container and load models from S3 on demand, and multi-container endpoints run different frameworks side by side.

The shop runs one model and needs none of these yet.

STEP 06 / 06

THE NEXT MODEL GOES TO 10% FIRST

update startsupdate done
  • new fleet starts, 2 instancesseveral minutes
  • 10% of traffic to the new fleet · bake10 min
  • 100% of traffic to the new fleetthen the old fleet stops
a CloudWatch alarm fires during the bake → roll backall traffic returns to the old fleet; the new one is removed
  • GPUs billed · 2 → 4 → 2
  • both fleets run, and bill, until the update finishes
Fig 4.6Read the timeline left to right. The accent box is the rollback a CloudWatch alarm triggers during the bake.

Deployment guardrails replace an endpoint’s instances with a blue/green update. All at once moves every request to the new fleet in one step, canary moves a small share first, and linear moves equal steps. Each step waits through a baking period, and if a CloudWatch alarm fires during it, SageMaker rolls back to the old fleet.

Both fleets run, and bill, until the update finishes, so a canary on shop-triage briefly runs four GPUs. Since March 2025, endpoints built on inference components can use rolling updates instead, replacing instances in batches. A shadow test sends a copy of live requests to a new variant and returns only the old variant’s answers.

WHAT YOU NOW KNOW

  • An endpoint is three objects: a Model, an EndpointConfig naming ml.g6.2xlarge × 2 that can’t be edited, and the Endpoint.
  • Real-time calls carry at most 6 MB and must finish in 60 seconds; the triage call sends about 6 KB.
  • The second instance, in a second Availability Zone, adds $1,115.51: the monthly bill is $4,979.96.
deep dive: VPC and network isolation, Model Monitor, JumpStart, Bedrock Marketplace and HyperPod

Network and data controls. VpcConfig on the Model puts the container’s network interfaces in the shop’s subnets. EnableNetworkIsolation blocks every inbound and outbound call from the container, and AWS Marketplace model packages require it. KmsKeyId on the EndpointConfig encrypts the attached storage volume; instance-store NVMe disks use hardware encryption instead. Data capture writes requests and responses to S3 for Model Monitor, which watches for drift (SAP 5.3.5).

Where else an endpoint comes from. SageMaker JumpStart lists hundreds of models; gated ones need accept_eula=True. Bedrock Marketplace deploys a model onto a SageMaker-managed endpoint that you size yourself, and calls it through Bedrock’s APIs. HyperPod inference (July 2025) runs JumpStart models on EKS clusters, scaled by KEDA.

Startup time. Fast Model Loader streams weights straight to the GPU; AWS reports loads up to 15 times faster. Container image caching (June 2026) cut AWS’s own Qwen3-8B example on ml.g6.2xlarge from 525 seconds to 258 seconds to start. Both matter only when the endpoint scales out.

On the exam

  • “a custom model needs a real-time API with high availability”a SageMaker real-time endpoint with at least two instances across Availability Zones · SAP 1.2.3
  • “deploy a new model version with automatic rollback on errors”blue/green deployment guardrails with canary traffic shifting and CloudWatch alarms
  • “host many fine-tuned variants of one base model cheaply”inference components with LoRA adapters, or a multi-model endpoint
  • “test a new model on live traffic without affecting users”a shadow test
  • “inference payloads are 200 MB, or take 20 minutes”an asynchronous endpoint (chapter 05)

Paying for the traffic the shop actually has

Two GPUs now run the triage model around the clock, and Claude is billed for every token. Neither choice has been checked against how the chats actually arrive: 10,000 a day, mostly in daytime. This chapter measures how busy the GPUs are, prices a cheaper triage the shop doesn’t take, and adds the nightly batch job that takes the monthly bill from $4,979.96 to $5,351.21.

STEP 01 / 06

TWO GPUS, BUSY ABOUT 3% OF THE TIME

  • instance in eu-central-1a · one day
  • instance in eu-central-1b · one day
busy ≈ 42 h of 1,460 h billed a month · 2.9%estimate: 300,000 calls × about 0.5 s of GPU time each, drawn to scale
Fig 5.1Twenty-four hours for each instance, to scale. The accent slivers are the time either GPU is actually busy.

This is an estimate, and labelled as one. Each triage call keeps a GPU busy for about 0.5 s: at least 0.44 s to write the 8-token label, plus reading the 1,500-token prompt. 300,000 calls × 0.5 s is 150,000 GPU-seconds, about 42 hours a month.

The bill pays for 2 × 730 = 1,460 GPU-hours, and the GPUs are busy for about 2.9% of them. Continuous batching would let each one hold about 27 chats at once, so the same two instances could serve far more traffic than they get.

STEP 02 / 06

FOUR WAYS TO RUN THE SAME MODEL

  • real-time
    • GPUsyes
    • largest request6 MB
    • time limit60 s
    • scales to zeroonly with inference components
    FITS THE TRIAGE CHAT
    yes: the chat waits for the label
  • serverless
    • GPUsno, CPU only
    • largest request4 MB
    • time limit60 s
    • scales to zeroyes, with cold starts
    FITS THE TRIAGE CHAT
    no: 6 GB of memory, the model is 16.4 GB
  • asynchronous
    • GPUsyes
    • largest request1 GB, through S3
    • time limit1 hour
    • scales to zeroyes
    FITS THE TRIAGE CHAT
    no: the answer arrives later, by notification
  • batch transform
    • GPUsyes
    • largest requesta whole dataset
    • time limitdays
    • scales to zeroruns only while the job runs
    FITS THE TRIAGE CHAT
    no: for files, not for a customer
Fig 5.2Four cards, four ways to run it. The accent card is the only one a waiting customer can use.

SageMaker can run a model four ways. Real-time keeps instances running, takes up to 6 MB and answers within 60 seconds. Serverless inference starts capacity per request and bills by the second, but it has no GPUs, at most 6 GB of memory and 200 concurrent requests per endpoint, and cold starts after idle. A 16.4 GB model can’t load there at all.

Asynchronous inference takes requests through S3, up to 1 GB and 1 hour each, announces the result by notification, and can scale its instances to zero. Batch transform runs a whole dataset and bills only while the job runs. A customer waiting in a chat window needs the label now, so real-time is the only fit.

STEP 03 / 06

SCALE ON WAITING REQUESTS, EVERY 10 SECONDS

requests in progress per instance, busiest hour≈ 0.07 · estimate
0target 8: add an instance above this10
about 1,000 chats × 0.5 s, spread over 2 instances; the metric arrives every 10 s
instances · 2 all daythe policy allows 2 to 4 and never needs a third
scaled to zero insteadthe next request gets an error for several minutes while an instance starts
Fig 5.3The accent bar is the shop’s busiest hour, far below the target of 8, so the instance count never leaves 2.

Application Auto Scaling changes an endpoint’s instance count to hold a metric at a target. The classic metric, SageMakerVariantInvocationsPerInstance, arrives once a minute. The high-resolution metric of requests in progress arrives every 10 seconds (July 2024), so a policy detects load up to 6 times sooner; it scales in no faster. The shop allows 2 to 4 instances, at 8 requests in progress per instance.

In the busiest hour, about 1,000 chats, that metric is about 0.07 per instance (an estimate), so the policy never adds a third. Scaling to zero needs inference components and a step-scaling alarm to wake the endpoint. Waking takes minutes, and until an instance is ready, requests get errors, not slow answers.

TWO TO FOUR INSTANCES, SCALED ON REQUESTS IN PROGRESS
aws application-autoscaling register-scalable-target \
  --service-namespace sagemaker \
  --resource-id endpoint/shop-triage/variant/main \
  --scalable-dimension sagemaker:variant:DesiredInstanceCount \
  --min-capacity 2 --max-capacity 4

aws application-autoscaling put-scaling-policy \
  --service-namespace sagemaker \
  --resource-id endpoint/shop-triage/variant/main \
  --scalable-dimension sagemaker:variant:DesiredInstanceCount \
  --policy-name triage-concurrency --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration '{"TargetValue": 8,
    "PredefinedMetricSpecification": {"PredefinedMetricType":
    "SageMakerVariantConcurrentRequestsPerModelHighResolution"}}'
STEP 04 / 06

THE SAME LABEL FOR $51 A MONTH

WHAT THE SHOP PAYS
2 × ml.g6.2xlarge
$2,231.02 a month
  • 2 × 730 instance-hours
  • the same price at any traffic
LEFT OFF THE BILL
Claude Haiku 5.5, EU profile
$50.82 a month
  • 1,500 in × $0.11 + 8 out × $0.55 per million
  • × 300,000 chats · list price
LEFT OFF THE BILL
Custom Model Import
per 5-minute window
  • the shop’s own Qwen3 weights, no instances
  • billed from the first call, zero when idle
  • cold starts of tens of seconds
break-even · about 13 million chats a month
below it the pay-per-token model is cheaper; above it the GPUs are
Fig 5.4Three ways to get the same label. The accent card is what the shop pays; the others are left off the bill.

A comparison the bill leaves out. Through its EU profile, eu.anthropic.claude-haiku-5-5, Claude Haiku 5.5 would read the same 1,500 tokens and write 8 for $0.000169 a chat at list price ($0.11 and $0.55 per million). That is $50.82 a month against $2,231.02 for the two GPUs, about 44 times less. The GPUs only win above about 13 million chats a month.

Bedrock Custom Model Import is a middle option. It serves the shop’s own Qwen3 weights in Frankfurt with no instances to rent, billed in 5-minute windows from the first call and scaled to zero when idle, with cold starts of tens of seconds. The shop keeps its endpoint because its reporting depends on these exact labels. The numbers say where the line is.

STEP 05 / 06

HALF PRICE IF IT CAN WAIT TILL MORNING

nightly summaries · claude haiku 4.5 · eu profile · batch
10,000 records a night × 30 nights
up to chapter 04$4,979.96
input tokens$247.50
450 million × $0.55 per million
output tokens$123.75
45 million × $2.75 per million
nightly summaries$371.25
half the on-demand price · 1,650 tokens a chat
monthly bill so far$5,351.21
Fig 5.5One night’s job on the receipt. The accent line is the summaries, priced at half the on-demand rate.

runs a file of requests from S3 and writes the results to S3 in the source Region, at half the on-demand price. A job takes 100 to 100,000 records, and its model ID may be a geographic profile. Batch isn’t offered for Claude Sonnet 5.5, Opus 5.5 or Haiku 5.5, so the summaries use Claude Haiku 4.5, which offers it only through cross-Region profiles.

Each night’s job has about 10,000 records, one per chat: 1,500 tokens in and 150 out. A month is 450 million input tokens at $0.55 per million and 45 million output tokens at $2.75: $247.50 plus $123.75 is $371.25. Claude Haiku 4.5 still needs Anthropic’s one-time use-case form. The monthly bill goes from $4,979.96 to $5,351.21.

LAST NIGHT’S CHATS, SUMMARIZED AT HALF PRICE
aws bedrock create-model-invocation-job \
  --region eu-central-1 \
  --job-name support-summaries-2026-11-15 \
  --model-id eu.anthropic.claude-haiku-4-5-20251001-v1:0 \
  --role-arn arn:aws:iam::111122223333:role/shop-bedrock-batch \
  --input-data-config '{"s3InputDataConfig": {"s3InputFormat": "JSONL",
      "s3Uri": "s3://shop-eu-batch/in/2026-11-14.jsonl"}}' \
  --output-data-config '{"s3OutputDataConfig": {"s3Uri": "s3://shop-eu-batch/out/"}}'

# 10,000 records, one JSON line per chat (a job takes 100 to 100,000).
# Results land in the same bucket in eu-central-1, usually within hours.
STEP 06 / 06

NO RESERVED CAPACITY FOR THE NEWEST CLAUDE

ways to buy Claude capacity on Bedrock
through the EU profile, November 2026
way to payClaude Sonnet 5.5Claude Haiku 4.5
on-demand, per token✓ the only one✓
batch, half price✗✓ through a profile
Reserved, 1 or 3 months✗✓ floor $4,818.00 a month
Provisioned Throughput✗✗ only Claude 3 models
Priority and Flex tiers: offered for some other providers’ models; Claude Sonnet 5.5’s model card shows neither
Fig 5.6Rows are Bedrock’s ways to buy capacity, columns are models. The accent row is the only one Claude Sonnet 5.5 offers.

Bedrock sells Claude capacity several ways, and the newest models take only one. On-demand is per token, with no commitment. The Reserved tier (November 2025) is a 1- or 3-month commitment priced per hour for each 1,000 tokens a minute, with at least 100,000 input tokens a minute. It covers Claude Sonnet 4.5 and 4.6, Opus 4.5 and Haiku 4.5, not Sonnet 5.5.

For Claude Haiku 4.5 through the EU profile, that floor alone costs 100 × $0.066 × 730 = $4,818.00 a month, whatever the traffic. Provisioned Throughput, sold in model units, now lists only Claude 3 models in one Region, and inference profiles don’t support it. The Priority and Flex tiers (November 2025) trade price for speed on some other providers’ models.

WHAT YOU NOW KNOW

  • The two GPUs are busy about 3% of the time; serverless can’t hold a 16.4 GB model, and scaling to zero answers with errors while waking.
  • Claude Sonnet 5.5 is on-demand only on Bedrock; batch and Reserved capacity apply to older Claude models such as Haiku 4.5.
  • Nightly summaries on Claude Haiku 4.5 batch add $371.25: the monthly bill is $5,351.21.
deep dive: Savings Plans, asynchronous endpoints, batch limits, and Bedrock’s other tiers

Savings Plans for the endpoint. A SageMaker Savings Plan, up to 64% off for a 1- or 3-year commitment, covers real-time inference and batch transform but not serverless inference. It is the usual answer for a steady fleet that runs all day. Spot capacity doesn’t apply to real-time endpoints.

Asynchronous endpoints in detail. InvokeEndpointAsync takes an S3 input location, or since June 2026 an inline body of up to 128,000 bytes. A request may run for up to 3,600 seconds and wait in the queue for up to 6 hours, and results are announced through SNS. Use it for large documents or slow models, not for a chat.

Batch details. A job reads one JSON record per line from S3, 1 GB per input file and 5 GB per job, and writes its output to the source Region’s bucket. Prompt caching doesn’t apply to batch. Check batch support on each model card: it differs even between models of one family.

Latency-optimized inference is still a preview, on Claude 3.5 Haiku in two US Regions. It is not an option for this design.

On the exam

  • “nightly processing of 100,000 documents, results needed by morning, lowest cost”Bedrock batch inference at 50% off, after checking the model supports it · SAP 3.1.4
  • “a model with spiky traffic and long idle periods; cost matters more than the first request’s latency”inference components with scale to zero, asynchronous inference, or Custom Model Import
  • “a GPU endpoint is mostly idle”consolidate with inference components, downsize, or move to a pay-per-token model
  • “guaranteed throughput for a Claude model”the Reserved tier where the model supports it; Provisioned Throughput is legacy
  • “LLM endpoints scale out too slowly”high-resolution concurrency metrics · SAP 5.3.5

Off the internet, inside the rules

Every call so far has left the chat server for a public AWS hostname, through the VPC’s NAT gateway. And nothing stops a developer from changing eu. back to global. in one line of code. Interface endpoints keep the calls inside the network; an endpoint policy, IAM and one organization-wide policy keep them on the EU profile; a guardrail and CloudTrail watch what passes. The monthly bill goes from $5,351.21 to $5,386.41.

STEP 01 / 05

TWO PRIVATE ADDRESSES FOR TWO SERVICES

NOW · INSIDE THE VPC
  1. chat server10.8.10.27 · app subnet
  2. calls #1–#3
  3. two interface endpointsone interface in each zone · 10.8.10.x and 10.8.11.x
  4. AWS network
  5. Bedrock runtime · SageMaker runtimesame hostnames, private addresses
BEFORE · OUT THROUGH THE NAT GATEWAY
  1. chat server10.8.10.27
  2. not used
  3. NAT gatewaypublic subnet
  4. not used
  5. the internetpublic AWS hostnames
shop-prod · eu-central-1 · one month
730 hours · 300,000 chats
interface endpoints, 2 services × 2 zones$35.20
2,920 endpoint-hours × $0.012 + 16.2 GB × $0.01
monthly bill$5,386.41
Fig 6.1Follow calls #1 to #3. The accent hops never reach the NAT gateway or the internet.

An puts a network interface for one AWS service inside your own subnets. The shop creates two, com.amazonaws.eu-central-1.bedrock-runtime and com.amazonaws.eu-central-1.sagemaker.runtime, each with an interface in both app subnets. With private DNS on, the usual hostnames resolve to addresses inside 10.8.0.0/16, so calls #1 to #3 never reach the NAT gateway, and not one line of code changes.

An endpoint bills per Availability Zone per hour, plus per GB it carries: 2 endpoints × 2 zones × 730 hours × $0.012 = $35.04, and 16.2 GB × $0.01 = $0.16. The monthly bill goes from $5,351.21 to $5,386.41. How private DNS swaps the address is in the AWS VPC tutorial’s endpoint chapter.

ONE ENDPOINT, TWO SUBNETS, PRIVATE DNS ON
aws ec2 create-vpc-endpoint \
  --region eu-central-1 \
  --vpc-id vpc-0e8a1c2b3d4f50617 \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.eu-central-1.bedrock-runtime \
  --subnet-ids subnet-0a1b2c3d4e5f6a7b8 subnet-0b2c3d4e5f6a7b8c9 \
  --security-group-ids sg-0d4e5f6a7b8c9d0e1 \
  --private-dns-enabled

# Same again with com.amazonaws.eu-central-1.sagemaker.runtime.
# bedrock-runtime.eu-central-1.amazonaws.com now resolves to 10.8.10.x and 10.8.11.x.
STEP 02 / 05

THE ENDPOINT POLICY NAMES THE PROFILE

CALL #2 · EU.
SAME CALL · GLOBAL.
1. IAM policy on shop-support-chatallows bedrock:InvokeModel on Claude Sonnet 5.5
✓ allow✓ allow
2. endpoint policyeu.anthropic.claude-sonnet-5-5 only, this role only
✓ allow✗ refuse
3. SCP on the workloads unitdenies global profiles (next step)
✓ allownot reached
a request must pass every check; the default endpoint policy would have allowed both
Fig 6.2Three checks a request passes in order. The accent check is the endpoint’s own policy, which the default leaves open.

An interface endpoint carries its own policy, and the default one allows every action on the service, by any principal. The shop replaces it: only the role shop-support-chat may call bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream, and only on the EU profile and Claude Sonnet 5.5.

A request through the endpoint now passes three checks, and each must allow it: the caller’s IAM policy, the endpoint policy, and the organization’s policy from the next step. A request naming global.anthropic.claude-sonnet-5-5 can pass IAM and still be refused here, before it leaves the VPC.

THE ENDPOINT POLICY THAT REPLACES THE DEFAULT
{
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::111122223333:role/shop-support-chat" },
    "Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"],
    "Resource": [
      "arn:aws:bedrock:eu-central-1:111122223333:inference-profile/eu.anthropic.claude-sonnet-5-5",
      "arn:aws:bedrock:*::foundation-model/anthropic.claude-sonnet-5-5"
    ]
  }]
}
STEP 03 / 05

ONE SCP FOR EVERY ACCOUNT

management account
AWS Organizations · 123456789012
workloads · unitSCP attached
shop-prod 111122223333 · the chat
shop-dev 444455556666 · developers
every role in both accounts is held to the SCP
STATEMENT 1 · DENY GLOBAL PROFILES
  • action bedrock:*
  • when aws:RequestedRegion = unspecified
  • and profile ARN matches global.*
no account can call global.anthropic.claude-sonnet-5-5
STATEMENT 2 · DENY REGIONS OUTSIDE THE EU
  • action * outside eu-* Regions
  • except requests with bedrock:InferenceProfileArn
so the eu. profile can still reach its six destinations
Fig 6.3One policy, every account. The accent statement is the one that makes global profiles impossible anywhere in the organization.

A (SCP) sets the most that any role may do in every account under the organizational unit it is attached to. It grants nothing; it only takes permissions away. The shop attaches two statements to its workloads unit, which holds shop-prod and shop-dev.

The first denies bedrock:* when aws:RequestedRegion is unspecified and the profile ARN matches global.*, which is how a global profile looks to a policy (chapter 01). No account can use one. The second denies every Region outside the EU, with an exception for requests that carry bedrock:InferenceProfileArn, so EU profiles still reach their destinations.

Blocking a new third-party model takes a deny on bedrock:InvokeModel. Denying aws-marketplace:Subscribe alone doesn’t stop the first call, which subscribes the account by itself.

THE STATEMENT THAT TURNS GLOBAL PROFILES OFF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyGlobalCrossRegionInference",
    "Effect": "Deny",
    "Action": "bedrock:*",
    "Resource": "*",
    "Condition": {
      "StringEquals": { "aws:RequestedRegion": "unspecified" },
      "ArnLike": { "bedrock:InferenceProfileArn": "arn:aws:bedrock:*:*:inference-profile/global.*" }
    }
  }]
}
STEP 04 / 05

A FILTER BETWEEN THE CUSTOMER AND CLAUDE

CUSTOMER PROFILE, AS SENT
order 48213 · Lyon
email marie.d@example.com
GUARDRAIL
sensitive information · mask
type EMAIL → replaced, order numbers kept
WHAT CLAUDE READS
order 48213 · Lyon
email {EMAIL}
shop-prod · eu-central-1 · one month
730 hours · 300,000 chats
sensitive-information filter$90.00
3 text units a chat × 300,000 × $0.10 per 1,000
would add · not in this bill$90.00
A SEPARATE CONTROL · REFUNDS
issue_refund above €100↓ the chat server startsAWS Step Functions workflow↓ waits fora person in the back office approves
this chat’s €64.90 refund is under the limit
Fig 6.4The accent box masks the customer’s email before it reaches Claude. The receipt shows what the filter would add.

A checks text before or after a model: denied topics, harmful-content filters, sensitive-information filters that mask or block personal data such as an email address, and checks that an answer is grounded in its sources. ApplyGuardrail runs a guardrail on any text without calling a model at all.

Bedrock bills guardrails per text unit of up to 1,000 characters: $0.15 per 1,000 units for content filters and denied topics, $0.10 for sensitive information, nothing for word and regex filters. The shop hasn’t added one. A sensitive-information filter on each message and reply, 3 text units a chat, would add $90.00 a month; it is not in this bill.

Refunds get a different control. When Claude asks for issue_refund above €100, the chat server doesn’t run it: it starts an AWS Step Functions workflow that waits until a person in the back office approves.

STEP 05 / 05

WHO CALLED WHICH MODEL, AND WHERE IT RAN

  • #1 InvokeEndpoint
    • CloudTrail default trail–
    • CloudTrail data eventsonly if enabled · no body
    • invocation log–
    • CloudWatch metrics✓ invocation counts
    • S3–
  • #2 Converse
    • CloudTrail default trail✓ role · inferenceRegion
    • CloudTrail data events–
    • invocation log✓ full text
    • CloudWatch metrics✓ invocation counts
    • S3–
  • #3 Converse
    • CloudTrail default trail✓ role · inferenceRegion
    • CloudTrail data events–
    • invocation log✓ full text
    • CloudWatch metrics✓ invocation counts
    • S3–
  • #4 CreateModelInvocationJob
    • CloudTrail default trail✓ CreateModelInvocationJob
    • CloudTrail data events–
    • invocation logbatch output instead
    • CloudWatch metrics✓ job metrics
    • S3✓ 10,000 records
CLOUDTRAIL HOLDS WHO AND WHERE · THE INVOCATION LOG HOLDS WHAT WAS SAID
Fig 6.5Four calls, four rows, a column per log. The accent cells are what the free default trail already holds.

CloudTrail’s default trail records calls #2 and #3 as management events, with the caller’s role and, in additionalEventData.inferenceRegion, the Region that computed each. It records nothing the customer wrote. The text of both calls is in the invocation log from chapter 02.

Call #1 is different. SageMaker’s InvokeEndpoint is a CloudTrail data event, which trails skip unless data events are turned on, and even then the request body isn’t logged. The endpoint’s CloudWatch metrics count it either way. Call #4 appears once, as CreateModelInvocationJob, and its 10,000 records sit in S3.

Across many accounts, an organization trail sends every account’s CloudTrail events to one log archive account. Invocation logs stay in the account and Region that made the calls, so they reach the archive by a log subscription or replication.

WHAT YOU NOW KNOW

  • Interface endpoints for bedrock-runtime and sagemaker.runtime keep all three live calls inside the VPC; with private DNS, no code changes.
  • An endpoint policy, IAM and one SCP denying Region “unspecified” keep every account’s chats on the EU profile.
  • Two endpoints in two Availability Zones add $35.20: the monthly bill is $5,386.41.
deep dive: centralized logging across accounts, human approval, and Guardrails pricing

Centralized logging. Invocation logs must go to a log group or bucket in the same account and Region as the calls, so collecting them centrally means a CloudWatch Logs subscription or S3 replication to a log archive account. CloudTrail organization trails cover management events in every account; data events, such as SageMaker’s InvokeEndpoint or Bedrock’s ApplyGuardrail and Retrieve, must be selected on the trail and are billed per event (SAP 2.3.2, 2.4.3).

Human approval. A Step Functions task that waits for a task token pauses the workflow until someone calls SendTaskSuccess or SendTaskFailure with that token, from a back-office page or an approval email. The model never holds the permission to refund; the workflow’s role does (SAP 2.5.3).

Guardrails details. A guardrail can be attached to a Converse call with guardrailConfig, or run on its own with ApplyGuardrail. Standard-tier guardrails use cross-Region inference within the geography, and a cross-Region guardrail profile adds no charge. ApplyGuardrail is a CloudTrail data event. Automated Reasoning checks cost $0.17 per 1,000 text units per policy.

Out of scope here. Access controls for AI agents (SAP 2.5.2, for example Amazon Bedrock AgentCore Identity) apply when a model acts with its own credentials. The shop’s Claude never does: every tool call runs in the shop’s code, under the chat server’s role.

On the exam

  • “calls to Bedrock must not traverse the internet”interface VPC endpoint for bedrock-runtime with private DNS · SAP 2.x
  • “only one approved model may be used from this VPC”an endpoint policy naming the model and profile ARNs
  • “no account in the organization may use global cross-Region inference”an SCP denying aws:RequestedRegion = unspecified for global.* profiles · SAP 1.2.1
  • “mask personal data before it reaches the model”a Bedrock Guardrails sensitive-information filter · SAP 2.5.1
  • “high-value actions by an AI agent need a human”an approval step in AWS Step Functions · SAP 2.5.3
  • “audit every prompt across all accounts in one place”invocation logging per account, shipped to a log archive account; an organization trail · SAP 2.3.2

$5,386.41 a month, line by line

One chat, four model calls, priced for a month of 300,000. Each line below was added by one chapter, whose number sits in the right-hand column. Claude is the largest line; the two GPUs that only sort chats are the second, at 81% of Claude’s cost, and they are busy about 3% of the time they are billed for.

SHOP-PROD · EU-CENTRAL-1 · ONE MONTH · 730 HOURS · 300,000 CHATS
BEDROCK
Claude Sonnet 5.5, global rates$2,490.00
9,750 tokens a chat × 300,000 chats
EU-only routing premium$249.00
10% on the same tokens
nightly summaries, Claude Haiku 4.5 batch$371.25
1,650 tokens a chat at half price
SAGEMAKER AI
triage endpoint, 2 × ml.g6.2xlarge$2,231.02
1,460 instance-hours · busy about 42 of them
NETWORK AND LOGS
interface endpoints, 2 services × 2 zones$35.20
2,920 endpoint-hours + 16.2 GB
prompt logs in Frankfurt$9.94
15 GB ingested, kept 30 days
monthly bill · 300,000 chats$5,386.41
per chat$0.018
WHERE THE FOUR CALLS RAN
  • Ireland
    eu-west-1 · Ireland
  • London
    eu-west-2 · UK
  • Stockholm#3
    eu-north-1 · Sweden
  • Paris#2
    eu-west-3 · France
  • Frankfurt#1
    eu-central-1 · Germanycalled from here
  • Zurich
    eu-central-2 · Switzerland
  • Spain
    eu-south-2 · Spain
  • Milan
    eu-south-1 · Italy
  • #1 InvokeEndpoint · shop-triage on ml.g6.2xlarge · eu-central-1
  • #2 Converse · eu.anthropic.claude-sonnet-5-5 · eu-west-3
  • #3 Converse · eu.anthropic.claude-sonnet-5-5 · eu-north-1
  • #4 CreateModelInvocationJob · eu.anthropic.claude-haiku-4-5-20251001-v1:0 · one of six EU Regions
one illustrative run · Bedrock picks a Region per request · all inside the EU
WHERE EACH EXAM LOOKS
Read next. AI numbers every AI engineer should know for why a GPU’s memory, not its arithmetic, sets how fast a model writes, and AWS KMS: how one invoice stays encrypted for seven years for the keys behind the encrypted logs.
stopReason: end_turn