ONE CLAIM · 6 LAYERS · 38 SPANS · 11 TOOL CALLS · 99 MINUTES PAUSED
An AI stack is the set of libraries that sit around a model call. The call itself is one HTTP request. A prompt goes out and text comes back, and the model keeps nothing in between. The seven rows above are what one insurance company put around that request, and each row owns exactly one job. This tutorial takes them in order, on a single claim, using real Python and real package names.
Each step carries a diagram, and most carry a code sample. open a short definition.
An AI stack is the set of libraries that sit around a model call. The call itself is one HTTP request. A prompt goes out, text comes back, and the model remembers none of it afterwards. Everything above that request is there because it has to run 4,000 times a night, on documents too large to send, with a person approving some of the answers. This chapter names the pieces the rest of the tutorial assumes you are already holding.
A is one HTTP POST to a single endpoint. The request carries a prompt and the name of a model. The response carries text and a count of , the chunks of about three quarters of a word each that everything on the bill is measured in. The model keeps nothing between one call and the next. Every layer above this request exists because of that last sentence.
import anthropic
client = anthropic.Anthropic()
# One HTTP POST. The prompt goes out, text comes back, the model keeps nothing.
message = client.messages.create(
model="claude-haiku-4-5",
max_tokens=1024,
messages=[{"role": "user", "content": claim_text + policy_clause}],
)
print(message.content[0].text) # 'Deny. Clause 14.2(b) excludes prior water damage.'
print(message.usage.input_tokens) # 18400
print(message.usage.output_tokens) # 2600
# 18,400 in at $1/M plus 2,600 out at $5/M is 3.1 cents, the whole bill for one claim.A is one library that owns one job about running that request many times. Layer 1 chooses which text the request carries. Layer 2 decides which request runs next and where the answer is written down. Layer 3 handles a step whose number of calls nobody knows in advance. Layers 4, 5 and 6 cover what happened, whether a change helped, and what starts at 02:00. None of the six makes the model better at reading a policy.
One event runs through every chapter. On Tuesday 28 July 2026 a file called claim-8842.pdf lands in a storage bucket at 09:41:02. It is 31 pages, the repair estimate is $12,400, and this insurer needs a person to approve anything over $10,000. The decision is written back at 11:20:41. Of those 99 minutes and 39 seconds, 12.6 seconds used a processor and the rest was spent as a row in a database. The whole thing cost 14 cents.
A is one timed operation inside a run. It carries a name, a parent, a start and an end time, and a set of attributes. A is the whole tree of spans for one run. Both come from OpenTelemetry, which was built for microservices years before anyone pointed it at a model. Claim 8842 emits 38 spans, made of 7 from layer 1, 9 from layer 2 and 22 from layer 3. That count is the number this tutorial tracks, and until chapter 05 nothing stores a single one of them.
Every layer here is a package you can install this afternoon. LangChain and LangGraph are MIT licensed and maintained by LangChain Inc. Langfuse is MIT licensed, was founded in 2023 and was bought by ClickHouse in January 2026; self-hosting it is a docker compose file rather than an enterprise conversation. Airflow belongs to the Apache Software Foundation. The Anthropic SDK and the Claude Agent SDK come from Anthropic, and those two need an API key, because a model call is the only part of this stack that bills anybody.
# Layer 0: the HTTP client for the model
pip install anthropic
# Layers 1 and 2: providers, typed output, retrieval, the state graph
pip install langchain langchain-anthropic langgraph langchain-postgres
# Layer 3: the agent loop that runs Claude Code, as a library
pip install claude-agent-sdk
# Layers 4 and 5: traces, datasets, scores
pip install langfuse
# Layer 6: the scheduler
pip install apache-airflow
# Five of the six are open source and run on one laptop. Only the model
# call needs an account:
export ANTHROPIC_API_KEY=sk-ant-api03-EXAMPLEWhat a layer costs before it does anything. An import is the cheap part. Every tool the agent in layer 3 may call sits in the system prompt as a schema, on every turn, whether or not it gets used. Every layer is also one more package to pin, upgrade and read the changelog of. The six here add roughly 40 transitive dependencies to a script that had one.
Framework and layer are different words. A framework is a package you install. A layer is a job that has to be done by something. LangChain 1.0 and LangGraph 1.0 are two packages covering two layers, and they shipped on the same day from the same release, because the second is the runtime the first runs on.
Where 38 comes from. Layer 1 emits 7 spans per claim: extract, embed, search, rerank, call, parse, retry. Layer 2 emits 9, one for each node entered and each checkpoint written. Layer 3 emits 22, one per tool call plus the model turns between them. That is 38 a claim and 152,000 a night, and every figure later in the tutorial is built on those three numbers.
Nobody designed this stack. Airflow was started in 2014, LangChain in 2022, Langfuse in 2023, LangGraph in 2024, the Claude Agent SDK in 2025. Each arrived for a reason that had nothing to do with the others, and the six ended up in one requirements file because one Tuesday needed all of them.
Sixty-two lines of Python, one dependency, and it decided claim 8842 correctly on the first try. It reads the PDF, pastes the text into one model call, prints a decision, and exits. That took 8.4 seconds and cost 3.1 cents. It is also the version that cannot survive tonight, and this chapter names the reasons one at a time. Layers so far: 0. Spans in the trace: 1.
The whole first version: open the PDF, extract 31 pages of text, paste it and one hand-picked policy section into a single messages.create call, print the decision, exit. One import. It runs in 8.4 seconds on 18,400 input tokens and costs 3.1 cents. It is correct on claim 8842, and correct on the next forty. Nothing in this tutorial is here because that script is bad.
The policy section was picked by hand, and that is the part which does not survive. The full policy set is 40,000 pages, about 26,000,000 tokens at 650 tokens a page. A is the largest number of tokens one request may contain, and the biggest on the price list is 200,000. That holds 0.8% of the policy set, so the corpus is 130 times too large to paste. The model is not the constraint here. Somebody has to choose eight paragraphs out of 40,000 pages before the model is called at all.
The script asks for JSON in the prompt and calls json.loads on whatever comes back. About one run in fifty returns a sentence before the opening brace, or a trailing comma, and throws. At 4,000 claims a night that is 80 crashes. A better prompt does not fix this. does, and it means constraining generation with a schema instead of asking politely: the model is handed a shape and can only emit something that fits it. Every vendor supports this and every vendor spells it differently.
import json
# The script's version: ask for JSON in the prompt and hope for JSON back.
reply = client.messages.create(model="claude-haiku-4-5", max_tokens=1024, messages=prompt)
decision = json.loads(reply.content[0].text) # JSONDecodeError on ~1 run in 50
# The version that cannot throw: the shape is a schema the model is made to fill.
RECORD = {
"name": "record_decision",
"strict": True,
"input_schema": {
"type": "object",
"properties": {"approve": {"type": "boolean"}, "clause": {"type": "string"}},
"required": ["approve", "clause"],
"additionalProperties": False,
}}
reply = client.messages.create(
model="claude-haiku-4-5", max_tokens=1024, messages=prompt, tools=[RECORD],
tool_choice={"type": "tool", "name": "record_decision"})
decision = reply.content[0].input # {'approve': False, 'clause': '14.2(b)'}Providers answer 529 when they are overloaded and 429 when the caller is over its rate limit, and the script handles neither. A safe retry here is four decisions rather than one line. Wait longer after each failure. Add jitter, so 4,000 clients do not all come back at the same instant. Cap the number of attempts. Send an , a stable id the provider matches repeat requests against, so a retried claim is never paid twice. Four attempts at 1, 2, 4 and 8 seconds put 15 seconds between a bad minute and an answer.
The roadmap, as a list rather than a promise. It cannot search 40,000 pages. It cannot stop for a human and start again tomorrow. It cannot run a step whose length nobody knows in advance. It cannot tell you what happened at 03:14. It cannot prove that a prompt change helped. It cannot run at 02:00 on its own. Six gaps, six layers, and each one is a different category of tool.
The token bill of the naive version. 18,400 input tokens per claim across 4,000 claims is 73.6 million tokens a night, and that is with the policy clause hand-picked. Pasting the whole policy set is not expensive, it is impossible: 26,000,000 tokens do not fit in a 200,000-token request at any price.
Why a bigger window is not the answer. Window sizes keep growing, and recall inside a long window does not grow with them. A model handed 400,000 tokens of policy is measurably worse at finding clause 14.2(b) than a model handed the eight paragraphs that contain it. The arithmetic on price per token is in the AI numbers tutorial.
The honest defence of the script. Under about 50 runs a day, with a person reading every output, none of the six layers pay for themselves. The retry is a try/except, the trace is a print statement, and the eval set is whoever is reading. This is why engineers who insist frameworks are overhead are usually describing a real system they really ran.
Two problems arrive together at layer 1. Four vendors spell the same request four different ways, and the policy set is 26,000,000 tokens against a 200,000-token limit. LangChain is one answer to both, and it was not a well-liked one for three years. Version 1.0 shipped on 22 October 2025 by moving most of the library out into a package called langchain-classic, and what is left is the message format, the tool schema, the retry policy and the retrieval pipeline. The claim goes from one span to seven.
The part that earns its place is dull and load-bearing. One message object, whatever the vendor. One tool-schema format. One retry and timeout policy. And a , meaning a single interface over several model vendors, so swapping Anthropic for Bedrock or Vertex is a change of string rather than a rewrite. Vendors disagree about where the system prompt goes, how a tool call is encoded, what the streaming events are called, and which field holds the token count. That disagreement is about 300 lines of code, and this is where it lives. LiteLLM does this one job and nothing else.
from langchain.chat_models import init_chat_model
from pydantic import BaseModel, Field
class Decision(BaseModel):
approve: bool = Field(description="True when the claim is payable as filed")
clause: str = Field(description="The policy clause the decision rests on")
# The vendor is one string. Bedrock and Vertex are this line, edited.
model = init_chat_model("anthropic:claude-haiku-4-5", max_retries=4, timeout=60)
# model = init_chat_model("bedrock_converse:anthropic.claude-haiku-4-5")
# Typed output happens inside the same call, not in a second one.
decision = model.with_structured_output(Decision).invoke(prompt)
print(decision.approve, decision.clause) # False 14.2(b)The other half of the layer is loaders, splitters, an and a vector store. An embedding is a list of numbers standing for a piece of text, arranged so that texts about the same subject land near each other. The 40,000 pages are cut into of about 400 tokens with 80 tokens of overlap, which gives roughly 81,000 of them. At 1,536 numbers per chunk and 4 bytes per number, the that answers “which of these are nearest to my question” is half a gigabyte. Retrieving the nearest 8 puts 3,200 tokens in the prompt instead of 26,000,000. Anthropic ships no embedding model, so this is where a second vendor turns up.
from langchain_openai import OpenAIEmbeddings
from langchain_postgres import PGVector
from langchain_text_splitters import RecursiveCharacterTextSplitter
# 400-token pieces with 80 tokens of overlap, counted here in characters.
splitter = RecursiveCharacterTextSplitter(chunk_size=1600, chunk_overlap=320)
chunks = splitter.split_documents(policy_pages) # 40,000 pages become 81,043 chunks
# Anthropic ships no embedding model, so this layer talks to a second vendor.
store = PGVector(
embeddings=OpenAIEmbeddings(model="text-embedding-3-small"), # 1,536 dimensions
collection_name="policy",
connection="postgresql+psycopg://localhost:5432/claims",
)
store.add_documents(chunks) # 81,043 x 1,536 x 4 bytes = 0.50 GB in Postgres
passages = store.similarity_search("prior water damage exclusion", k=8)
# 3,200 tokens go into the prompt instead of 26,000,000The library doing those two jobs has a longer history than the jobs do. Harrison Chase wrote the first version between 16 and 25 October 2022, as a side project, six weeks before ChatGPT existed. By mid-2026 the repository has passed 140,000 stars and the Python and JavaScript packages average around 90 million downloads a month. The reputation for heavy abstractions and the download count are both real, and they are about different things. The first is a statement about the 2023 API. The second is about who is still shipping on it.
Version 1.0 arrived on 22 October 2025 and cut the package down, moving the rest to langchain-classic. create_agent replaced the old prebuilt agent and runs on the LangGraph runtime. Typed output moved into the agent loop itself, so a structured answer no longer costs a second model call. Middleware landed for human approval, summarization and PII redaction. Python 3.10 is the floor, and the promise is no breaking changes until 2.0.
The bill for layer 1 is that the claim now costs seven operations instead of one. Extract the PDF, embed the question, search the index, rerank, call the model, parse the typed answer, retry once. Wall clock drops from 8.4 to 5.2 seconds because the prompt got smaller. There are now seven places to be slow and seven places to be wrong, and no way to look at any of them. Layers: 1. per claim: 7. Spans you can read: still 0.
The unbundled version of this layer. LiteLLM for provider routing only. Instructor or BAML for typed output only. LlamaIndex for document ingestion and indexing. Haystack as the older full-pipeline option. A 2026 stack built from four narrow libraries instead of one wide one is a normal choice, and the trade is more imports against fewer surprises.
What the abstraction complaint was actually about. The old high-level chains assembled the prompt for you and did not show it, so debugging a bad answer meant reading library source to find out what was sent. create_agent plus middleware makes both the prompt and the loop inspectable, which is the specific thing that changed.
Which package is which. langchain-core holds the interfaces and depends on almost nothing. langchain holds the agent and middleware. Provider packages such as langchain-anthropic hold one vendor each. langchain-classic holds everything 1.0 stopped shipping.
Where the retrieval detail lives. Chunk sizes, overlap, hybrid search, reranking and citation are the subject of the RAG tutorial. The boring default for the store is pgvector, because the policy index is 0.5 GB and the team already runs Postgres.
Three things do not fit inside a function call. A branch that depends on the model’s answer. A step that waits 99 minutes for a person in a different building. A crash at second 8 of a 12-second run that you would rather not pay for twice. LangGraph answers all three by treating the feature as a state machine with a durable store behind it. The claim goes from seven spans to sixteen, and nine of the new ones are bookkeeping.
A is a program written as three declarations. First a typed state object, which is the only thing that travels between steps. Then , plain functions that take the whole state and return just the part they changed. Then edges, which decide what runs next. Conditional edges branch on the state, which is how “$12,400 is over $10,000” becomes routing instead of an if-statement buried inside a function. The claim graph has six nodes: ingest, retrieve, decide, threshold, await_approval, investigate. LangGraph appeared in early 2024 and reached 1.0 in October 2025.
from typing import TypedDict
from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.graph import END, START, StateGraph
class ClaimState(TypedDict): # the only thing that travels between nodes
claim_id: str
amount_cents: int
decision: str
def over_threshold(state: ClaimState) -> str:
# "$12,400 is over $10,000" is routing here, not an if-statement inside a node.
return "await_approval" if state["amount_cents"] > 1_000_000 else END
builder = StateGraph(ClaimState)
builder.add_node("decide", decide) # each node is an ordinary function
builder.add_node("await_approval", await_approval)
builder.add_edge(START, "decide")
builder.add_conditional_edges("decide", over_threshold)
with PostgresSaver.from_conn_string(DB_URI) as saver:
graph = builder.compile(checkpointer=saver) # state is written after every nodePersistence is what changes the range of possible failures. With a configured, the state is written to durable storage after every step, filed under a thread id. Kill the process after node 3 of 6 and the rerun starts at node 4, so the retrieval and the $0.09 decision call are not paid for twice. Three checkpoint writes show up in this claim’s trace. In-memory checkpointers exist for tests, and are exactly as durable as the process.
The approval step calls interrupt(). The state is written and the run ends, so nothing is left waiting. At 11:20:14 the approver’s decision is passed back with the same thread id, the node runs again from the checkpoint, and execution continues. From 09:41:09 that is 99 minutes and 5 seconds using no memory, no connection and no compute. Every product feature labelled “human in the loop” is this plus a queue for the person.
from langgraph.types import Command, interrupt
def await_approval(state: ClaimState) -> dict:
# Writes the state and ends the run. Nothing is left waiting anywhere.
answer = interrupt({"claim": state["claim_id"], "cents": state["amount_cents"]})
return {"decision": answer}
thread = {"configurable": {"thread_id": "claim-8842"}}
# 09:41:09. Runs as far as await_approval, then the process exits.
graph.invoke({"claim_id": "8842", "amount_cents": 1_240_000}, thread)
# 11:20:14, a different process, the same thread id. interrupt() returns "approve".
graph.invoke(Command(resume="approve"), thread)The honest limit of this layer, and the reason the next paragraph names a different tool. A checkpoint records where the run stopped. Nothing in the graph library detects that a run stopped, retries it, or times it out: if the process dies, the state is safe and the run is simply not running until something outside notices. engines invert that. The engine owns the run, retries an activity itself, accepts a signal carrying the human’s answer, and treats a sub-agent as a child workflow.
The alternatives, each with the reason someone picks it. The OpenAI Agents SDK for a model-driven loop with 100 or more models behind LiteLLM. CrewAI for role-based multi-agent prototypes. Pydantic AI when every input and output should be a validated model. Google’s ADK for Gemini-first teams. Temporal when durability matters more than graph ergonomics. Plain Python when the graph has four steps and never branches. LangGraph’s own case is its deployment list and about 39 million PyPI downloads a month. Layers: 2. Spans per claim: 16.
Threads, and rewinding one. Every run is a thread of checkpoints, so a run can be rewound to an earlier step and replayed with a changed input. That is how a disputed decision gets reproduced instead of argued about: load thread claim-8842, rewind to decide, change the clause, run forward.
Reducers, and the bug they prevent. Each state key declares how updates combine: messages appends, amount overwrites. Two nodes writing the same key in one step is a real error, and the reducer is where the developer states what the framework should do about it.
Serialization is the tax. Everything in state has to be storable, which rules out open file handles, live clients and lambdas. The usual fix is to keep ids in state and look the objects up inside the node, which is more code and one fewer way to lose a run.
Why 1.0 landed on the same day. create_agent in LangChain 1.0 is a prebuilt graph running on this runtime, so the two packages are one release. Reaching for LangChain’s agent and reaching for LangGraph are the same decision at different levels of detail.
Node 6 is called investigate and nobody can draw it. Checking whether a repair shop has a history means reading prior claims, then possibly the adjuster’s note, then possibly nothing else, and the number of steps depends on what the first step finds. That is a loop, not a graph, and it is where the second kind of tool lives. One node adds 22 spans to this claim.
What investigate has to do: look up prior claims on the policy, check the repair shop against past claims, read the adjuster’s note, and stop once there is enough to write one paragraph. On claim 8842 that took 11 tool calls. On the next claim it took 3. Drawing this as a graph means drawing every path in advance, and the number of paths is the number of orderings of five tools, not the number of tools. A sequence whose length is decided while it runs is an .
A is a library that already contains the agent loop, the tool execution and the context management, so what you supply is tools and rules rather than control flow. The Claude Agent SDK is the harness behind Claude Code, packaged for Python and TypeScript. You supply tools, a system prompt, and a policy about what is allowed. It also loads skills, slash commands and memory from a project’s .claude/ directory, and reaches external systems over MCP. From another language, run the CLI as a subprocess with -p and JSON output.
import anyio
from claude_agent_sdk import ClaudeAgentOptions, query
options = ClaudeAgentOptions(
model="claude-haiku-4-5",
system_prompt="Check this repair shop against prior claims. Stop when one paragraph is supported.",
allowed_tools=["Read", "Grep", "mcp__claims__prior_claims"],
max_turns=20,
)
async def investigate(claim_id: str) -> str:
messages = []
async for message in query(prompt=f"Claim {claim_id}", options=options):
messages.append(message)
return messages[-1].result # 11 tool calls on claim 8842, 3 on the next one
anyio.run(investigate, "8842")In a graph you decide what happens next. In a harness you decide what is allowed to happen at all. A marks each tool as automatic, needing approval, or unavailable. Hooks run your own code at lifecycle points, and PreToolUse is the one that matters, because it is the last place to refuse a call before it happens. In this system, reading prior claims is automatic, emailing the policyholder asks first, and writing to the claims database is not on the agent’s list at all.
from claude_agent_sdk import ClaudeAgentOptions, HookMatcher
async def refuse_claims_writes(input_data, tool_use_id, context):
# PreToolUse runs after the model has asked and before anything happens.
if input_data["tool_input"].get("table") == "claims":
return {"hookSpecificOutput": {
"hookEventName": "PreToolUse",
"permissionDecision": "deny",
"permissionDecisionReason": "This agent never writes to the claims database.",
}}
return {}
options = ClaudeAgentOptions(
allowed_tools=["Read", "Grep", "mcp__claims__prior_claims"], # runs automatically
disallowed_tools=["mcp__claims__write_row"], # not on the list at all
hooks={"PreToolUse": [HookMatcher(matcher="mcp__claims__.*", hooks=[refuse_claims_writes])]},
)Two mechanisms keep a long run inside the window. A is a separate agent instance for one focused subtask, with its own context, so “read 40 prior claims and return one line” does not put 40 claims in the main transcript. summarizes older messages as the window fills, without being asked. The investigation read 61,000 tokens and handed back 240. Without the subagent the main run would carry all 61,000 into every later turn, and pay for them again each time.
The rule, stated once. Known steps, an audit requirement and a typed output mean a graph. Unknown steps, open-ended tool use and an answer in prose mean a harness. This system does both. Node 6 of the LangGraph graph opens an Agent SDK session, and that session returns a typed finding back into the graph state. The alternatives at this layer are the OpenAI Agents SDK and Google’s ADK. Anthropic’s Managed Agents runs the same loop as a hosted service instead of a library. Layers: 3. Spans per claim: 38.
Sessions, and forking one. A session can be resumed later or forked in place, which is the cheap way to try two investigation strategies on the same claim and keep whichever one found the prior claims. Forking costs a copy of the transcript and nothing else.
The hooks people actually use. There are 25 lifecycle points and three carry most of the traffic: PreToolUse to refuse a call, PostToolUse to log what a call returned, and PreCompact to keep a copy of the transcript compaction is about to summarize.
Tool definitions cost tokens before anything runs. Every tool the agent may call sits in the system prompt as a schema, on every turn. The measured bill for one popular MCP server is in the MCP tutorial, and it is larger than most people expect.
The adjuster’s note is untrusted text. An agent that reads free-form text written by someone outside your company, and that also holds tools, is the setup described in the prompt-injection tutorial. The permission table above is the mitigation, not the prompt.
At 03:14 last night 41 claims out of 4,000 came out wrong, which is one percent. Four layers ran on every one of them and one of the four is responsible. The runs emitted 38 spans each and nobody can read a single one. Emitting a span and storing a span are different jobs, and nothing in the stack so far does the second one.
Nothing is added to the claim’s execution in this chapter. LangChain, LangGraph and the Agent SDK have been emitting since chapter 02, and with no receiver configured they went nowhere. Turning tracing on is a client and an endpoint, not a rewrite. The waterfall for claim 8842 holds 7 spans from layer 1, 9 from layer 2 and 22 from layer 3, over 12.6 seconds end to end once the 99-minute pause is taken out.
# The pipeline has been emitting spans since chapter 02 with nowhere to send them.
# This is the whole change: two keys, one endpoint, one header.
export LANGFUSE_PUBLIC_KEY=pk-lf-1a2b3c4d
export LANGFUSE_SECRET_KEY=sk-lf-5e6f7a8b
export LANGFUSE_HOST=https://langfuse.claims.internal
AUTH=$(printf '%s:%s' "$LANGFUSE_PUBLIC_KEY" "$LANGFUSE_SECRET_KEY" | base64)
export OTEL_EXPORTER_OTLP_ENDPOINT="$LANGFUSE_HOST/api/public/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $AUTH"
# No Python was edited. The next run stores 38 spans per claim, 152,000 a night.
python -m claims.nightlyWhat is recorded: a name, a parent, a start and end time, and attributes. The standardize the AI-specific ones, including gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.response.finish_reasons. Prompt and completion text is opt-in, because a claim PDF in a span attribute is a claim PDF in your logging vendor. As of v1.42.0 on 12 June 2026 the whole gen_ai group moved to its own repository, and it is still pre-stable.
{
"name": "chat claude-haiku-4-5",
"trace_id": "8842f01c9b3e4a7d",
"span_id": "c37a09e1",
"parent_span_id": "b41d6e02",
"start_time_unix_nano": 1785231669412000000,
"end_time_unix_nano": 1785231671034000000,
"attributes": {
"gen_ai.operation.name": "chat",
"gen_ai.provider.name": "anthropic",
"gen_ai.request.model": "claude-haiku-4-5",
"gen_ai.usage.input_tokens": 18400,
"gen_ai.usage.output_tokens": 2600,
"gen_ai.response.finish_reasons": ["end_turn"],
"langfuse.observation.cost_details.total": 0.031
}
}Langfuse was founded in 2023 by Marc Klingen, Maximilian Deichmann and Clemens Rawert, went through Y Combinator’s W23 batch, is MIT licensed, and treats self-hosting as a normal deployment rather than an enterprise favour. Its own SDKs are built on OpenTelemetry and it accepts OTLP at /api/public/otel, so anything that speaks the protocol can send to it, including stacks with no LangChain anywhere in them. It also stores prompts under version control, outside your repository.
A stored turns into per-run cost, p95 latency, token counts per layer, and the ability to filter to the 41 wrong claims and read them. Claim 8842: $0.14, 12.6 seconds of compute, 86,000 tokens, of which 61,000 belong to the investigation. Across 4,000 claims that is $560 a night and $16,800 a month, and layer 3 is 84% of it. Nobody guessed that split before the traces existed.
Langfuse is MIT licensed with self-hosting as an ordinary deployment. LangSmith is a proprietary backend with the deepest zero-setup LangChain and LangGraph integration, and self-hosting only on enterprise plans. Arize Phoenix is open source and runs locally or in a container. Braintrust ties tracing to datasets and CI gates in one product. General tracing vendors now read the same gen_ai attributes, so a backend you have run for years may already qualify. Layers: 4. Spans you can read: 38.
The volume, and what sampling costs. 152,000 spans a night is a small number for a tracing backend and a large one for a Postgres table. Head sampling at 10% throws away nine of every ten rare failures, which are the rows you turned tracing on for. The usual answer is to keep every error and a percentage of successes.
Content capture crosses a boundary. Span attributes holding prompts and completions carry the claimant’s name, the estimate and the adjuster’s note into whatever stores the traces. The opt-in default exists for that reason, and redaction at the SDK is cheaper than a data-processing agreement.
Prompts outside the repository. Storing prompts in the platform lets a prompt change ship without a deploy, which is the upside. The downside is that a checkout of the repository no longer reproduces production, so the version id belongs in the trace.
Why a database company bought a tracing tool. At this volume the hard part is the store rather than the UI. High-cardinality columns, cheap aggregate queries, retention measured in months. Langfuse was acquired by ClickHouse in January 2026, which is the same observation with money behind it.
A prompt change might help and might make things worse, and nothing built so far can tell the difference in advance. An engineer on the claims team proposes swapping one word, from summarize to quote, because 9 of the 41 wrong claims paraphrased a clause instead of citing it. The trace says what happened last night. It cannot say whether tomorrow gets better. Those are two questions answered from the same stored rows, which is why one product usually answers both.
An here is 180 real claims pulled out of stored traces with the correct decision attached: the 41 that came out wrong, plus 139 sampled from the ones that came out right, so a fix cannot break those without anyone noticing. The traces are the raw material, which is the whole reason the two products are one product. Synthetic test cases exist and are worse, because they never contain the adjuster who writes in capitals.
Where there is a right answer, score it in code. The decision matched or it did not. The cited clause number matched or it did not. Both checks run over 180 items in under a second and cost nothing. Where there is no answer key, against a written rubric. That costs one model call per item, and it has to be checked against human labels before anyone believes it. On these 180 items, two checks are exact and one is a judge.
from langfuse import Langfuse
langfuse = Langfuse()
dataset = langfuse.get_dataset("claims-180")
def clause_matches(output: dict, expected: dict) -> int:
# Exact check, written in code. All 180 items scored in under a second, for nothing.
return int(output["clause"] == expected["clause"])
def quotes_rather_than_paraphrases(output: dict) -> int:
# A judge. One model call per item, so 180 calls and about 40 seconds.
verdict = judge_model.invoke(RUBRIC.format(answer=output["reason"]))
return int(verdict.text().strip() == "QUOTES")
for item in dataset.items:
output = decide(item.input)
langfuse.create_score(name="clause", value=clause_matches(output, item.expected_output))
langfuse.create_score(name="quotes", value=quotes_rather_than_paraphrases(output))Run both prompt versions over the same 180 items and compare. Changing summarize to quote fixed 6 of the 9 paraphrase failures and broke 9 claims that had been correct, because the new word made the model cite a clause even when none applied. Net worse by 3, so it does not ship. The gate belongs in CI, failing the branch when the score drops, in the same place a failing test fails.
This is not a separate vendor category, and the reason is structural: an eval needs stored inputs, stored outputs and somewhere to put scores, and a tracing product already has all three. Langfuse, LangSmith and Braintrust ship both halves. DeepEval and Ragas run as test libraries in CI with no platform behind them, and Ragas carries retrieval-specific metrics. Online scoring closes the loop by running the same scorer on a sample of live traffic. Layers: 5.
180 items is a small number. A two-item difference between versions is noise, and a fifteen-item difference is not. Running the same fixed set every time beats sampling fresh claims, because a moving set turns every comparison into an argument about which claims were harder.
The judge drifts. The grader is a model, the model gets upgraded, and last month’s scores stop being comparable. Pin the judge’s model version and record it beside the score, or accept that the eval history is a chart of two things changing at once.
The set grows from incidents. Every wrong claim someone complains about is one more row with a known answer. This is the only artifact in the tutorial that gets more valuable with age, and the only one worth backing up separately from the database it lives in.
Everything about rubrics and shipping decisions is in the evals tutorial, which follows one prompt change from complaint to deploy. The only question here is which layer owns the scoring.
Three things went wrong while nobody was looking. The policy team changed 312 pages yesterday and the index still holds the old text. The 180-item eval set has not run since Friday. Last night’s 4,000 traces are sitting in a table nobody has read. Every layer so far starts when a claim arrives, and none of these start at all. This one runs when the clock says so, and that difference is the whole reason a 2014 data-engineering tool is sitting in an AI stack.
The tool for this job is a scheduler running a , which is a set of tasks and the dependencies between them, with no cycles, written as ordinary Python. LangGraph and Airflow are both directed graphs of steps, which is why people confuse them, and they answer different questions. A graph run starts because a claim arrived, carries typed state between nodes, and can pause for a person. A DAG run starts because a , a five-field expression naming the minutes and hours a job fires on, said 02:00. It passes data through storage rather than state, and it is expected to be repeatable for any past date.
The nightly run, five tasks. Pull yesterday’s 4,000 traces. Re-embed the 312 policy pages that changed and upsert them into the index. Rebuild what depends on them, run the 180-item eval set against the current prompt, and post a scorecard. 41 minutes end to end, of which re-embedding is 31. Two tasks retry on failure, one carries an SLA, and the whole run is repeatable for any past date. Re-embedding all 40,000 pages instead of the 312 that changed would take 66 hours, so a , meaning a re-run of the same job for a date in the past, is the feature here rather than the schedule.
import pendulum
from airflow.sdk import dag, task
# The schedule is a clock, not a claim. catchup=False means backfill on purpose only.
@dag(schedule="0 2 * * *", catchup=False,
start_date=pendulum.datetime(2026, 6, 1, tz="UTC"))
def claims_nightly():
@task(retries=2)
def reembed_changed_pages(data_interval_start=None) -> int:
pages = policy_pages_changed_since(data_interval_start) # 312 of 40,000
store.add_documents(split(pages), ids=chunk_ids(pages)) # upsert by id, 31 minutes
return len(pages)
@task(retries=2)
def run_eval(page_count: int) -> float:
return score_dataset("claims-180") # 4 min, then post the scorecard
run_eval(reembed_changed_pages()) # 41 min across all five tasks
claims_nightly()The scheduler doing that work is older than everything above it. Maxime Beauchemin started Airflow in October 2014 at Airbnb, open source from the first commit, announced in June 2015, an Apache incubator project in March 2016 and a top-level Apache project in January 2019. None of it was designed for models. The transformer paper came out two years and eight months after the first commit.
Airflow 3.0 became generally available on 22 April 2025, the largest release since 2.0 in 2020. It brought DAG versioning, a task execution API that lets a task run outside the scheduler’s environment and in another language, event-driven scheduling, and an asset-centric interface. The change that matters here is the removal of the rule that every run needs a unique execution date, which is what let inference and training jobs fit without workarounds. 3.3.0 shipped on 6 July 2026.
Prefect and Dagster answer the same scheduled-batch question, Dagster by making the produced asset the unit rather than the task. Temporal answers a different question: a that owns one long-running execution, retries its activities, and takes a signal when a human replies, which is chapter 03’s problem and not this one. Plenty of platforms run a scheduler and a durable engine side by side, because the two jobs really are different. Layers: 6. Spans a night: 152,000.
Every task here runs twice sooner or later. A retried embedding upsert has to overwrite by chunk id, not append, or the index grows duplicate chunks that outrank the original and change what the model reads. This is the same idempotent requirement as chapter 01’s retry, one layer up.
Backfilling 90 days is 90 runs. Repairing an index after a bad embedding model means re-running the DAG once per past date, not once with a date range, because each run’s output is scoped to its own day. That property is the reason the schedule and the run are separate concepts.
The failure that actually happens. The eval task passes while the embedding task produced an empty index, so the scorecard says everything is fine and the claims get worse in the morning. Freshness checks on the asset catch this; retries and alerts on the crash do not, because there was no crash.
The option nobody writes down. A cron entry and a shell script. It is the right answer until the first backfill, the first task that has to run somewhere else, or the first time someone asks which version of the index produced Tuesday’s decisions.
Claim 8842 arrived at 09:41:02 and was written back at 11:20:41, having crossed six layers, emitted 38 spans, spent 99 of those minutes as a row in a database, and cost 14 cents. Three boards follow. The first is the map: a question per row, with every product name beside it an answer to that one question. The second is the practical version, keyed on what is going wrong rather than on what the layer is called. The third lists every question this tutorial set out to answer, and where it was settled.