ONE CLAIM · 6 LAYERS · 38 SPANS · 11 TOOL CALLS · 99 MINUTES PAUSED
Six libraries ran on this claim and not one of them made the model better. Retrieval, because the policy is 40,000 pages. A state graph, because the run stops for 99 minutes to ask a person. An agent loop because one step has no fixed length, traces because 41 claims came out wrong at 03:14, and a scheduler because the index goes stale while nobody is watching. The scenes play themselves so watch or just scroll. are tappable. You are the claim.
Sixty-two lines of Python, one dependency, and it decided claim 8842 correctly on the first try. It reads the PDF, pastes the text into one model call, prints a decision, and exits. That took 8.4 seconds and cost 3.1 cents. It is also the version that cannot survive tonight, and this chapter names the reasons one at a time. Layers so far: 0. Spans in the trace: 1.
The whole first version: open the PDF, extract 31 pages of text, paste it and one hand-picked policy section into a single messages.create call, print the decision, exit. One import. It runs in 8.4 seconds on 18,400 input tokens and costs 3.1 cents. It is correct on claim 8842, and correct on the next forty. Nothing in this tutorial is here because that script is bad.
The policy section was picked by hand, and that is the part which does not survive. The full policy set is 40,000 pages, about 26,000,000 tokens at 650 tokens a page. A 200,000-token holds 0.8% of it, so the corpus is 130 times too large to paste. The model is not the constraint here. Somebody has to choose eight paragraphs out of 40,000 pages before the model is called at all.
The script asks for JSON in the prompt and calls json.loads on whatever comes back. About one run in fifty returns a sentence before the opening brace, or a trailing comma, and throws. At 4,000 claims a night that is 80 crashes. A better prompt does not fix this. does, through a tool schema or a provider’s typed-output mode, and each vendor spells it differently.
Providers answer 529 when they are overloaded and 429 when you are over your rate limit, and the script handles neither. A retry that is safe here is four decisions rather than one line: exponential backoff, jitter so 4,000 clients do not all come back at the same instant, a cap, and an so a retried claim is not paid twice. Four attempts at 1, 2, 4 and 8 seconds put 15 seconds between a bad minute and an answer.
The roadmap, as a list rather than a promise. It cannot search 40,000 pages. It cannot stop for a human and start again tomorrow. It cannot run a step whose length nobody knows in advance. It cannot tell you what happened at 03:14. It cannot prove that a prompt change helped. It cannot run at 02:00 on its own. Six gaps, six layers, and each one is a different category of tool.
The token bill of the naive version. 18,400 input tokens per claim across 4,000 claims is 73.6 million tokens a night, and that is with the policy clause hand-picked. Pasting the whole policy set is not expensive, it is impossible: 26,000,000 tokens do not fit in a 200,000-token request at any price.
Why a bigger window is not the answer. Window sizes keep growing, and recall inside a long window does not grow with them. A model handed 400,000 tokens of policy is measurably worse at finding clause 14.2(b) than a model handed the eight paragraphs that contain it. The arithmetic on price per token is in the AI numbers tutorial.
The honest defence of the script. Under about 50 runs a day, with a human reading every output, none of the six layers pay for themselves. The retry is a try/except, the trace is a print statement, and the eval set is you. This is why engineers who insist frameworks are overhead are usually describing a real system they really ran.
A model provider is one HTTP endpoint, so the layer that talks to it should be thin. LangChain was not thin for three years and earned a reputation for it. Version 1.0 shipped on 22 October 2025 by moving most of the library out into a package called langchain-classic, and what is left is worth knowing: one message format, one tool schema, one retry policy, and the retrieval pipeline that turns 40,000 pages into eight paragraphs. The claim goes from one span to seven.
Harrison Chase wrote the first version between 16 and 25 October 2022, as a side project, six weeks before ChatGPT existed. By mid-2026 the repository has passed 140,000 stars and the Python and JavaScript packages average around 90 million downloads a month. The reputation for heavy abstractions and the download count are both real, and they are about different things: the first is a statement about the 2023 API, the second about who is still shipping on it.
Version 1.0 arrived on 22 October 2025 and cut the package down, moving the rest to langchain-classic. create_agent replaced the old prebuilt agent and runs on the LangGraph runtime. Typed output moved into the agent loop itself, so a structured answer no longer costs a second model call. Middleware landed for human approval, summarization and PII redaction. Python 3.10 is the floor, and the promise is no breaking changes until 2.0.
The part that earns its place is dull and load-bearing: one message object, one tool-schema format, one retry and timeout policy, and a that swaps Anthropic for Bedrock or Vertex by changing a string. Vendors disagree about where the system prompt goes, how a tool call is encoded, what the streaming events are called, and which field holds the token count. That disagreement is about 300 lines of code, and this is where it lives. LiteLLM does this one job and nothing else.
The other half of the layer: loaders, splitters, an and a vector store. 40,000 pages at 650 tokens each, split into 400-token with 80 tokens of overlap, gives about 81,000 chunks. At 1,536 dimensions and 4 bytes a number, the is roughly half a gigabyte. Retrieving the top 8 puts 3,200 tokens in the prompt instead of 26,000,000. How the retrieval itself works is a tutorial of its own; what matters here is that some package owns it, and LlamaIndex owns it in more depth when documents are the whole product.
The bill for layer 1: the claim now costs seven operations instead of one. Extract the PDF, embed the question, search the index, rerank, call the model, parse the typed answer, retry once. Wall clock drops from 8.4 to 5.2 seconds because the prompt got smaller. There are now seven places to be slow and seven places to be wrong, and no way to look at any of them. Layers: 1. per claim: 7. Spans you can read: still 0.
The unbundled version of this layer. LiteLLM for provider routing only. Instructor or BAML for typed output only. LlamaIndex for document ingestion and indexing. Haystack as the older full-pipeline option. A 2026 stack built from four narrow libraries instead of one wide one is a normal choice, and the trade is more imports against fewer surprises.
What the abstraction complaint was actually about. The old high-level chains assembled the prompt for you and did not show it, so debugging a bad answer meant reading library source to find out what was sent. create_agent plus middleware makes both the prompt and the loop inspectable, which is the specific thing that changed.
Which package is which. langchain-core holds the interfaces and depends on almost nothing. langchain holds the agent and middleware. Provider packages such as langchain-anthropic hold one vendor each. langchain-classic holds everything 1.0 stopped shipping.
Where the retrieval detail lives. Chunk sizes, overlap, hybrid search, reranking and citation are the subject of the RAG tutorial. The boring default for the store is pgvector, because the policy index is 0.5 GB and the team already runs Postgres.
Three things do not fit inside a function call. A branch that depends on the model’s answer. A step that waits 99 minutes for a person in a different building. A crash at second 8 of a 12-second run that you would rather not pay for twice. LangGraph answers all three by treating the feature as a state machine with a durable store behind it. The claim goes from seven spans to sixteen, and nine of the new ones are bookkeeping.
A is three declarations: a typed state object, that are plain functions taking the state and returning the part they changed, and edges deciding what runs next. Conditional edges branch on the state, which is how “$12,400 is over $10,000” becomes routing instead of an if-statement buried inside a function. The claim graph has six nodes: ingest, retrieve, decide, threshold, await_approval, investigate. LangGraph appeared in early 2024 and reached 1.0 in October 2025.
Persistence is what changes the range of possible failures. With a configured, the state is written to durable storage after every step, filed under a thread id. Kill the process after node 3 of 6 and the rerun starts at node 4, so the retrieval and the $0.09 decision call are not paid for twice. Three checkpoint writes show up in this claim’s trace. In-memory checkpointers exist for tests, and are exactly as durable as the process.
The approval step calls interrupt(). The state is written and the run ends, so nothing is left waiting. At 11:20:14 the approver’s decision is passed back with the same thread id, the node runs again from the checkpoint, and execution continues. From 09:41:09 that is 99 minutes and 5 seconds using no memory, no connection and no compute. Every product feature labelled “human in the loop” is this plus a queue for the person.
The honest limit of this layer, and the reason the next paragraph names a different tool. A checkpoint records where the run stopped. Nothing in the graph library detects that a run stopped, retries it, or times it out: if the process dies, the state is safe and the run is simply not running until something outside notices. engines invert that. The engine owns the run, retries an activity itself, accepts a signal carrying the human’s answer, and treats a sub-agent as a child workflow.
The alternatives, each with the reason someone picks it. The OpenAI Agents SDK for a model-driven loop with 100 or more models behind LiteLLM. CrewAI for role-based multi-agent prototypes. Pydantic AI when every input and output should be a validated model. Google’s ADK for Gemini-first teams. Temporal when durability matters more than graph ergonomics. Plain Python when the graph has four steps and never branches. LangGraph’s own case is its deployment list and about 39 million PyPI downloads a month. Layers: 2. Spans per claim: 16.
Threads, and rewinding one. Every run is a thread of checkpoints, so a run can be rewound to an earlier step and replayed with a changed input. That is how a disputed decision gets reproduced instead of argued about: load thread claim-8842, rewind to decide, change the clause, run forward.
Reducers, and the bug they prevent. Each state key declares how updates combine: messages appends, amount overwrites. Two nodes writing the same key in one step is a real error, and the reducer is where you say what the framework should do about it.
Serialization is the tax. Everything in state has to be storable, which rules out open file handles, live clients and lambdas. The usual fix is to keep ids in state and look the objects up inside the node, which is more code and one fewer way to lose a run.
Why 1.0 landed on the same day. create_agent in LangChain 1.0 is a prebuilt graph running on this runtime, so the two packages are one release. Reaching for LangChain’s agent and reaching for LangGraph are the same decision at different levels of detail.
Node 6 is called investigate and nobody can draw it. Checking whether a repair shop has a history means reading prior claims, then possibly the adjuster’s note, then possibly nothing else, and the number of steps depends on what the first step finds. That is a loop, not a graph, and it is where the second kind of tool lives. One node adds 22 spans to this claim.
What investigate has to do: look up prior claims on the policy, check the repair shop against past claims, read the adjuster’s note, and stop once there is enough to write one paragraph. On claim 8842 that took 11 tool calls. On the next claim it took 3. Drawing this as a graph means drawing every path in advance, and the number of paths is the number of orderings of five tools, not the number of tools. A sequence whose length is decided while it runs is an .
The Claude Agent SDK gives you the tools, agent loop and context management that run Claude Code, as a library in Python or TypeScript. In a you do not write the loop and you do not draw the edges. You supply tools, a system prompt, and a policy about what is allowed. It also loads skills, slash commands and memory from a project’s .claude/ directory, and reaches external systems over MCP. From another language, run the CLI as a subprocess with -p and JSON output.
In a graph you decide what happens next. In a harness you decide what is allowed to happen at all. A marks each tool as automatic, needing approval, or unavailable. Hooks run your own code at lifecycle points, and PreToolUse is the one that matters, because it is the last place to refuse a call before it happens. In this system, reading prior claims is automatic, emailing the policyholder asks first, and writing to the claims database is not on the agent’s list at all.
Two mechanisms keep a long run inside the window. A is a separate agent instance for one focused subtask, with its own context, so “read 40 prior claims and return one line” does not put 40 claims in the main transcript. summarizes older messages as the window fills, without being asked. The investigation read 61,000 tokens and handed back 240. Without the subagent the main run would carry all 61,000 into every later turn, and pay for them again each time.
The rule, stated once. Known steps, an audit requirement and a typed output mean a graph. Unknown steps, open-ended tool use and an answer in prose mean a harness. This system does both, and node 6 of the LangGraph graph opens an Agent SDK session that returns a typed finding into the graph state. The alternatives at this layer are the OpenAI Agents SDK and Google’s ADK; Anthropic’s Managed Agents runs the same loop as a hosted service instead of a library. Layers: 3. Spans per claim: 38.
Sessions, and forking one. A session can be resumed later or forked in place, which is the cheap way to try two investigation strategies on the same claim and keep whichever one found the prior claims. Forking costs a copy of the transcript and nothing else.
The hooks people actually use. There are 25 lifecycle points and three carry most of the traffic: PreToolUse to refuse a call, PostToolUse to log what a call returned, and PreCompact to keep a copy of the transcript compaction is about to summarize.
Tool definitions cost tokens before anything runs. Every tool the agent may call sits in the system prompt as a schema, on every turn. The measured bill for one popular MCP server is in the MCP tutorial, and it is larger than most people expect.
The adjuster’s note is untrusted text. An agent that reads free-form text written by someone outside your company, and that also holds tools, is the setup described in the prompt-injection tutorial. The permission table above is the mitigation, not the prompt.
At 03:14 last night 41 claims out of 4,000 came out wrong, which is one percent. Four layers ran on every one of them and one of the four is responsible. The runs emitted 38 spans each and you can read none of them, because emitting a span and storing a span are different jobs, and nothing in the stack so far does the second one.
Nothing is added to the claim’s execution in this chapter. LangChain, LangGraph and the Agent SDK have been emitting since chapter 02; with no receiver configured they went nowhere. Turning tracing on is a client and an endpoint, not a rewrite. The waterfall for claim 8842: 7 spans from layer 1, 9 from layer 2, 22 from layer 3, and 12.6 seconds end to end once the 99-minute pause is taken out.
What is recorded: a name, a parent, a start and end time, and attributes. The standardize the AI-specific ones, including gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.response.finish_reasons. Prompt and completion text is opt-in, because a claim PDF in a span attribute is a claim PDF in your logging vendor. As of v1.42.0 on 12 June 2026 the whole gen_ai group moved to its own repository, and it is still pre-stable.
Langfuse was founded in 2023 by Marc Klingen, Maximilian Deichmann and Clemens Rawert, went through Y Combinator’s W23 batch, is MIT licensed, and treats self-hosting as a normal deployment rather than an enterprise favour. Its own SDKs are built on OpenTelemetry and it accepts OTLP at /api/public/otel, so anything that speaks the protocol can send to it, including stacks with no LangChain anywhere in them. It also stores prompts under version control, outside your repository.
A stored turns into per-run cost, p95 latency, token counts per layer, and the ability to filter to the 41 wrong claims and read them. Claim 8842: $0.14, 12.6 seconds of compute, 86,000 tokens, of which 61,000 belong to the investigation. Across 4,000 claims that is $560 a night and $16,800 a month, and layer 3 is 84% of it. Nobody guessed that split before the traces existed.
Langfuse is MIT licensed with self-hosting as an ordinary deployment. LangSmith is a proprietary backend with the deepest zero-setup LangChain and LangGraph integration, and self-hosting only on enterprise plans. Arize Phoenix is open source and runs locally or in a container. Braintrust ties tracing to datasets and CI gates in one product. General tracing vendors now read the same gen_ai attributes, so a backend you have run for years may already qualify. Layers: 4. Spans you can read: 38.
The volume, and what sampling costs. 152,000 spans a night is a small number for a tracing backend and a large one for a Postgres table. Head sampling at 10% throws away nine of every ten rare failures, which are the rows you turned tracing on for. The usual answer is to keep every error and a percentage of successes.
Content capture crosses a boundary. Span attributes holding prompts and completions carry the claimant’s name, the estimate and the adjuster’s note into whatever stores the traces. The opt-in default exists for that reason, and redaction at the SDK is cheaper than a data-processing agreement.
Prompts outside the repository. Storing prompts in the platform lets a prompt change ship without a deploy, which is the upside. The downside is that a checkout of the repository no longer reproduces production, so the version id belongs in the trace.
Why a database company bought a tracing tool. At this volume the hard part is the store rather than the UI: high-cardinality columns, cheap aggregate queries, and retention measured in months. Langfuse was acquired by ClickHouse in January 2026, which is the same observation with money behind it.
You want to change one word in the prompt, from summarize to quote, because 9 of the 41 wrong claims paraphrased a clause instead of citing it. The trace says what happened last night. It cannot say whether tomorrow gets better. Those are two questions answered from the same stored rows, which is why one product usually answers both.
An here is 180 real claims pulled out of stored traces with the correct decision attached: the 41 that came out wrong, plus 139 sampled from the ones that came out right, so a fix cannot break those without anyone noticing. The traces are the raw material, which is the whole reason the two products are one product. Synthetic test cases exist and are worse, because they never contain the adjuster who writes in capitals.
Where there is a right answer, score it in code: the decision matched or it did not, the cited clause number matched or it did not, exactly and for nothing. Where there is no answer key, against a written rubric, which costs a model call per item and has to be checked against human labels before anyone believes it. On these 180 items, two checks are exact and one is a judge.
Run both prompt versions over the same 180 items and compare. Changing summarize to quote fixed 6 of the 9 paraphrase failures and broke 9 claims that had been correct, because the new word made the model cite a clause even when none applied. Net worse by 3, so it does not ship. The gate belongs in CI, failing the branch when the score drops, in the same place a failing test fails.
This is not a separate vendor category, and the reason is structural: an eval needs stored inputs, stored outputs and somewhere to put scores, and a tracing product already has all three. Langfuse, LangSmith and Braintrust ship both halves. DeepEval and Ragas run as test libraries in CI with no platform behind them, and Ragas carries retrieval-specific metrics. Online scoring closes the loop by running the same scorer on a sample of live traffic. Layers: 5.
180 items is a small number. A two-item difference between versions is noise, and a fifteen-item difference is not. Running the same fixed set every time beats sampling fresh claims, because a moving set turns every comparison into an argument about which claims were harder.
The judge drifts. The grader is a model, the model gets upgraded, and last month’s scores stop being comparable. Pin the judge’s model version and record it beside the score, or accept that the eval history is a chart of two things changing at once.
The set grows from incidents. Every wrong claim someone complains about is one more row with a known answer. This is the only artifact in the tutorial that gets more valuable with age, and the only one worth backing up separately from the database it lives in.
Everything about rubrics and shipping decisions is in the evals tutorial, which follows one prompt change from complaint to deploy. The only question here is which layer owns the scoring.
Nobody asked for anything at 02:00. The policy team changed 312 pages yesterday, the index still holds the old text, and the 180-item eval set has not run since Friday. Every layer so far starts when a claim arrives. This one starts when the clock says so, and that difference is the whole reason a 2014 data-engineering tool is sitting in an AI stack.
Maxime Beauchemin started Airflow in October 2014 at Airbnb, open source from the first commit, announced in June 2015, an Apache incubator project in March 2016 and a top-level Apache project in January 2019. A is Python that declares tasks and the dependencies between them; a scheduler decides when they run and a worker runs them. None of it was designed for models, and the transformer paper came out two years and eight months after the first commit.
LangGraph and Airflow are both directed graphs of steps, which is why people confuse them, and they answer different questions. A graph run starts because a claim arrived, carries typed state between nodes, and can pause for a person. A DAG run starts because a said 02:00, passes data through storage rather than state, and is expected to be repeatable for any past date. The scheduler fires whether or not anyone made a request, which is the position a clock occupies and not the position a phone does.
Airflow 3.0 became generally available on 22 April 2025, the largest release since 2.0 in 2020. It brought DAG versioning, a task execution API that lets a task run outside the scheduler’s environment and in another language, event-driven scheduling, and an asset-centric interface. The change that matters here is the removal of the rule that every run needs a unique execution date, which is what let inference and training jobs fit without workarounds. 3.3.0 shipped on 6 July 2026.
The nightly run, five tasks. Pull yesterday’s 4,000 traces. Re-embed the 312 policy pages that changed and upsert them into the index. Rebuild what depends on them, run the 180-item eval set against the current prompt, and post a scorecard. 41 minutes end to end, of which re-embedding is 31. Two tasks retry on failure, one carries an SLA, and the whole run is repeatable for any past date, which is how a bad index gets repaired. Re-embedding all 40,000 pages instead of the 312 that changed would take 66 hours, so the is the feature, not the schedule.
Prefect and Dagster answer the same scheduled-batch question, Dagster by making the produced asset the unit rather than the task. Temporal answers a different question: a that owns one long-running execution, retries its activities, and takes a signal when a human replies, which is chapter 03’s problem and not this one. Plenty of platforms run a scheduler and a durable engine side by side, because the two jobs really are different. Layers: 6. Spans a night: 152,000.
Every task here runs twice sooner or later. A retried embedding upsert has to overwrite by chunk id, not append, or the index grows duplicate chunks that outrank the original and change what the model reads. This is the same idempotent requirement as chapter 01’s retry, one layer up.
Backfilling 90 days is 90 runs. Repairing an index after a bad embedding model means re-running the DAG once per past date, not once with a date range, because each run’s output is scoped to its own day. That property is the reason the schedule and the run are separate concepts.
The failure that actually happens. The eval task passes while the embedding task produced an empty index, so the scorecard says everything is fine and the claims get worse in the morning. Freshness checks on the asset catch this; retries and alerts on the crash do not, because there was no crash.
The option nobody writes down. A cron entry and a shell script. It is the right answer until the first backfill, the first task that has to run somewhere else, or the first time someone asks which version of the index produced Tuesday’s decisions.
Claim 8842 arrived at 09:41:02 and was written back at 11:20:41, having crossed six layers, emitted 38 spans, spent 99 of those minutes as a row in a database, and cost 14 cents. The board below is the part worth keeping: the left column is a question, and every product name beside it is somebody’s answer to that one question.