SPANS · OTLP · STORAGE · EVALS · PROMPTS · EXPERIMENTS · PERSONAL DATA

AI agent tracing:
how a Phoenix setup is put together

Tracing records every step an AI agent takes, each model call, tool call and search, as a timed record you can open later. Phoenix is an open-source server that stores those records, lets you score and test them, and keeps the prompts they were made with.

The system on this page is the returns chat of a bike-parts shop in Berlin. A customer writes in, and an AI agent looks up the order in a second service, searches the returns policy and replies. Every step lands in a Phoenix server the shop runs itself, so each chapter can open the same conversation, and the last two can ask what the shop owes the customer whose words are in it.

Here is the whole setup. Most boxes won’t mean much yet; each chapter explains one of them.

MapSolid outlines are machines and processes the shop runs; the dashed region is the model API outside it. Spans flow left to right, from the process that made them to Phoenix.

You need to know what an HTTP request, a process and a database table are; no tracing or legal background is assumed. The versions are Phoenix 22.0.0 and OpenTelemetry Python 1.45.2, checked on 20 November 2026. The conversation and every duration, size, count and score on this page are a made-up example, chosen to be realistic; figures that show them carry an illustrative tag. Chapters 09 and 10 explain what the GDPR asks of a trace store; they are not legal advice. Tap any figure to open it full screen and zoom. are tappable.

Spans and traces: one agent run, one tree

The returns agent answers a customer in a few seconds: two model calls, an order lookup in another service and a policy search. When a reply is wrong, the reply is the only evidence. Nothing records which step produced which input, how long each step took, or which one failed. OpenTelemetry is the open standard for recording that, as timed records nested inside each other. What does a record of one agent run contain, and what produces it?

STEP 01 / 04

THE MAP SO FAR

Fig 1.1The accent box is new: the tracer provider inside the agent’s process. Faint parts arrive in later chapters.

A is the OpenTelemetry SDK object inside a process that creates spans and hands each finished one on. Here it lives in returns-agent, the shop’s Python process, next to the agent loop that calls the model and runs the tools.

This chapter adds only that. Nothing leaves the process yet: sending spans to Phoenix is chapter 04, and storing them is chapter 05.

STEP 02 / 04

A TRACE IS A TREE OF TIMED SPANS

Fig 1.2Bars share one time scale; the accent row is the tool call, and the rows under it ran inside it.

A is one timed operation: a name, a start and an end time, and the id of its parent span. A is every span of one run, all sharing one trace id. The root span, the one with no parent, covers the whole run.

Here is turn 1 of the hero conversation, with illustrative timings. The root span, returns-agent, lasts 3.84 s. Under it come a model call (1.41 s), the lookup_order tool (312 ms, with the HTTP call to orders-api inside it), the search_policy search (86 ms) and a second model call (1.96 s).

Indentation is parentage and bar length is time. The two model calls take 3.37 s, 88% of the run, so a slow turn is usually a slow model call.

STEP 03 / 04

EVERY SPAN CARRIES THE SAME FEW FIELDS

Fig 1.3Each row is one field of the tool span; the accent row is the link that puts it in the tree.

Every span has the same fields, whatever library made it. Three are ids: a trace id of 16 bytes (32 hex digits), a span id of 8 bytes (16 hex digits) and the parent’s span id, which places the span in the tree. Then come a name, a kind, a start and an end time in nanoseconds since 1970, and a status.

are key–value pairs, such as tool.name = lookup_order. Events are timed notes inside the span, such as an exception. Links point at spans of other traces. By default a span keeps up to 128 attributes and 128 events, and a value’s length has no limit.

The kind field is OpenTelemetry’s own: INTERNAL for work inside one process, CLIENT and SERVER for the two ends of a request. The OpenInference kinds of chapter 02 are an attribute, a separate thing.

A SPAN OPENED BY HAND
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("returns-agent")

def search_policy(query: str) -> list[str]:
    with tracer.start_as_current_span("search_policy") as span:
        span.set_attribute("policy.top_k", 3)
        try:
            return index.search(query, k=3)   # success: the status stays UNSET
        except TimeoutError as exc:
            span.record_exception(exc)        # an "exception" event in the span
            span.set_status(Status(StatusCode.ERROR))
            return []                         # handled: the reply still goes out
        # an exception that escapes the with block is recorded the same way
STEP 04 / 04

A FAILED STEP IS A SPAN WITH STATUS ERROR

Fig 1.4The accent rows are the failure: status ERROR on the tool span and the exception event inside it.

A later turn of the conversation failed: orders-api did not answer, and the lookup_order span ended after 5 s with ERROR. Inside it is one event named exception, holding the exception’s type (httpx.ReadTimeout), its message and the stack trace.

OpenTelemetry’s with block wrote both: an exception that escapes it becomes an exception event, and the span’s status becomes ERROR with the class name and message.

The root span’s status stays UNSET, which counts as success, because the agent caught the error and replied “I can’t see your order right now. Could you try again in a few minutes?” A parent never inherits a child’s error. Only the trace shows that a tool failed and how long the customer waited for it.

COMMON QUESTIONS

Is a trace the same as a log line?
No. A log line is one event with no duration and no parent. A span has both, so a trace shows what ran inside what, and for how long.
Does tracing slow the agent down?
Creating a span costs microseconds, and the model calls take seconds. The costs that matter are export and storage, chapters 04 and 05.
Do I need OpenTelemetry to use Phoenix?
Phoenix receives OpenTelemetry spans. Its helper package, phoenix.otel, sets up an OpenTelemetry tracer provider for you, so you rarely touch the SDK directly.

WHAT YOU NOW KNOW

  • A trace is every span of one agent run, linked by parent ids into a tree under one trace id of 32 hex digits.
  • A span has a name, a kind, start and end times in nanoseconds, attributes, events and a status; hero turn 1 has 7.
  • A failed tool call is a span with status ERROR and an exception event, even when the reply looks normal.
DEEP DIVE: OpenTelemetry’s span kinds, an event or a child span, links, clocks and the current span, span limits

OpenTelemetry’s span kinds

The span’s own kind field has five values. INTERNAL is work inside one process and the default. CLIENT and SERVER are the two ends of a request, and PRODUCER and CONSUMER the two ends of a message on a queue. Backends pair a CLIENT span with the SERVER span under it to draw which service calls which.

This kind says how a span relates to other processes. openinference.span.kind (chapter 02) says what an AI step was. They are independent: the hero trace’s tool span is INTERNAL and TOOL at once.

An event or a child span

An event is a point in time inside a span, with a name and attributes but no duration: a retry, a cache miss, an exception. A child span has a duration and its own status, and Phoenix can filter and sort on it.

The rule of thumb: if you would ever ask “how long did this take?” or “which of these failed?”, make it a span. If it only marks that something happened, make it an event.

Links

A link points from a span to a span in another trace, by trace id and span id, without making it a parent. It fits fan-in: an evaluator job that reads many traces can open one span of its own and link it to each span it scored, instead of joining every one of their trees.

Clocks and the current span

Each process stamps its spans with its own clock. If the machine running orders-api is a few milliseconds behind, its server span can appear to start before the client span that caused it. Keep every machine on NTP, and read cross-machine gaps of a few milliseconds as noise.

Inside one process, the SDK keeps the current span in a Python contextvar. A span opened while another is current becomes its child, which is how the tree builds itself without passing ids around. Chapter 03 follows that context across threads and processes.

Span limits

By default a span keeps 128 attributes and 128 events; the SDK drops any beyond that and records how many it dropped. OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT and OTEL_SPAN_EVENT_COUNT_LIMIT change them. OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT has no default, so a long prompt is stored as one long value. Chapter 05 shows what that costs on disk.

Instrumentors and OpenInference: what a span says

The tracer provider can make spans, but only for code that asks it to. The agent’s model calls happen inside the Anthropic SDK, which knows nothing about tracing, and a span named POST /v1/messages says nothing about tokens or messages. Phoenix needs to know which span was a model call, which was a tool, and what went in and out. How does a span say it was a model call, and who writes that without changing your code?

STEP 01 / 04

THE MAP SO FAR

Fig 2.1The accent box is new: instrumentors wrap the SDK calls inside the agent and hand their spans to the tracer provider.

An is a library that wraps another library’s functions, so each call opens and closes a span. is Arize’s set of instrumentors and attribute names for LLM applications, built on OpenTelemetry. Its package openinference-instrumentation-anthropic wraps the Anthropic SDK.

Two calls at start-up do it: register() from phoenix.otel builds the tracer provider, and AnthropicInstrumentor().instrument() patches the SDK. After that, every client.messages.create call in the agent is a span, and the agent’s own code is unchanged.

TWO CALLS AT START-UP
from anthropic import Anthropic
from openinference.instrumentation.anthropic import AnthropicInstrumentor
from phoenix.otel import register

tracer_provider = register(
    project_name="returns-agent",
    endpoint="http://phoenix.shop.internal:6006/v1/traces",
    batch=True,
)
AnthropicInstrumentor().instrument(tracer_provider=tracer_provider)

client = Anthropic()   # every client.messages.create(...) is now an LLM span
STEP 02 / 04

A SPAN KIND SAYS WHAT THE STEP WAS

Fig 2.2Each row is one OpenInference kind; the accent rows are the kinds in the hero trace.

The attribute openinference.span.kind holds one of 11 values: LLM, EMBEDDING, CHAIN, RETRIEVER, RERANKER, TOOL, AGENT, GUARDRAIL, EVALUATOR, PROMPT, DECISION. Phoenix draws each with its own icon and filters on it, so span_kind == 'LLM' finds every model call.

Hero turn 1 uses AGENT for the root, LLM for both model calls, TOOL for lookup_order and RETRIEVER for search_policy. The two HTTP spans have no kind, because plain OpenTelemetry instrumentation wrote them.

STEP 03 / 04

AN LLM SPAN RECORDS THE MESSAGES BY DEFAULT

Fig 2.3Each row is one attribute of the first model call’s span; the accent rows hold the conversation text itself.

The Anthropic instrumentor fills the first model call’s span with OpenInference attributes: the provider and model, every input message, the tool call the model asked for, and the token counts, 1,840 in and 96 out (illustrative). input.value and output.value hold the whole request and response as JSON.

Lists are flattened into numbered keys counting from zero, so the customer’s message is llm.input_messages.1.message.content, word for word, “cut my hand” included. The second model call also carries orders-api’s answer as a tool result: her name, email and address.

This is where OpenInference and OpenTelemetry’s own gen_ai differ. gen_ai leaves message content out unless you turn it on; OpenInference records it unless an OPENINFERENCE_HIDE variable turns it off (chapter 09). the AI stack tutorial shows the gen_ai names.

THE FIRST MODEL CALL’S SPAN, TRIMMED
{
  "openinference.span.kind": "LLM",
  "llm.provider": "anthropic",
  "llm.model_name": "claude-sonnet-5-5",
  "llm.invocation_parameters": "{\"max_tokens\": 1024}",
  "llm.input_messages.0.message.role": "system",
  "llm.input_messages.0.message.content": "You answer returns questions for the shop...",
  "llm.input_messages.1.message.role": "user",
  "llm.input_messages.1.message.content": "My order R-10482 came with a bent derailleur hanger, and I cut my hand fitting it. Can I send it back?",
  "llm.output_messages.0.message.role": "assistant",
  "llm.output_messages.0.message.tool_calls.0.tool_call.function.name": "lookup_order",
  "llm.output_messages.0.message.tool_calls.0.tool_call.function.arguments": "{\"order_id\": \"R-10482\"}",
  "llm.token_count.prompt": 1840,
  "llm.token_count.completion": 96,
  "llm.token_count.total": 1936
}
STEP 04 / 04

YOUR OWN STEPS NEED A SPAN YOU OPEN

Fig 2.4Compare the columns: the instrumentor covers SDK calls, and the accent column is the code only you can mark.

search_policy and lookup_order are the shop’s own functions, and no instrumentor knows them. The tracer you get from phoenix.otel’s tracer provider is OpenInference’s, and its decorators open a span of a given kind: @tracer.tool, @tracer.chain, @tracer.agent and others record the arguments and the return value.

For a block rather than a function, start_as_current_span takes an openinference_span_kind argument. The root AGENT span is one such block around the whole turn, so every span the turn opens is its child.

Spans written by other instrumentation work too. Since version 15.10, Phoenix converts gen_ai.* attributes to OpenInference ones on arrival, and an OpenInference value already on the span wins.

SPANS FOR THE SHOP’S OWN CODE
tracer = tracer_provider.get_tracer(__name__)   # an OpenInference tracer

@tracer.tool
def lookup_order(order_id: str) -> dict:
    """Fetch one order from orders-api."""
    return httpx.get("http://orders-api/orders/" + order_id, timeout=5).json()

def search_policy(query: str) -> list[str]:
    with tracer.start_as_current_span(
        "search_policy", openinference_span_kind="retriever"
    ) as span:
        span.set_input(query)
        paragraphs = index.search(query, k=3)
        span.set_output(paragraphs)
        return paragraphs

with tracer.start_as_current_span("returns-agent", openinference_span_kind="agent"):
    reply = run_agent(message)

COMMON QUESTIONS

Which libraries have instrumentors?
OpenInference ships 38 Python packages named openinference-instrumentation-<library>: OpenAI, Anthropic, LangChain, LlamaIndex, the OpenAI Agents SDK, the Claude Agent SDK, Bedrock, MCP, DSPy and more.
Can Phoenix read spans written with the gen_ai names?
Yes. Since 15.10 it converts gen_ai attributes to OpenInference ones on arrival, and values already in OpenInference form win.
Is auto_instrument=True enough?
It turns on every OpenInference instrumentor installed in the environment. It can’t mark your own functions; those still need a decorator or a with block.

WHAT YOU NOW KNOW

  • An instrumentor wraps a library so each call becomes a span; the Anthropic one turns every messages.create into an LLM span.
  • openinference.span.kind says what a step was; the hero trace uses AGENT, LLM, TOOL and RETRIEVER.
  • LLM spans record messages, tool calls and token counts by default, so the customer’s words are in the trace unless hidden.
DEEP DIVE: the attribute groups, gen_ai and OpenInference side by side, cost attributes, images, framework instrumentors

The attribute groups

input.value and output.value (with input.mime_type) are on every kind. llm.* holds the model, messages, tools, invocation parameters and token counts; tool.* the name, description and parameters; retrieval.documents.N.document.* each retrieved document’s id, content, score and metadata.

Four more cross every kind: session.id and user.id (chapter 03), metadata, a JSON string of your own fields, and tag.tags, a list of labels to filter on.

gen_ai and OpenInference side by side

The two name the same facts differently: gen_ai.usage.input_tokens is llm.token_count.prompt, and gen_ai.request.model is llm.model_name. The bigger difference is content. gen_ai records messages only when asked: the spec’s variable is OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT, while the Python packages read OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT. OpenInference records them unless hidden.

OPENINFERENCE_ENABLE_GENAI_SEMCONV=true makes OpenInference instrumentors write gen_ai attributes alongside their own, for a second backend that reads only those.

Cost attributes

OpenInference defines llm.cost.prompt, llm.cost.completion and llm.cost.total, in US dollars, for code that knows its own prices. Phoenix also prices calls itself from the token counts when the model is in its price table, and keeps the result in a table of its own, span_costs (chapter 05).

Images

An image sent to the model arrives as a base64 data URI, which can be megabytes. OPENINFERENCE_BASE64_IMAGE_MAX_LENGTH (default 32,000 characters) caps it: a longer image is replaced by __REDACTED__, unless a blob uploader is configured to store it elsewhere and record its address.

Framework instrumentors

An agent built on a framework, such as LangChain or the OpenAI Agents SDK, gets an instrumentor for the framework too. It adds CHAIN, AGENT and TOOL spans for the framework’s own steps around the SDK’s LLM spans. Install both and check one trace in Phoenix before trusting token totals, since one model call can then appear at two levels.

Trace context: one trace across two services

The lookup_order tool calls orders-api over HTTP, and orders-api is a separate process with its own tracer provider. Without help, its spans start a new trace, and what happened inside the tool call is lost from the agent’s trace. A conversation also has 3 turns, so 3 traces, and nothing yet says they belong together. How does a second service join the same trace, and how do separate traces become one conversation?

STEP 01 / 03

THE MAP SO FAR

Fig 3.1The accent parts are new: orders-api with its own tracer, and the request that carries the trace into it.

is the trace id and the current span id, carried with a request so the next process can continue the same trace. orders-api is the shop’s second service: GET /orders/{id} returns an order’s lines with the customer’s name, email and address.

orders-api gets its own tracer provider and instrumentation, which read the context from every incoming request. The accent arrow from the agent loop is the order lookup, now carrying the trace with it.

STEP 02 / 03

traceparent CARRIES THE TRACE INTO THE NEXT PROCESS

Fig 3.2Follow 1 to 3: the client span’s id leaves in the header and becomes the parent of orders-api’s span.

When lookup_order calls orders-api, the httpx instrumentor opens a CLIENT span and writes the header onto the request: version 00, the 32-hex-digit trace id, the client span’s 16-hex-digit id, and flags 01, meaning sampled. This is the W3C Trace Context format, which every OpenTelemetry SDK reads and writes.

orders-api’s instrumentation reads the header and opens its SERVER span in that trace, as a child of the client span. Writing and reading headers is a propagator’s job; the default set is tracecontext,baggage. The code block shows the header with this trace’s illustrative ids.

Inside one process, the current span lives in a Python contextvar, and the next span opened finds its parent there. asyncio tasks inherit it; a plain thread starts with none, so its spans begin a new trace unless the context is passed along.

WHAT ORDERS-API RECEIVES
GET /orders/R-10482 HTTP/1.1
host: orders-api
traceparent: 00-7d3e1a9c52f04b8e9a61c0d2f4b85e17-5f2a8d6c0e1b9437-01
#            version  trace id (32 hex)          parent span id   sampled
STEP 03 / 03

A SESSION GROUPS THE TRACES OF ONE CONVERSATION

Fig 3.3The turns’ traces sit on one time line; the accent bar is the session that groups them.

Each turn of the chat is its own trace. The attribute session.id groups them: Phoenix keeps one row per in its project_sessions table and shows the turns in order. A second attribute, user.id, names the person the conversation was with.

using_session and using_user from openinference.instrumentation wrap one turn, and every OpenInference span opened inside the block gets both attributes. The hero conversation, chat_7f3a, is 3 traces over 144.1 s, all with user.id cus_48213.

That id is the first the shop sends to Phoenix on purpose. Chapter 09 replaces it with a keyed hash.

ONE TURN, INSIDE A SESSION
from openinference.instrumentation import using_session, using_user

def handle_turn(chat_id: str, customer_id: str, message: str) -> str:
    with using_session(chat_id), using_user(customer_id):
        # chat_id "chat_7f3a", customer_id "cus_48213" (chapter 09 hashes it)
        with tracer.start_as_current_span(
            "returns-agent", openinference_span_kind="agent"
        ):
            return run_agent(message)

COMMON QUESTIONS

Does orders-api need Phoenix installed?
No. It needs the OpenTelemetry SDK and an exporter pointed at Phoenix. Phoenix only sees the spans each process sends.
What if orders-api is written in Go?
That works. traceparent is a W3C standard, and every OpenTelemetry SDK reads and writes it.
Can I put the customer id in baggage instead?
Baggage travels in a header to every downstream service, including ones you don’t run, so keep personal data out of it.

WHAT YOU NOW KNOW

  • The traceparent header carries the trace id and the caller’s span id, so orders-api’s span joins the tool span’s trace.
  • Context lives in contextvars inside a process; asyncio keeps it, plain threads need it passed.
  • session.id groups a conversation’s 3 traces; user.id says whose it was.
DEEP DIVE: tracestate and baggage, sampling at the root, async code and threads, which project a trace lands in, spans from the browser

tracestate and baggage

Two more headers can travel beside traceparent. tracestate holds vendor-specific key–value pairs, at most 32 of them, for tracing systems that need more than the two ids. baggage holds application key–value pairs that every downstream service receives and can read; W3C asks propagators to carry at least 64 entries and 8,192 bytes.

Baggage is not copied onto spans unless a processor does it, and it reaches services and vendors you don’t control. Customer ids, emails and session ids stay out of it.

Sampling at the root

The default sampler is ParentBased(AlwaysOn): the root span keeps every trace, and each child follows its parent’s sampled flag, the 01 at the end of traceparent. To keep a share of traces, the root samples by trace id; the spec’s newer ProbabilitySampler replaces TraceIdRatioBased.

A root decides before it knows whether the turn fails. To keep only failing traces, the Collector’s tail_sampling processor (beta) waits decision_wait (30 s by default) after a trace’s first span, then keeps or drops the whole trace by rules such as “any span has status ERROR”.

Async code and threads

An asyncio task copies the context when it is created, and so does asyncio.to_thread. A threading.Thread or a ThreadPoolExecutor job starts with an empty context, so a span opened there has no parent and starts a new trace. The tool call then shows up in Phoenix as a separate trace of one span.

Pass the context in by hand with contextvars.copy_context().run(fn), or install opentelemetry-instrumentation-threading, which does it for every thread and executor job.

Which project a trace lands in

Each process names its Phoenix project in the resource attribute openinference.project.name; phoenix.otel sets it from project_name or PHOENIX_PROJECT, and a process without it lands in default. Phoenix files a whole trace under the project of the first span it stores for that trace id, and later spans join that trace whatever their own project says.

Child spans end, and are exported, before their parents, so orders-api’s spans often arrive first. A plain OpenTelemetry service sets the name with OTEL_RESOURCE_ATTRIBUTES=openinference.project.name=returns-agent.

Spans from the browser

The chat page could start the trace itself: OpenTelemetry’s JavaScript SDK opens a span for the customer’s request and sends traceparent with it. The browser then exports spans to a public endpoint, usually a Collector that must allow cross-origin requests, and anyone can send spans to it. Most teams start the trace at their first server instead.

The span processor and OTLP: getting spans out

Finished spans now pile up inside two processes, returns-agent and orders-api, and nothing outside them can see one. They have to cross the network to Phoenix without slowing the chat, and without vanishing when a process restarts. , the OpenTelemetry Protocol, is the wire format every OpenTelemetry SDK and backend speaks: protobuf messages over HTTP or gRPC. When does a span leave the process, in what format, and what can lose it on the way?

STEP 01 / 04

THE MAP SO FAR

Fig 4.1The accent parts are new: the processor and exporter in returns-agent, both OTLP lanes, and Phoenix’s receiver.

Two SDK parts take a finished span out of the process. A receives each span the moment it ends and decides when to pass it on; an exporter turns what it is passed into network requests. The OTLP exporter speaks , and Phoenix’s OTLP receiver accepts it on port 6006 at /v1/traces over HTTP, or on port 4317 over gRPC.

orders-api has the same two parts behind its handler, too small to draw. Both lanes run through two faint boxes, a redactor and a Collector. Until chapter 09 adds them, spans pass straight through.

STEP 02 / 04

A BATCH PROCESSOR SENDS EVERY FIVE SECONDS

Fig 4.2The accent row is the difference: batch sends from a background thread, so a slow Phoenix never delays a reply.

SimpleSpanProcessor exports each span the moment it ends, on the thread that ended it, so the agent waits for every request. BatchSpanProcessor puts finished spans on a queue, and a background thread exports them every 5 s or as soon as 512 are waiting. The queue holds up to 2,048 spans; when it is full, Python drops the oldest and logs a “Queue full” warning.

phoenix.otel’s register() installs the simple processor unless you pass batch=True, and prints a warning when it does. In the example (its counts and sizes are illustrative), the 6 spans returns-agent records for turn 1 leave in one request, not 6. At a normal exit, the tracer provider flushes what is still queued.

REGISTER() WITH THE BATCH PROCESSOR
from phoenix.otel import register

tracer_provider = register(
    project_name="returns-agent",     # resource attribute openinference.project.name
    endpoint="http://phoenix.shop.internal:6006/v1/traces",   # a path: OTLP over HTTP
    batch=True,                       # BatchSpanProcessor; the default is False
)
# with no endpoint at all, register() sends gRPC to localhost:4317
STEP 03 / 04

ONE OTLP REQUEST IS SPANS GROUPED BY RESOURCE

Fig 4.3Read top to bottom: the accent row is the resource attribute that files every span in the request under one project.

An OTLP request is four levels of nested messages. One ExportTraceServiceRequest holds one ResourceSpans per process, and its lists attributes of that process, such as service.name. Inside it, one ScopeSpans per instrumentation scope (the tracer that made the spans: the Anthropic instrumentor, the httpx one, your own code) holds the spans themselves.

register(project_name=…) puts openinference.project.name on the resource, so Phoenix files every span in the request under that project. Phoenix’s HTTP receiver accepts protobuf only, and JSON gets HTTP 415; gzip and deflate are accepted. Turn 1’s request from returns-agent is 19.6 KB, or 5.9 KB gzipped, and the LLM spans are about 69% of it.

STEP 04 / 04

THE ENDPOINT DECIDES THE PROTOCOL

Fig 4.4Each row is one setting; the accent row is the default, which sends gRPC to port 4317.

register() picks the protocol from the endpoint’s shape. An endpoint with a path, such as http://phoenix.shop.internal:6006/v1/traces, means OTLP over HTTP with protobuf. A bare origin means gRPC, and register() swaps in the gRPC port, 4317, whatever port you wrote. With nothing set at all, it sends gRPC to localhost:4317, which works on a laptop and nowhere else.

Each argument has an environment variable: PHOENIX_COLLECTOR_ENDPOINT, PHOENIX_PROJECT and PHOENIX_API_KEY, sent as a Bearer header. An API key is a secret Phoenix issues to identify a sender, and since version 21 the Docker image of Phoenix accepts spans only with one. A plain OpenTelemetry exporter instead treats OTEL_EXPORTER_OTLP_ENDPOINT as a base URL and appends v1/traces.

THE SAME SETTINGS AS ENVIRONMENT VARIABLES
# read by phoenix.otel.register() when its arguments are left out
export PHOENIX_COLLECTOR_ENDPOINT="http://phoenix.shop.internal:6006/v1/traces"
export PHOENIX_PROJECT="returns-agent"
export PHOENIX_API_KEY="..."   # sent as a Bearer header; Docker Phoenix needs one

# a plain OpenTelemetry exporter: the base URL gets v1/traces appended
export OTEL_EXPORTER_OTLP_ENDPOINT="http://phoenix.shop.internal:6006"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"

COMMON QUESTIONS

Does the chat wait for Phoenix?
Not with the batch processor: export runs on a background thread. With the simple processor, every span ends with a network call on the agent’s thread.
What happens when Phoenix is down?
The exporter retries temporary failures, such as 503, with growing waits until its timeout (10 seconds by default), then drops that batch. Meanwhile the queue fills, and past 2,048 spans the oldest are dropped.
HTTP or gRPC?
Either. HTTP on 6006 shares the UI’s port; gRPC on 4317 needs a second open port. Both carry the same protobuf messages.

WHAT YOU NOW KNOW

  • A batch processor queues finished spans and exports every 5 s or 512 spans; register() needs batch=True to use it.
  • One OTLP request groups spans under their resource, and the resource’s openinference.project.name picks the Phoenix project.
  • An endpoint ending in /v1/traces means HTTP protobuf; with nothing set, register() sends gRPC to port 4317.
DEEP DIVE: retries and partial success, queue settings, compression, two backends, config discovery

Retries and partial success

The OTLP spec lists 429, 502, 503 and 504 as retryable and asks a client to wait as long as a Retry-After header says. Python’s exporters retry temporary failures with exponential backoff until their timeout runs out, 10 seconds by default (OTEL_EXPORTER_OTLP_TIMEOUT), then drop the batch and log an error. A 401 or a 415 is not retried.

A server can also accept part of a request. It answers HTTP 200 with partial_success.rejected_spans and a message, and the client must not retry, since resending would only repeat the rejection. The rejected spans are not stored anywhere.

Queue settings

BatchSpanProcessor reads four environment variables: OTEL_BSP_SCHEDULE_DELAY (5 s), OTEL_BSP_MAX_EXPORT_BATCH_SIZE (512 spans), OTEL_BSP_MAX_QUEUE_SIZE (2,048 spans) and OTEL_BSP_EXPORT_TIMEOUT. Python accepts the last one and does not apply it to exports; its source notes there is no way to pass a timeout to export. The exporter’s own timeout bounds a slow send.

A larger queue survives a longer outage at the cost of memory, since every queued span holds all its attributes, message lists included. Size it from spans per second times the longest outage you want to ride out.

Compression

Python’s OTLP exporters send uncompressed unless OTEL_EXPORTER_OTLP_COMPRESSION=gzip is set. Phoenix accepts gzip and deflate. In the example, gzip takes turn 1’s request from 19.6 KB to 5.9 KB, because LLM spans repeat the same message text from call to call.

Two backends

A tracer provider can hold several span processors, each with its own exporter, so the same spans can go to Phoenix and to a second backend. Every process then carries both configurations and needs both network paths.

A Collector, chapter 09, receives spans once and fans them out to several exporters, so each service keeps a single endpoint and the routing lives in one place.

Config discovery

phoenix.otel can also pick up settings from a .env.phoenix file it discovers. In production, set PHOENIX_DISCOVER_CONFIG=false, so only the arguments and environment variables you can see decide where spans go and which key they carry.

ERRORS THIS CHAPTER EXPLAINS

HTTP 401 Unauthorized
Phoenix has auth on (the Docker image does since v21) and the request carried no valid API key
set PHOENIX_API_KEY where the exporter runs
HTTP 415 Unsupported Media Type
the exporter sent OTLP as JSON; Phoenix’s receiver takes protobuf only
use the http/protobuf protocol
HTTP 503 Service Unavailable
Phoenix already holds 20,000 spans waiting to be written
the exporter retries; if it persists, look at the database’s write speed (chapter 05)
Queue full, dropping …
the batch processor’s queue is full because exports are slower than new spans
find why exports are slow, or raise OTEL_BSP_MAX_QUEUE_SIZE
StatusCode.UNAVAILABLE
the gRPC exporter found nothing listening, often register() defaulting to localhost:4317
set the endpoint, with /v1/traces for HTTP

The Phoenix server: receiver, database and API

Spans now reach Phoenix in batches of protobuf. To be useful they must be stored, so a person can open one conversation and a script can ask for every failing tool call this week. Phoenix keeps them in an ordinary SQL database: SQLite in one file by default, or PostgreSQL. What does Phoenix do with a span when it arrives, and what does one look like in the database?

STEP 01 / 04

THE MAP SO FAR

Fig 5.1The accent parts are new: the database inside Phoenix, and the API in front of it that people and scripts use.

Phoenix’s OTLP receiver decodes each request and puts its spans on an in-memory queue. A bulk inserter, a loop inside the server, takes spans off that queue and writes them to the database in transactions of up to 100 operations, and checks the queue again every 0.1 s when it is empty. When more than 20,000 spans are waiting, the receiver answers 503 and the exporter retries later.

The UI, the REST API and the GraphQL API sit in front of that database. The UI is how a person opens one conversation; REST and GraphQL are how scripts, such as the evaluator job of chapter 06, read and write.

STEP 02 / 04

ONE SPAN IS ONE ROW, ITS ATTRIBUTES ONE JSON VALUE

Fig 5.2Each row is one span of hero turn 1; the accent root row’s cumulative cells add up every span under it.

Phoenix writes each span as one row of the spans table. Fixed fields get their own columns: span_id, parent_id, name, span_kind, start_time, end_time, status_code. Every attribute goes into one JSON column, attributes, and the events into another. Each trace is a row in the traces table, and every span row points to it through trace_rowid.

Token counts get columns too, shown here with the example’s illustrative values. llm_token_count_prompt is filled only on model-call spans; cumulative_llm_token_count_prompt adds up a span and every span under it. So turn 1’s root row holds 4,150 prompt tokens, both model calls together. When the model is in Phoenix’s price table, each call is also priced into span_costs.

THE HERO TRACE, STRAIGHT FROM THE SQLITE FILE
sqlite3 ~/.phoenix/phoenix.db "
  SELECT name, span_kind, status_code,
         llm_token_count_prompt, cumulative_llm_token_count_prompt
  FROM spans
  WHERE trace_rowid = (SELECT id FROM traces
                       WHERE trace_id = '7d3e1a9c52f04b8e9a61c0d2f4b85e17')
  ORDER BY start_time;"
# the root row: its cumulative count (4150) adds up both model calls
STEP 03 / 04

EVERYTHING ELSE IS TABLES IN THE SAME DATABASE

Fig 5.3Each panel is one group of tables; the accent panel holds what the receiver writes, and the rest arrive in later chapters.

The holds far more than spans. The same schema keeps annotations (chapter 06), prompts and their versions (07), datasets and experiment results (08), and users, API keys and retention policies (10): 66 tables in Phoenix 22.0.0. Alembic migrations, scripts that update the schema, run every time the server starts.

By default the database is a SQLite file, ~/.phoenix/phoenix.db, under PHOENIX_WORKING_DIR. Set PHOENIX_SQL_DATABASE_URL to use PostgreSQL 14 or later instead. The shop runs PostgreSQL in production; the SQL in this chapter uses SQLite, which is what you get on a laptop.

STEP 04 / 04

AN LLM SPAN IS THE LARGEST ROW

Fig 5.4Each bar is the average stored size of one span kind; the accent bar is the model call, which carries the conversation.

An LLM span is the largest row because each model call stores the whole message list sent so far: system prompt, the customer’s messages, earlier replies and tool results. In the example an LLM span’s attributes average 6.8 KB, against 1.1 KB for a TOOL span. Turn 1 comes to 19.4 KB, and a 3-turn conversation to 58.3 KB.

At 2,000 conversations a day that is 116.6 MB of attributes a day, and 3.5 GB if the shop keeps 30 days, the limit chapter 10 sets. To find rows, use the API rather than SQL: the client filters with a SpanQuery, and REST GET /v1/projects/{id}/spans returns 100 spans a page, up to 1,000.

FAILED TOOL CALLS, THROUGH THE CLIENT
from phoenix.client import Client
from phoenix.client.types.spans import SpanQuery

client = Client(base_url="http://phoenix.shop.internal:6006", api_key="...")

failed_tools = client.spans.get_spans_dataframe(
    project_identifier="returns-agent",
    query=SpanQuery().where("span_kind == 'TOOL' and status_code == 'ERROR'"),
)
print(failed_tools[["name", "start_time", "status_message"]])

COMMON QUESTIONS

SQLite or PostgreSQL?
SQLite for a laptop or a test. PostgreSQL for anything shared: concurrent writers, a read replica (PHOENIX_SQL_DATABASE_READ_REPLICA_URL), managed backups and point-in-time recovery.
Can I query the database directly?
You can, for a one-off look. The schema changes with Phoenix migrations, so scripts should use the REST or GraphQL API.
Does Phoenix keep the raw OTLP request?
No. It decodes each span into a row and discards the protobuf.

WHAT YOU NOW KNOW

  • The receiver queues spans in memory and a bulk inserter writes them in small transactions; past 20,000 queued spans it answers 503.
  • One span is one row with its attributes as one JSON value; root rows carry cumulative token and error counts for their whole trace.
  • Everything Phoenix knows sits in one SQL database: SQLite by default, PostgreSQL 14 or later in production.
DEEP DIVE: the ingestion path, indexes and costs, sizing the database, the GraphQL API, upgrades, the licence

The ingestion path

The bulk inserter is a loop in the server process. Each round it takes what is queued, up to 100 operations, writes them in one transaction, and starts the next round at once; it waits 0.1 s only when the queue is empty or a write failed. Writing in rounds keeps SQLite, which allows one writer at a time, from taking a lock per span.

PHOENIX_MAX_SPANS_QUEUE_SIZE (20,000 spans by default) is the back-pressure. Past it the receiver answers 503, and exporters back off and retry. A 503 under steady load means database writes are the bottleneck, not the network.

Indexes and costs

The spans table is indexed on parent_id and start_time, which serve the two common reads: a span’s children, and the spans in a time range. A filter on an attribute, such as user.id, reads the JSON column of every row in range, so narrow by time first.

span_costs is filled when a span is inserted: Phoenix looks the span’s model up in generative_models, takes its token_prices and multiplies them by the token counts. It refreshes that price list from a manifest it fetches, which PHOENIX_DISABLE_MODEL_COST_AUTO_UPDATE turns off. A model missing from the table gets no cost.

Sizing the database

Step 4’s arithmetic is the whole sizing model: bytes per conversation × conversations a day × days kept. For the example that is 58.3 KB × 2,000 × 30, about 3.5 GB of attributes alone. Indexes, events and the database’s own row overhead come on top.

Measure your own averages rather than trusting these. On PostgreSQL, SELECT span_kind, avg(pg_column_size(attributes)) FROM spans GROUP BY 1 gives the stored size per kind, and the answer moves with prompt length more than with traffic.

The GraphQL API

The UI is itself a client of Phoenix’s GraphQL API at /graphql, and scripts can use it too. Lists are paged with first and after, and first takes at most 1,000 items. For scripts, prefer the REST API under /v1 and the phoenix-client package: they are versioned, and GraphQL is shaped around what the UI needs.

Upgrades

Phoenix runs its Alembic migrations when it starts, so an upgrade changes the schema the moment the new version boots. Back up first: a migration over a large spans table can take a while, and going back means restoring that backup. PHOENIX_DANGEROUSLY_DISABLE_MIGRATIONS skips them, for setups that migrate as a separate step.

The licence

The Phoenix server is under the Elastic License 2.0: you may run it yourself, for your own systems, without paying, but you may not offer it to others as a managed service. The packages your code imports, arize-phoenix-otel and arize-phoenix-client, are under Apache-2.0.

Annotations and evals: scoring what the agent did

Phoenix now holds every turn of every conversation, thousands of them a day. Nobody can read them all, and a trace says what happened, not whether it was right. A score has to be attached to the trace it judges, so you can filter for the bad ones. Phoenix calls such a score an annotation. What is an annotation, who writes one, and how does a judge’s verdict reach the right span?

STEP 01 / 04

THE MAP SO FAR

Fig 6.1The accent parts are new: the annotation table, and a job outside Phoenix that reads spans and writes scores back.

An is a score or a label stored with a span, a trace, a session or one retrieved document. Phoenix keeps annotations in tables of their own, one per kind of target, and each row points at the thing it judges.

The evaluator job is a script you run outside Phoenix on a schedule. It reads spans through the API, grades them and writes annotations back. Phoenix doesn’t grade live traffic by itself: online evaluation is a feature of Arize AX, Arize’s commercial platform.

STEP 02 / 04

AN ANNOTATION IS A NAME, A LABEL, A SCORE AND A REASON

Fig 6.2One row per annotation on the reply span; the accent row is the judge’s, and its explanation quotes the reply.

Each row holds a name, a label, a numeric score, an explanation, metadata, and an annotator_kind: LLM, CODE, HUMAN, DECISION or AGENT. A source column says whether it came through the API or the app. The values here are illustrative, like every count and score in this chapter.

One span can carry many annotations. A row is unique by span, name and identifier, so logging policy_grounded again replaces the judge’s last verdict, while a reviewer’s label, stored under the reviewer’s own identifier, sits beside it. The UI and span queries filter on them: annotations['policy_grounded'].score == 0 lists every ungrounded reply.

STEP 03 / 04

A JUDGE READS SPANS AND WRITES ANNOTATIONS BACK

Fig 6.3Follow 1 to 4: spans out through the API, verdicts from the judge, annotations back onto the same span ids.

An is a model call that grades another model’s output against a rubric. The job pulls the day’s final-reply spans with get_spans_dataframe, builds a classifier named policy_grounded with phoenix-evals, runs it over every row and converts the verdicts with to_annotation_dataframe.

log_span_annotations_dataframe sends them back, each carrying the span id its row came with. Of 200 replies, 23 come back ungrounded. The label-link check is a plain Python function in the same job, logged with annotator kind CODE: it needs no model, so it costs nothing per span.

THE JUDGE JOB, RUN ONCE A DAY
from phoenix.evals import LLM, create_classifier, evaluate_dataframe
from phoenix.evals.utils import to_annotation_dataframe

replies = client.spans.get_spans_dataframe(
    project_identifier="returns-agent",
    query=SpanQuery().where("span_kind == 'LLM'"),
)   # then keep the final reply of each trace, renamed to policy and reply columns

judge = create_classifier(
    name="policy_grounded",
    prompt_template="Policy: {policy}\nReply: {reply}\nDoes the reply follow the policy?",
    llm=LLM(provider="anthropic", model="claude-sonnet-5-5"),
    choices={"grounded": 1.0, "ungrounded": 0.0},
)
results = evaluate_dataframe(dataframe=replies, evaluators=[judge])
client.spans.log_span_annotations_dataframe(
    dataframe=to_annotation_dataframe(results), annotator_kind="LLM"
)
STEP 04 / 04

PEOPLE LABEL IN THE UI, AND THE LABELS CHECK THE JUDGE

Fig 6.4Rows pair a reviewer’s label with the judge’s on the same span; the accent cells are where they disagree.

A reviewer opens a trace in the UI and adds a HUMAN annotation from an annotation config, a fixed set of labels defined once and attached to the project, so every reviewer picks from the same choices. Here a reviewer labels 50 reply spans the judge already scored.

They agree on 45 and disagree on 5. The 3 ungrounded replies the judge passed are the costly cell: customers got answers the policy doesn’t back, and the judge’s count left them out. How many labels you need, and how to tune a judge until it agrees, is the subject of the AI evals tutorial.

COMMON QUESTIONS

Can the judge run inside Phoenix?
Not on live traces. Phoenix’s server-side evaluators run on datasets and experiments, which chapter 08 covers. For production traces, schedule the job yourself, as a cron job or a pipeline step.
Where do thumbs-up clicks from the chat go?
Send them from your backend as HUMAN annotations through the API, keyed by the reply span’s id. Keep that id when the reply goes out, so the click can find it.
Is the judge’s own model call traced?
If the job registers a tracer provider, yes, as traces of their own. phoenix-evals opens an EVALUATOR span for each verdict, with the judge’s LLM span inside it. Send them to a separate project so they don’t mix with customer traffic.

WHAT YOU NOW KNOW

  • An annotation is a name, label, score and explanation attached to a span, trace, session or document, unique by name and identifier.
  • The judge job reads 200 reply spans through the API, finds 23 ungrounded, and logs every verdict back under its span id.
  • Reviewer labels on the same spans check the judge: here it agrees on 45 of 50 and passes 3 replies people failed.
DEEP DIVE: built-in evaluators, annotation configs, document annotations, which spans to judge, what judging costs

Built-in evaluators

phoenix.evals.metrics ships ready-made judges, among them HallucinationEvaluator, FaithfulnessEvaluator, CorrectnessEvaluator, DocumentRelevanceEvaluator and ToolSelectionEvaluator, plus code checks such as exact_match and MatchesRegex. Each LLM one carries a tested prompt and fixed choices.

They are starting points. Read the prompt, and check the evaluator against your own labels before you trust it. A rule like policy_grounded still needs a classifier of your own, because only the shop knows what its policy says.

Annotation configs

An annotation config fixes what a reviewer can enter. Categorical configs list labels, each with an optional score; continuous ones take a score between a lower and an upper bound; freeform ones take text. Categorical and continuous configs also set an optimization direction (maximize, minimize or none), which tells the UI whether a higher score is better.

Attach a config to a project and it appears in the trace view’s annotation panel. Labels typed in by hand, without a config, drift: “ungrounded”, “not grounded” and “wrong policy” become three categories.

Document annotations

A retriever span returns several documents, here the policy paragraphs search_policy found. A document annotation scores one of them, keyed by the span and the document’s position in the result list, so you can see which paragraph was irrelevant, not only that the reply went wrong. DocumentRelevanceEvaluator produces them, and the client logs them with log_document_annotations.

Which spans to judge

Judging every reply is simplest while volume is low. Past that, judge a fixed random share plus every reply that is likely to be bad: a thumbs-down, an ERROR span on the way, a hand-off to a person. Record in the annotation’s metadata which sample a span came from, or the ungrounded rate mixes the two and means nothing.

What judging costs

Each verdict is one model call whose prompt holds the policy paragraphs and the reply, so judging every reply adds a model call per chat turn. The job also draws on the same API rate limits as the agent: run it off-peak, and let phoenix-evals retry rate-limited calls (max_retries) instead of failing the batch.

log_span_annotations_dataframe returns before Phoenix has written the rows; pass sync=True when the job needs the inserted ids back, for example to check that every row landed.

Prompt versions: changing a prompt without a deploy

The label-link check fails on too many replies, and the fix is one line in the system prompt. Today that prompt is a string in the agent’s code: changing it means a deploy, and the traces don’t say which wording produced which reply. Phoenix can hold the prompt itself, in versions. How does the agent get the current prompt, and how does a trace say which version it used?

STEP 01 / 04

THE MAP SO FAR

Fig 7.1The accent parts are new: the prompt tables, and the agent asking the API for the version tagged production.

The prompt store keeps prompts in Phoenix’s database. Each prompt has , which never change once saved, and , names such as production that point at one version and can move to another.

The agent no longer carries its system prompt as a string in the code. When it starts, and again when its copy expires, it asks Phoenix’s API for the version the production tag points at, returns-reply v3.

STEP 02 / 04

A TAG POINTS AT ONE VERSION

Fig 7.2Each row is one version; the accent row is the one the production tag points at today.

returns-reply has 4 versions. Each stores the template (Mustache here, so a variable reads {{order_summary}}), the model name and provider, and invocation parameters such as the token limit. Phoenix gives each version an opaque id; this page calls them v1 to v4. The values are illustrative.

Two tags point into the list: production at v3, candidate at v4. A tag name is unique per prompt, so adding production to another version moves it there, and rolling back is moving it back. Prompt and tag names must match ^[a-z0-9]([_a-z0-9-]*[a-z0-9])?$: lowercase letters, digits, hyphens and underscores.

STEP 03 / 04

THE AGENT FETCHES BY TAG AND FORMATS LOCALLY

Fig 7.3Follow 1 to 4: hops 1 and 2 happen once per cache period, hops 3 and 4 on every chat turn.

client.prompts.get(prompt_identifier="returns-reply", tag="production") returns a PromptVersion. Its format(variables=…, sdk="anthropic") fills the template and returns keyword arguments shaped for the Anthropic SDK’s messages.create, model name included. Underneath it is one REST call: GET /v1/prompts/returns-reply/tags/production.

Phoenix never sees the filled-in prompt at this point: the variables are substituted inside the agent’s process. They reach Phoenix later only as span attributes, which is the next step. Leave out the tag and get returns the latest version, which is whatever someone saved last.

FETCH ONCE, FILL ON EVERY TURN
from functools import lru_cache

@lru_cache(maxsize=1)          # in production, add a time limit and a repo fallback
def reply_prompt():
    return client.prompts.get(prompt_identifier="returns-reply", tag="production")

prompt = reply_prompt()        # the version the production tag points at: v3
kwargs = prompt.format(
    variables={"order_summary": summary, "policy": paragraphs},
    sdk="anthropic",
)
response = anthropic_client.messages.create(**kwargs)
STEP 04 / 04

THE SPAN RECORDS WHICH VERSION IT USED

Fig 7.4The accent row is the only link between the reply’s span and the prompt version that made it.

No foreign key links a span to a prompt version. The link is three span attributes you set yourself. Wrap the model call in using_prompt_template(template=…, variables=…, version="v3") and the Anthropic instrumentor copies them onto the LLM span as llm.prompt_template.template, .variables and .version.

With the version on every reply span, a span query splits replies by version, and when production moves to a new version, the label-link failures before and after the move can be compared. The playground opens a recorded span, swaps in another version and reruns the same call.

THREE ATTRIBUTES THAT NAME THE VERSION
from openinference.instrumentation import using_prompt_template

with using_prompt_template(
    template=TEMPLATE,                      # the version’s template text
    variables={"order_summary": summary, "policy": paragraphs},
    version="v3",
):
    response = anthropic_client.messages.create(**kwargs)
# sets llm.prompt_template.template, .variables and .version on the LLM span

COMMON QUESTIONS

Should prompts live in Phoenix or in git?
Either can be the source. Phoenix gives tags without a deploy and a playground; git gives review. Many teams keep both and sync one way, from git to Phoenix in CI.
Who can move a tag?
Any user or API key whose role can write to Phoenix: admins and members, not viewers. Chapter 10 covers roles.
Does the prompt store work with other SDKs?
Yes. format() fills the argument shape of the Anthropic, OpenAI, Google GenAI and boto3 SDKs, and the model name and provider travel with the version.

WHAT YOU NOW KNOW

  • A prompt has versions that never change and tags that move; returns-reply’s production tag points at v3.
  • The agent fetches the tagged version, caches it with a time limit, and fills each request locally with format().
  • Only the llm.prompt_template attributes link a reply’s span to the version that produced it.
DEEP DIVE: template formats, review in git, playground traces, labels, deleting a prompt

Template formats

A version’s template is Mustache ({{order_summary}}), F-string ({order_summary}) or none, and its type is a chat (a list of messages) or a single string. F-string templates go through Python’s string formatting, so a literal brace, in a JSON example inside the prompt, has to be doubled. Mustache leaves single braces alone.

Review in git

Keep the template in the repo and review changes in pull requests. On merge, CI calls client.prompts.create(name=…, version=…), which saves a new version, and client.prompts.tags.create(prompt_version_id=…, name="candidate"). The production tag moves only after the experiment in chapter 08 says the new version is no worse.

Playground traces

Every run in the playground is recorded as a trace in a project named playground. If you paste a customer’s message from a span into the playground, it is stored there too, outside the agent’s project and its retention policy.

Labels are not tags

Prompt labels attach to a prompt, not a version: support, billing, experimental. They group and filter prompts in the UI and never select a version. Only a tag decides what the agent gets.

Deleting a prompt

client.prompts.delete(prompt_identifier=…) removes the prompt with all its versions, tags and labels, and the database refuses it while an evaluator still runs that prompt. Spans that recorded the template keep their own copy in the llm.prompt_template attributes, so old traces stay readable. An agent still asking for the production tag gets an error and needs its repo copy.

Datasets and experiments: testing a change before it ships

Version v4 of the prompt adds the label link, and the playground shows one good reply. One reply proves little: v4 could break the 30-day rule for another kind of order. You want the same inputs run through v3 and v4 and scored by the same checks. Phoenix keeps those inputs as a dataset and the runs as experiments. Where do test cases come from, and how do you compare two versions on them?

STEP 01 / 04

THE MAP SO FAR

Fig 8.1The accent parts are new: the dataset tables, and a runner outside Phoenix that runs the agent on every example.

A is a versioned set of examples, each an input, an expected output and metadata. An runs a task, here the agent’s reply function, on every example of one dataset version, and scores each output with evaluators: functions or judges that return a score or a label.

The experiment runner is your script, built on the Phoenix client. It reads the dataset through the API, calls the model the way the agent does, and writes each run and each score back.

STEP 02 / 04

A DATASET EXAMPLE IS A COPY OF A SPAN’S INPUT

Fig 8.2Follow the arrows: the span’s input is copied into the example, and only a pointer leads back.

The dataset starts from traces. A span query picks replies, flagged ones first (annotations['policy_grounded'].score == 0), then a spread of ordinary ones, 40 in all. Like every count in this chapter, that number is illustrative. create_dataset(…, span_id_key="context.span_id") turns each row into an example.

The example’s input and output are copies of the span’s values, stored in the dataset’s own tables. The only link back is dataset_examples.span_rowid, a pointer to the span’s row. Every change to the dataset, an added, edited or deleted example, writes a new dataset version, and earlier versions stay readable.

EXAMPLES COPIED FROM FLAGGED TRACES
picked = client.spans.get_spans_dataframe(
    project_identifier="returns-agent",
    query=SpanQuery().where("annotations['policy_grounded'].score == 0"),
)   # flagged replies first, then a spread of ordinary ones: 40 rows

dataset = client.datasets.create_dataset(
    name="returns-golden",
    dataframe=picked,
    input_keys=["question"],
    output_keys=["reply"],
    span_id_key="context.span_id",    # each example keeps a pointer to its span
)
STEP 03 / 04

AN EXPERIMENT RUNS THE AGENT ON EVERY EXAMPLE

Fig 8.3Each row is one example; its two squares are the evaluators’ results for v4’s output.

run_experiment calls the task once per example and passes each output to every evaluator. A task is any Python function of the example’s input; here it is the agent’s reply function pinned to prompt v4, and the evaluators are the label-link check and the policy_grounded judge from chapter 06.

Phoenix stores one run per example: the output, start and end time, any error, token counts and a trace id, because every run is itself traced. Each evaluator’s result is an annotation on the run. Here v4 finishes all 40 runs with 0 errors. Pass dry_run=True first: it runs one random example and records nothing.

PROMPT V4 ON EVERY EXAMPLE
from phoenix.client.experiments import run_experiment

def answer_with_v4(input):
    return answer(input["question"], prompt_version="v4")

def mentions_label_link(output) -> bool:
    return "/returns/label/" in output

experiment = run_experiment(
    dataset=dataset,
    task=answer_with_v4,
    evaluators=[mentions_label_link, policy_grounded],
    experiment_name="v4",
)
STEP 04 / 04

COMPARE V3 AND V4 ON THE SAME EXAMPLES

Fig 8.4Each pair is one evaluator on the same 40 examples; the accent pair is the fix v4 was written for.

Run the same experiment with the v3 task and you have two experiments on one dataset version. The label link goes from 27 of 40 to 39; policy grounding goes from 36 to 37. Phoenix’s compare view lines the runs up by example, so each change shows per example, not only in the totals.

Example by example, v4 passes 12 more on the link and 1 more on grounding, and no example that passed under v3 fails under v4. That is the rule for moving the production tag: the target score improves and no evaluator gets worse. Moving the tag is the step from chapter 07, and it ships v4 without a deploy.

COMMON QUESTIONS

How many examples do I need?
Enough to cover every kind of request the agent gets, past failures first. At 40 examples, a gain of 1 is within noise. The AI evals tutorial covers sizing and confidence.
Can an experiment run in CI?
Yes. The runner is a Python script: run it in the pipeline, read the scores from the experiment it returns, and fail the build if one drops below the last release’s.
Do experiments call the real model?
Yes. The task calls whatever the agent calls, so 40 examples cost 40 agent runs of tokens, plus one judge call per example for policy_grounded.

WHAT YOU NOW KNOW

  • A dataset is a versioned set of examples copied from traces; each example keeps only a pointer back to its span.
  • An experiment runs the task and evaluators on every example, and each run is a trace: v4 ran 40 with 0 errors.
  • On one dataset version, v4 lifts the label link from 27 to 39 of 40 and makes no example worse.
DEEP DIVE: splits and repetitions, server-side evaluators, experiment tags, synthetic examples, deleting an example

Splits and repetitions

A dataset split names a subset of examples, set with split_key names the column that holds it when the dataset is created, here flagged or ordinary. get_dataset(…, splits=["flagged"]) loads one split, and an experiment on it runs only those examples.

repetitions=3 runs the task three times per example. A model’s output varies between calls, so a score from one run per example can move on a rerun. retries, three by default, reruns a task that raised before Phoenix records an error.

Server-side evaluators

Evaluators can also live in Phoenix. An LLM or built-in code evaluator attached to a dataset in the UI runs on Phoenix’s side when you run an experiment from the playground, with its inputs mapped from the example’s input, output, reference and metadata. The client-side evaluators in this chapter run in your script instead.

Experiment tags

An experiment tag names one experiment per dataset, and assigning it to another experiment moves it, like a prompt tag. The reserved tag baseline marks the experiment later runs are compared against: tag the v3 run baseline and every candidate is read against it.

Synthetic examples

Every example copied from a trace copies a customer’s words: order numbers, names, sometimes health details such as the cut hand in Lena Weber’s message. A dataset written by hand, or by a model from the policy, tests the same cases without them. Chapter 09 covers what to strip, and chapter 10 what an erasure has to reach.

Deleting an example

Deleting an example appends a DELETE revision and writes a new dataset version. Earlier versions still hold the example’s input and output, because experiments that ran on them must stay readable. To remove the text itself, delete the whole dataset, or rebuild it without the example and delete the old one.

Redaction: keeping personal data out of traces

Every part is now on the map, and the customer’s personal data has spread through most of them: her message sits in LLM spans, her name and address in a tool result, and copies of her words in judge explanations and dataset examples. The GDPR asks the shop to collect only what a purpose needs, and the cheapest place to do that is before a span leaves the process. This chapter and the next explain what the law asks of a trace store; they are not legal advice. Which fields carry personal data, and where can you remove it before Phoenix stores it?

STEP 01 / 05

THE MAP SO FAR

Fig 9.1The accent boxes are new: a redactor inside the agent, and a Collector that every service’s spans pass through.

Two parts can remove personal data before Phoenix stores it. The redactor is code inside returns-agent that wraps the OTLP exporter and rewrites each span on its way out. The is a separate process that both services export to; it rewrites the spans of all of them and forwards the result to Phoenix.

Neither changes what the agent does or what the customer sees. They change only the record of each step, before it leaves the shop’s processes.

STEP 02 / 05

PERSONAL DATA IS IN MOST SPANS OF A TRACE

Fig 9.2Each row is one field of hero turn 1; the accent row is free text, which holds whatever the customer typed.

is any information about an identified or identifiable person (GDPR Art. 4(1)). In hero turn 1 that covers user.id cus_48213, the session id, her message, the lookup_order result with her name, email and address, and the model’s reply, which uses her name. Policy paragraphs, token counts, durations and span names are not. Like every count in this chapter, these values are illustrative.

Her message also says she cut her hand fitting the part. That is health data, a (Art. 9) with stricter rules. Data minimisation (Art. 5(1)(c)) asks that what you keep is limited to what the purpose needs, and data protection by default (Art. 25(2)) asks that only that much is processed unless someone decides otherwise.

STEP 03 / 05

THE HIDE SWITCHES REMOVE WHOLE FIELDS

Fig 9.3Compare the columns: with the switches on, the accent rows are gone and only counts and timings remain.

OpenInference instrumentors record inputs, outputs, messages and retrieved documents by default. A set of environment variables turns each off, read by every instrumentor at startup (TraceConfig does the same in code). With OPENINFERENCE_HIDE_INPUTS and OPENINFERENCE_HIDE_OUTPUTS on, input.value and output.value hold the string __REDACTED__. The message switches drop the llm.input_messages and llm.output_messages attributes from the span altogether.

Token counts, the model name, timings, span kinds and errors stay, so latency and cost views still work. In the illustrative example, the stored attributes of one LLM span fall from 6.8 KB to 640 bytes. What goes is every word you would read to find out why a reply was wrong.

THE SWITCHES, SET BEFORE THE AGENT STARTS
# read by every OpenInference instrumentor at startup; all default to false
export OPENINFERENCE_HIDE_INPUTS=true            # input.value becomes __REDACTED__
export OPENINFERENCE_HIDE_OUTPUTS=true           # output.value becomes __REDACTED__
export OPENINFERENCE_HIDE_INPUT_MESSAGES=true    # llm.input_messages.* dropped
export OPENINFERENCE_HIDE_OUTPUT_MESSAGES=true   # llm.output_messages.* dropped
export OPENINFERENCE_HIDE_RETRIEVAL_DOCUMENTS=true
STEP 04 / 05

A REDACTING EXPORTER KEEPS THE TEXT AND MASKS THE PERSON

Fig 9.4Follow 1 to 3: a finished span is copied, its matching values masked, and only the copy is exported.

A span processor’s on_end receives a read-only span, so redaction happens one step later, in a wrapper around the exporter. Its export() builds a copy of each span with rewritten attributes and hands the copies to the real OTLP exporter. Regular expressions replace emails with <email>; patterns for phone numbers, IBANs and street addresses are added the same way.

user.id is treated differently: the redactor replaces cus_48213 with u_1746668a5f55b442, an HMAC-SHA256 of the id under a key kept in a vault, never in Phoenix. That is (Art. 4(5)). The shop holds the key and can re-link the hash, so for the shop it is still personal data (Recital 26). It also lets chapter 10’s erasure script find her traces.

Patterns can’t read meaning. “I cut my hand fitting it” matches none of them and is exported as written, so free text still needs the HIDE switches, a detector that reads language, or a short retention period.

AN EXPORTER WRAPPER THAT MASKS AND HASHES
import hashlib, hmac, re
from opentelemetry.sdk.trace import ReadableSpan
from opentelemetry.sdk.trace.export import SpanExporter

EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
KEY = load_key_from_vault()        # kept outside Phoenix

def clean(key, value):
    if key == "user.id":
        return "u_" + hmac.new(KEY, value.encode(), hashlib.sha256).hexdigest()[:16]
    return EMAIL.sub("<email>", value) if isinstance(value, str) else value

class RedactingExporter(SpanExporter):
    def __init__(self, inner: SpanExporter):
        self.inner = inner

    def export(self, spans):
        return self.inner.export([ReadableSpan(
            name=s.name, context=s.context, parent=s.parent, resource=s.resource,
            attributes={k: clean(k, v) for k, v in s.attributes.items()},
            events=s.events, links=s.links, kind=s.kind, status=s.status,
            start_time=s.start_time, end_time=s.end_time,
            instrumentation_scope=s.instrumentation_scope,
        ) for s in spans])

    def shutdown(self):
        self.inner.shutdown()
STEP 05 / 05

THE COLLECTOR APPLIES ONE RULE TO EVERY SERVICE

Fig 9.5Follow the spans top to bottom through the pipeline; the accent stage masks values that match the blocked patterns.

The Collector runs pipelines: receivers take spans in, processors change them in order, and exporters send them on. Here both services export to it instead of to Phoenix, and its redaction processor masks every attribute value that matches a blocked_values pattern. orders-api gets the same rules as returns-agent without a code change or a redeploy.

The redaction processor ships in the contrib distribution (otelcol-contrib 0.165.0) and is beta for traces. It fails closed: with an empty allowed_keys list and no allow_all_keys: true, it removes every attribute. A match is replaced with asterisks, or with a hash when hash_function is set. summary: info adds counts of what it redacted to each span.

ONE PIPELINE FOR EVERY SERVICE
receivers:
  otlp:
    protocols:
      http: { endpoint: 0.0.0.0:4318 }
processors:
  memory_limiter: { check_interval: 1s, limit_mib: 512 }
  redaction:
    allow_all_keys: true          # without this or allowed_keys, every attribute goes
    blocked_values:
      - '[\w.+-]+@[\w-]+\.[\w.]+'      # email addresses
      - 'DE\d{20}'                        # German IBANs
    summary: info
  batch: {}
exporters:
  otlphttp:
    endpoint: http://phoenix.shop.internal:6006
    headers: { authorization: "Bearer <system API key>" }
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, redaction, batch]
      exporters: [otlphttp]

COMMON QUESTIONS

Can I just hide everything?
You can, and traces still show timings, token counts and errors, which is enough for latency and cost. You lose the words, so you can’t see why a reply was wrong.
Is a keyed hash anonymous?
No. The shop can recompute it, so for the shop it stays personal data. The Court of Justice held in EDPS v SRB (C-413/23 P, 2025) that it may not be for a recipient with no reasonable means to re-identify.
Do we need consent to keep traces?
Not necessarily. Legitimate interests (Art. 6(1)(f)) can cover debugging and quality review, after a written balancing test, with the purpose and that interest named in the privacy notice.

WHAT YOU NOW KNOW

  • Personal data sits in user.id, messages, tool outputs and replies; free text such as “I cut my hand” can hold health data (Art. 9).
  • The HIDE switches drop whole fields (an LLM span falls from 6.8 KB to 640 bytes); a redacting exporter masks patterns and hashes user.id.
  • The Collector applies one redaction rule to every service; its redaction processor removes every attribute unless keys are allowed.
DEEP DIVE: lawful basis and the privacy notice, personal data in free text, purpose limitation, the model provider’s side, hashing in the Collector

Lawful basis and the privacy notice

Every processing needs a lawful basis (Art. 6(1)). For traces kept to debug and improve a support agent, the usual candidate is legitimate interests (Art. 6(1)(f)). The EDPB’s Guidelines 1/2024, final version 2.0 adopted on 10 November 2026, set three cumulative conditions: a legitimate interest, processing that is necessary for it, and the customer’s interests and rights not overriding it.

Write the balancing test down and keep it. The privacy notice names the purpose and legal basis (Art. 13(1)(c)), the legitimate interest pursued (Art. 13(1)(d)), and the retention period or the criteria that set it (Art. 13(2)(a)). Chapter 10’s 30-day policy is the period that sentence quotes.

Personal data in free text

Regular expressions catch formats: emails, phone numbers, IBANs. Names, loosely written addresses and health details need a detector that reads language, such as Microsoft Presidio, which combines named-entity recognition with pattern recognisers. It runs in the redactor, before export.

A detector costs CPU time on every span and still misses entities, and a false positive masks words you needed. Treat it as one layer beside the HIDE switches and a short retention period, never as the only one.

Purpose limitation

Traces collected to debug the agent can’t simply be reused for an incompatible purpose (Art. 5(1)(b)). Training or fine-tuning a model on them is a new purpose. Art. 6(4) lists what decides whether it is compatible: the link between the purposes, the context, the kind of data, the consequences, and safeguards such as pseudonymisation.

Minimisation needs access control beside it. What you do keep must be readable only by people who need it (Art. 32), which chapter 10 sets up.

The model provider’s side

Every message in a trace was already sent to the model API. The provider processes it as the shop’s processor, under its own contract (Art. 28) and its own retention, and tracing changes nothing there. Phoenix holds a second copy under the shop’s control, and redaction before export protects that copy, not the request the model already received.

Hashing in the Collector

The Collector has two ways to hash a value. The attributes processor’s hash action is SHA-256 with no key, and the OpenTelemetry docs warn that hashes of a small input space, such as customer ids, can be reversed by hashing every candidate. The redaction processor’s hash_function accepts hmac-sha256 and hmac-sha512 with an hmac_key, which keeps the property the redactor’s HMAC has.

Hash a field in one place only. If the redactor and the Collector both hash user.id, with different keys or formats, the erasure script has two values to search for.

ERRORS THIS CHAPTER EXPLAINS

Spans arrive in Phoenix with names and no attributes
The redaction processor has an empty allowed_keys list and no allow_all_keys: true, so it removes every attribute.
List the keys to keep, or set allow_all_keys: true and let blocked_values do the masking.
401 Unauthorized in the Collector’s otlphttp exporter log
Phoenix requires an API key, and the Collector sends none.
Put a system API key in otlphttp.headers as authorization: Bearer ….

Retention, access and erasure: how long, who, and how to forget one person

Redaction keeps most personal data out of Phoenix, but some stays, and the keyed user id is there on purpose so the shop can find one customer’s traces. Three questions are left: how long Phoenix keeps a trace, who can read it, and what happens when Lena Weber asks the shop to erase her data. Phoenix has a part for each: a retention sweeper, access control, and the REST API an erasure script calls. How do retention, roles and one erasure request work, and what does a delete leave behind?

STEP 01 / 05

THE MAP SO FAR

Fig 10.1The accent parts are new: the sweeper inside Phoenix, the access check every client passes, and the erasure script.

Three parts finish the map. The retention sweeper runs inside Phoenix and deletes traces past an age or a count, under a . Access control sits in front of the UI and the API, and decides who signs in and what each role may do.

The erasure script is the shop’s own tool, run from its ops tooling. It answers a request under the through Phoenix’s REST API, with an API key, like any other client.

STEP 02 / 05

THE DEFAULT POLICY KEEPS TRACES FOREVER

Fig 10.2The lower ruler is one project’s traces by age; the accent slice is what a 30-day policy keeps.

Storage limitation (Art. 5(1)(e)) says data that identifies a person is kept no longer than its purpose needs. Phoenix starts with one policy, Default, whose max_days comes from PHOENIX_DEFAULT_RETENTION_POLICY_DAYS; its default, 0, means keep forever. A policy’s rule is max_days, max_count or max_days_or_count, and a project can have a policy of its own.

The shop sets 30 days for project returns-agent, the period its privacy notice states. The sweeper runs on the cron schedule 0 0 * * 0, Sundays at 00:00, and deletes in batches of 1,000 traces. Deleting a trace removes its spans, their annotations and their costs. The trace ages in the figure are illustrative.

THE DEFAULT POLICY, SET WHEN THE CONTAINER STARTS
docker run -d --name phoenix \
  -p 6006:6006 -p 4317:4317 \
  -e PHOENIX_SQL_DATABASE_URL="postgresql://phoenix@db.shop.internal/phoenix" \
  -e PHOENIX_DEFAULT_RETENTION_POLICY_DAYS=30 \
  -e PHOENIX_DEFAULT_ADMIN_INITIAL_PASSWORD="..." \
  arizephoenix/phoenix:22.0.0
# 0 (the default) keeps traces forever; the sweep runs on cron "0 0 * * 0"
STEP 03 / 05

A TRACE STORE IS A DATABASE OF CONVERSATIONS, SO LOCK IT

Fig 10.3Each row is one role or key; the accent row is read-only and can still read every conversation.

Security of processing (Art. 32) asks for protection that fits the risk, and a trace store holds customers’ own words. Phoenix’s access control is off for a local phoenix serve on loopback. Since v21 it turns on by itself when Phoenix binds a non-loopback address, which the Docker image does, so a Docker Phoenix asks for sign-in and API keys. PHOENIX_ENABLE_AUTH sets it explicitly.

Roles are global: ADMIN, MEMBER and VIEWER each apply to every project on the server. Exporters and jobs use system API keys, which only an admin creates; people use user keys, which carry their own role. Sign-in can come from OAuth2 and OpenID Connect or LDAP, and TLS from the PHOENIX_TLS_* settings. Encrypt the database volume and its backups too.

A leaked trace store is a personal data breach. The shop reports it to its supervisory authority within 72 hours where feasible (Art. 33), and tells the customers affected when the risk to them is high (Art. 34).

STEP 04 / 05

AN ERASURE REQUEST REACHES FURTHER THAN THE TRACES

Fig 10.4Each row is a place her data was stored; the accent rows survive the trace delete and need steps of their own.

Lena Weber asks the shop to erase her data (Art. 17), and it answers within one month (Art. 12(3)). The erasure script computes the keyed hash of cus_48213, u_1746668a5f55b442, and queries root spans with that user.id. It deletes each trace with DELETE /v1/traces/{id}, then each session with DELETE /v1/sessions/{id}, which also removes any turn that carried no user.id.

In this illustrative example that is 5 traces, 35 spans and 2 sessions. Nothing copied from them goes too. The 2 dataset examples that copied her messages keep their text with the span link set to NULL, earlier dataset versions still hold them, and 4 experiment runs keep outputs whose trace ids point at nothing. Each needs its own delete, and so do exported files.

The model API provider received the same messages. Deletion there runs through the shop’s contract with it as a processor (Art. 28(3)(e) and (g)), and the shop tells each recipient it disclosed her data to about the erasure (Art. 19).

ONE CUSTOMER’S TRACES, FOUND BY KEYED ID
import hashlib, hmac, httpx

def erase_customer(customer_id: str) -> None:
    uid = "u_" + hmac.new(KEY, customer_id.encode(), hashlib.sha256).hexdigest()[:16]
    roots = client.spans.get_spans_dataframe(
        project_identifier="returns-agent",
        query=SpanQuery().where("user.id == '" + uid + "'"),
        root_spans_only=True,
    )
    with httpx.Client(base_url=PHOENIX, headers=AUTH) as http:
        for trace_id in roots["context.trace_id"].unique():
            http.delete("/v1/traces/" + trace_id).raise_for_status()
    sessions = roots["attributes.session.id"].dropna().unique().tolist()
    client.sessions.bulk_delete(session_ids=sessions)   # emptied sessions, and any
                                                        # turn that had no user.id
    erasure_log.append(uid)   # re-run this list after every restore
    # still to do by hand: dataset examples, experiment outputs, exports
STEP 05 / 05

DELETED ROWS STAY IN THE FILE UNTIL IT IS COMPACTED

Fig 10.5Each row is one layer that can still hold a deleted row; the accent row is the one a DELETE never reaches.

A DELETE removes a row from every query, and its bytes can stay on disk. Phoenix opens SQLite with journal_mode = WAL and never sets secure_delete, which the SQLite docs say is normally off. So a deleted row’s bytes stay in freed pages until a write reuses them or VACUUM rewrites the file, and the -wal file holds recent copies of changed pages until a checkpoint.

PostgreSQL behaves the same way at a different layer. A DELETE leaves a dead row version; VACUUM marks its space free for reuse, and only VACUUM FULL writes a new table file without it. WAL archives and base backups keep older copies for as long as they are retained.

Backups keep everything. The EDPB’s report on the right to erasure, adopted in February 2026, says editing a backup is not always advisable, and that a controller who leaves it intact should track erasure requests and apply them on restored systems. Keep an erasure log, the keyed ids and dates and never the data, and re-run it after every restore.

WHAT VACUUM CHANGES IN PHOENIX’S SQLITE FILE
sqlite3 ~/.phoenix/phoenix.db "PRAGMA journal_mode;"     # wal: Phoenix sets it
sqlite3 ~/.phoenix/phoenix.db "PRAGMA secure_delete;"    # 0: freed pages keep old bytes

# stop Phoenix first, then rewrite the file without the freed pages
sqlite3 ~/.phoenix/phoenix.db "VACUUM; PRAGMA wal_checkpoint(TRUNCATE);"

COMMON QUESTIONS

Does the AI Act make us keep agent logs?
Its log-keeping duties (Art. 12, 19 and 26(6)) cover high-risk systems, and a shop’s returns chat isn’t one. Its rule for chatbots, applying since 2 August 2026, is to tell people they are interacting with an AI (Art. 50(1)).
Can we use hosted Phoenix instead?
Then the vendor is a processor: you need an Art. 28 contract, its list of sub-processors and where it stores data. Phoenix’s docs no longer state a hosted region, so ask before sending traces.
Does an access request include traces?
Yes. Art. 15 gives her a copy of her personal data wherever it is kept, so the keyed id that finds her traces for erasure also answers access requests.

WHAT YOU NOW KNOW

  • Phoenix keeps traces forever by default; a 30-day policy deletes older ones each Sunday, with their spans, annotations and costs.
  • Roles and API keys decide who reads conversations; every role covers every project, and since v21 a Docker Phoenix requires sign-in.
  • An erasure deletes traces and sessions, then each copy separately; deleted rows stay on disk until VACUUM and in backups until they expire.
DEEP DIVE: records and DPIAs, processors and transfers, sizing retention, the AI Act’s dates, fines

Records and DPIAs

Art. 30 asks for a record of processing activities. The exemption for organisations with fewer than 250 employees doesn’t apply when the processing is not occasional, includes special categories or is likely to result in a risk (Art. 30(5)), and a trace store that records every chat is not occasional.

A data protection impact assessment (Art. 35) is required when processing is likely to result in a high risk, particularly with new technologies. Free text that can hold health data, recorded at scale by an LLM agent, is a strong reason to write one before launch.

Processors and transfers

Every vendor that touches traces is a processor under an Art. 28 contract: hosted Phoenix if you use it, the model API provider, and the cloud that holds the disks and backups. The contract has the processor help answer data subject requests (Art. 28(3)(e)) and delete or return the data when the service ends (Art. 28(3)(g)).

Sending personal data outside the EEA needs a basis under Chapter V. For a US provider certified under the EU–US Data Privacy Framework, the Commission’s adequacy decision (EU) 2023/1795 is that basis; otherwise, standard contractual clauses (Decision (EU) 2021/914) with a transfer impact assessment.

Sizing retention

Retention also sizes the disk. In the illustrative example, one conversation of 3 turns stores 58.3 KB of span attributes. At 2,000 conversations a day that is 116.6 MB a day, and 3.5 GB for 30 days of attribute JSON alone.

A max_count rule caps rows instead of age. It bounds the disk, but the period a trace is kept then depends on traffic, and a privacy notice states a period, so max_days or max_days_or_count is the rule you can promise.

The AI Act’s dates

The AI Act (Regulation (EU) 2024/1689) applies in stages. Its transparency rule for systems that interact with people, Art. 50(1), applies from 2 August 2026: a chatbot tells people they are talking to an AI unless that is obvious.

Regulation (EU) 2026/1744, the digital omnibus on AI, in force since 27 July 2026, moved the high-risk rules: stand-alone Annex III systems from 2 December 2027, and systems in products covered by Annex I from 2 August 2028. Keeping logs for at least six months (Art. 19 and 26(6)) is one of those high-risk duties, so a returns chat isn’t covered.

Fines

Art. 83 sets two ceilings. Breaches of the controller’s and processor’s duties, such as Art. 25, 28, 30, 32 and 35, reach €10 million or 2% of worldwide annual turnover. Breaches of the principles, the lawful basis and data subjects’ rights, such as Art. 5, 6, 9, 15 and 17, reach €20 million or 4%. For an undertaking, whichever is higher applies.

ERRORS THIS CHAPTER EXPLAINS

A session still lists turns after the erasure
Some turns carried no user.id, so the query by keyed id missed their traces.
Delete the whole session with DELETE /v1/sessions/{id}, which cascades to its traces.
The disk didn’t shrink after retention ran
Deleted rows leave free pages (SQLite) or dead row versions (PostgreSQL) inside the files.
Stop Phoenix and run VACUUM on SQLite; on PostgreSQL, VACUUM FULL or pg_repack rewrites the table.

The whole picture

Every part from the ten chapters, numbered. Every numbered part has a row in the table below.

Numbers match the table below. Spans travel left to right; people and scripts reach Phoenix from below, through access control.
  1. 1tracer provider

    creates spans and tracks the current one

    runs in
    the agent’s process
    decides
    trace and span ids, parents, start and end times, status
    when wrong
    spans with no parent: every step its own trace
  2. 2instrumentors

    turn SDK calls into typed spans

    runs in
    the agent’s process
    decides
    span kind, messages, token counts, tool calls
    when wrong
    model calls as bare HTTP spans with no tokens
  3. 3orders-api tracer and traceparent

    continue the trace in a second service

    runs in
    orders-api
    decides
    the caller’s trace id and span id
    when wrong
    a tool span with nothing under it
  4. 4span processor and OTLP exporter

    queue finished spans and send them

    runs in
    every traced process
    decides
    when to send, batch size, endpoint, protocol
    when wrong
    no traces, or traces missing their last five seconds
  5. 5OTLP receiver

    accept spans

    runs in
    Phoenix
    decides
    protobuf over HTTP on 6006, gRPC on 4317
    when wrong
    401, 415 or 503 in the exporter’s log
  6. 6database

    store every span as a row

    runs in
    Phoenix’s SQLite file or PostgreSQL
    decides
    attributes as JSON, cumulative counts, costs
    when wrong
    an empty Phoenix after a container restart
  7. 7UI, REST and GraphQL API

    read the database for people and scripts

    runs in
    Phoenix, port 6006
    decides
    filters, span queries, pages of 100 to 1,000
    when wrong
    scripts that break after an upgrade because they read tables directly
  8. 8annotations and evaluator job

    score spans

    runs in
    annotation tables; a job you schedule
    decides
    name, label, score, explanation, annotator kind
    when wrong
    scores nobody can trace back to a span
  9. 9prompt store

    hold prompt versions and tags

    runs in
    Phoenix’s database
    decides
    which version a tag points at
    when wrong
    replies you can’t tie to a prompt version
  10. 10datasets and experiment runner

    test a change on fixed examples

    runs in
    Phoenix’s database; your script
    decides
    examples, runs, scores
    when wrong
    a “better” version tested on a different dataset version
  11. 11redactor

    mask personal data before export

    runs in
    the agent’s process
    decides
    which values are masked; the customer id becomes a keyed hash
    when wrong
    customer emails in Phoenix’s search
  12. 12Collector

    one pipeline for every service

    runs in
    its own process
    decides
    redaction and attribute rules, where spans go
    when wrong
    spans with names and no attributes
  13. 13retention policy

    delete old traces

    runs in
    Phoenix
    decides
    maximum age or count, and the schedule
    when wrong
    traces older than the privacy notice promises
  14. 14access control

    decide who reads and who sends

    runs in
    Phoenix
    decides
    sign-in, roles, API keys
    when wrong
    anyone on the network reading conversations
  15. 15erasure script

    forget one customer

    runs in
    the shop’s ops tooling
    decides
    which traces and sessions go, then the copies
    when wrong
    an erased customer back after a restore
  16. ·session

    group one conversation’s traces

    runs in
    the session.id attribute
    decides
    which traces are one chat
    when wrong
    each turn alone, with no conversation view
  17. ·prompt template attributes

    link a span to its prompt version

    runs in
    the llm.prompt_template attributes
    decides
    template, variables, version
    when wrong
    no way to filter replies by version

THREE THINGS TO KEEP

  • A trace is a timed tree of every step an agent took. Phoenix stores each step as one database row.
  • Scores, prompt versions and test sets all point back at those rows, so one trace can explain a bad answer.
  • A trace holds the customer’s own words. Decide what to keep, for how long, and how to erase it first.
Read next. AI evals for how the scores in chapters 06 and 08 are made trustworthy, and AI agent patterns for how an agent like this one is built.

every span closed, every copy accounted for