Run & observe
Logs & traces
Every run on the platform — playground chats, swarm nodes, notebook calls — is recorded as a trace. Reading traces is the core debugging skill in agentic systems, and the one course environments almost never let you practice.
The traces table
/traces lists every run with the agent name, provider and model, latency, tokens in and out, dollar cost, status, and timestamp. Sort or scan for the rows that look wrong — the red statuses, the latency outliers, the runs that cost ten times their neighbours.
What a trace contains
Selecting a run opens the full record:
- Metrics
- Latency, tokens in, tokens out, and cost computed from model pricing — attributed to the user, agent and credential that caused it.
- Resolved system prompt
- What the model was ACTUALLY told, after retrieval, memory and routing guidance were folded in — not the template you configured. Usually the most surprising part of the record, and the first thing to read.
- Tool calls
- Each tool the model called, with the arguments it chose and the result it got back. If a tool you expected is never called, check the request payload to confirm it was offered at all.
- Request / response payloads
- The raw provider request and response. This is the ground truth: the exact message array, parameters, and tool definitions the model actually received, and exactly what it returned.
- Error
- For failed runs, the error message the runtime captured.
The playground inspector
While chatting in the Playground, the inspector panel shows the same information live, in three tabs: the latest request/response exchange, the stream of tool events as they happen (with a running call count), and the trace for the current conversation. For swarm runs, the observability view adds the per-node timeline.
Why it works this way
Three habits
- Reproduce, then read. Re-run the failing input, open the trace, and read the request payload before forming a theory. Most "the model is broken" reports turn out to be "the model was sent something other than what I assumed".
- Change one thing. Adjust a single line of prompt, one parameter, or the model — then run the same input and compare the two traces. Keeping the old trace open in a second tab is the closest thing prompt engineering has to a scientific method.
- Watch cost as a signal. A run whose cost jumps an order of magnitude usually means a loop, a context blow-up, or a tool feeding the model far more text than intended — the trace shows which.
Symptom to cause
The same handful of failures account for most of them, and each has a signature in the trace that identifies it in seconds. Read the row that matches what you saw:
| What you saw | What the trace shows | Cause |
|---|---|---|
| A confident answer that is simply untrue | The resolved system prompt contains no retrieved context, or a retrieval block with zero passages | Retrieval matched nothing and the model answered from training. Fix the collection or the prompt's refusal rule — not the model. |
| It ignored a tool you know it has | The request payload's tool list does not contain it | Not enabled on this agent, or not allow-listed. If it IS in the list, the description is too vague to match the question. |
| A number that is wrong but plausible | No sql_query call — the answer came from a document | It read a figure out of prose instead of counting. Attach the table and enable the tool. |
| The answer stops mid-sentence | Tokens out sits exactly at the configured maximum | max_tokens truncation. Common cause of JSON that will not parse. |
| One run cost 20× its neighbours | Tokens in is enormous while the question is short | Retrieval or conversation history is dominating the prompt — and being paid for every turn. |
| It says it did something it did not do | No tool call for the action it described | The model narrated an intention. Only a tool call in the trace is evidence that anything happened. |
| It worked yesterday, fails today | A tool call returning an error the answer never mentioned | An upstream change. Models rarely announce a failed tool; they answer around it. |
An empty result and a failed call look identical in the answer
A debugging order that works
- 1
Read the resolved system prompt first
Half of all surprises are here — a retrieval block that came back empty, memory that recalled something stale, or routing guidance that pushed the model at the wrong tool. - 2
Then the tool calls, in order
Look for a call that returned an error or an empty result. Models rarely announce that a tool failed; they answer anyway. - 3
Then the token counts
A large input with a small question means retrieval or conversation history is dominating — and paying for it every turn. - 4
Only then change the prompt
Most prompt edits made before reading the trace fix the wrong thing.
Swarm traces
A swarm run records per-node steps, so you can see which branch a router chose, which nodes were skipped, where an approval waited, and what each node wrote to flow state. Runs triggered through the API are traced identically and attributed to the key that started them.
Retention, and what a regulated tenant can turn off
Everything on this page rests on storing what people typed and what models replied. That is what makes debugging good and what makes the trace store sensitive. Two controls, and they do different jobs:
| Control | Where | Effect |
|---|---|---|
PERSIST_PROMPT_BODIES | Environment (default ON) | Set it to false and free text is never written: prompts, model responses, node inputs and outputs, chain-of-thought. |
Trace retention (trace_retention_days) | Admin → IAM → Settings | Deletes traces older than the window. 0 keeps them indefinitely. |
Turning bodies off keeps the skeleton
It is not retroactive