Make Every Run Auditable

Sooner or later, someone will question a Run: an auditor, a customer, a teammate, or you at 9am after a night of scheduled Runs. An auditable Agent answers that question in minutes — what happened, what was produced, and whether it was right — from the record the platform already keeps. This page covers what Duvo records automatically, how to read it, and the parts the platform cannot do for you: durable outputs, stable identifiers, and alerting.

​

What the Platform Records for Every Run

Every Run keeps a durable record you can open later from the Runs List:

• The full transcript — every message and tool call, kept on the Run permanently.
• Status and timestamps — one of eight statuses (Completed, Failed, Stopped, Needs Input, and the in-flight states), with start and completion times.
• A screenshot replay for browser work — after the Run ends, the Live View tile is replaced by a step-by-step replay, one frame per browser action.
• The Case timeline for Queue work — every claim, update, settlement, and approval on the Case, with each Run as a card (see below).
• An Evaluation — an automatic quality verdict on every completed Run (next section).
• Workspace files — everything the Run wrote in its sandbox, browsable and downloadable from the Run detail. These are temporary (see “Outputs Must Outlive the Sandbox”).

One honest caveat: Duvo does not write a prose summary of what a Run did. The transcript is the record of what happened; the Evaluation is the verdict on whether it went well. If you want a human-readable recap per Run, instruct it in the AOP — end every Run with a short report of what was done, where results were written, and under which identifiers. For Queue work, have the Agent write that recap into the Case data, where it outlives everything else.

​

Evaluations: Your Built-In QA

Every Run that completes is evaluated automatically — no setup, no opt-in. An LLM judge (separate from the model that did the work) reads the complete Run trace and scores it against two sets of criteria:

• Platform evaluations — a fixed catalog of failure checks every Run is scored on, grouped by cause: Agent Misbehavior, AOP Gap, Platform Issue, and User Access Gap.
• Agent specific evaluations — up to five Pass/Fail questions derived automatically from your AOP, scoped to the Revision the Run actually ran against.

Each criterion gets a verdict (Pass, Failed, or Not applicable), a severity (Critical — blocks the outcome, Medium — a defect needing intervention, Low — verified harmless), and a short plain-language reason. The judge also writes an AI Analysis: the single most useful change to make the Agent faster or more reliable next time — a recommendation, not a recap. Where to read them:

• Runs List — each row carries a badge: Success, Issues (colored by worst severity), Evaluating…, or N/A. Filter the list with the Issues dropdown (All issues / Critical / Medium / Low).
• Run detail — the evaluation banner opens the Run Evaluation Analysis dialog with the per-criterion breakdown and reasons.
• The Agent’s Evaluations tab — issue rates across Runs (“Issues found”, Today / Last 7 days) with a per-criterion breakdown, scopable to a specific revision, plus Issues analysis, an AI summary of recurring problems across Runs.

Two practices make Evaluations dramatically more useful:

• Write measurable success criteria into the AOP. The Agent specific evaluations are derived from it. “Reply to every review in its own language, using the matching template” derives a sharp Pass/Fail question; “handle reviews appropriately” derives nothing testable. This is the same advice as Writing Effective AOPs, paying off a second time.
• Curate the Agent specific evaluations for Agents you rely on. They are editable via the API, MCP tools, or the CLI (duvo agents eval-rubrics list | add | update | remove) — each is a short title plus a one-to-two sentence Pass condition, at most five per Revision. Replace a vague generated question with the check your business actually cares about.

​

Turn Evaluations into Alerts

By default, evaluations are pull, not push: nobody is emailed when a Run fails its checks. Two opt-in switches change that: For programmatic monitoring, the public API exposes the same signal: the Runs list endpoint supports has_issues=true and issue_severity=critical|medium|low filters, getRunEvaluation returns the full per-criterion result, and getEvalScores returns issue counts and flag distribution over a time window — enough to chart a trend or wire a check into your own monitoring. See Monitoring Runs via API.

​

Outputs Must Outlive the Sandbox

The files a Run produces live in its sandbox workspace, and the sandbox is reclaimed roughly 12 hours after the Run. Until then you can browse and download them as a ZIP from the Run detail; after that, the Files panel reports the sandbox no longer exists and the files are gone. The transcript and evaluation survive; loose files do not. The rule that follows: a deliverable is not delivered until it’s in a system of record. Make the AOP’s final steps write the output somewhere durable through a Connection — the spreadsheet, the drive folder, the ERP, the email — and then record where it went:

• Give every item a stable identifier (order number, ticket ID, invoice number) and use it in everything the Agent writes to external systems, so a result can be traced back to its Run and Case.
• Have the Agent state its outputs at the end of the Run: what was written, to which system, under which identifier. For Queue work, into the Case data; otherwise, as the Run’s closing message.

​

The Case Is the Audit Trail for Queue Work

For Agents working a Queue, the Case is the record with the longest memory. Its Activity timeline records every event — created, updated, completed, failed, postponed, re-processed, priority changes, manual interventions, and every approval requested and resolved — and each Run appears as a card carrying its status, trigger, timestamps, and evaluation verdict. When someone asks “why was this order handled that way?”, the Case answers without opening a single transcript. Cases also get their own evaluation: when a Case settles as completed or failed (including manual settlements), it is judged against outcome-oriented Pass/Fail criteria derived from the AOPs of every Agent wired to the Queue — did the Case end with the required fields, the correct final decision, and the settlement those AOPs call for. The verdict appears in a Case evaluation panel on the Case detail, each criterion with the judge’s reason, always against the criteria that were in force when the Case ran. That record is only as good as what the Agent writes into it. Update the Case data with the structured results of each stage — the fields checked, the values computed, the decision taken — not just a final status. Designing Work Around Queues and Cases covers the settlement rules; the audit angle is the same discipline with a second payoff.

​

Know How You’ll Hear About Problems

Be deliberate about your alerting surface, because the defaults are quieter than most teams assume: Two consequences worth designing for:

• Scheduled and API Runs fail silently by default. If a failure must page someone, either opt into evaluation emails, poll the API, register a webhook_url when starting Runs via the API (run_completed / run_failed / run_interrupted events), or make notification part of the work: give the Agent a Slack or email Connection and an AOP step to report failures to your ops channel.
• A “quiet week” can mean an auto-paused schedule. The circuit breaker protects you from infinite skips, but the resume is manual — reconnect, then switch the schedule back on.

For the human rhythm around all of this in an Agent’s early days — daily checks, spot-checking outputs, anomaly response — see Monitor the First Week. For the compliance-grade record of who did what in the workspace, see the Audit Log.

​

Related