Decision records

Context

Transparency is a core goal of PawDesk, and it has two separate audiences:

1. Runtime: how the agents processed a specific booking (spans, prompts, costs, approvals). The observability dashboard covers this.
2. Build time: how each feature was specified, decided, and verified.

Build-time transparency needs structured source data that is tied to the code and tests. Hand-written prose goes out of date as the code changes.

Options considered

1. No spec tool. A build-history page written by hand. This is simple, but it isn't tied to the code and goes out of date.
2. A home-grown spec format. Full control, but it rebuilds what an existing tool already provides, and an unfamiliar format makes the project harder to follow.
3. OpenSpec. Each change gets a proposal, spec deltas (requirements with WHEN/THEN scenarios), a design, and tasks. Finished changes are archived with a date, and their spec changes merge into the current specs.

Decision

Option 3. Every feature starts as an OpenSpec change. Each change is reviewed and approved before implementation begins, and archived when it's done.

Runtime transparency and build-time transparency are shown on separate pages: the observability dashboard and the build walkthrough.

Each kind of information lives in one place:

| Information | Lives in | |---|---| | Current system behavior, and the plan for each change | openspec/ | | Decisions that span more than one change | docs/decisions/ (a change's design.md links here) |

Consequences

• Good: The build walkthrough page is generated from repo data (archived changes, specs, test results), so it can't drift from reality.
• Good: Requirements are traceable: spec → test → latest result.
• Cost: Every test must name the requirement it verifies. The convention is set with the first spec, because adding it to existing tests later is painful.
• Cost: An extra tool to install (a global npm package that needs Node ≥ 20.19). Its anonymous telemetry is turned off.

Context

PawDesk has three runtime pieces: a FastAPI backend, a Next.js frontend, and PostgreSQL. Development happens on Windows with Docker Desktop. A single docker compose up should start the whole system for demos and end-to-end checks.

Docker Desktop on Windows and macOS shares source files into a Linux VM. File-change events often fail to cross that boundary or arrive late. The Next.js documentation warns that Fast Refresh inside such containers can take seconds or even minutes, and it advises against working around this with polling, because polling adds latency and CPU load.

Options considered

1. Everything in Compose, with source bind-mounted and hot reload in the containers. One command for everything, but slow or broken reloads on Windows, and polling for file watching.
2. Postgres only in Compose; apps always on the host. Fast reloads, but there's no single command for the full system, and the containers are never exercised before CI.
3. Both modes. Compose builds and runs the full stack from images with no bind mounts. Daily development runs only Postgres in Docker and the apps on the host.

Decision

Option 3.

• docker compose up --build: Postgres, the backend (uvicorn without reload), and the frontend (a standalone Next.js production build). This is used for demos and end-to-end checks.
• Daily development: docker compose up -d postgres, then uv run uvicorn app.main:app --reload and pnpm dev on the host.

Configuration differs only by environment variables. The backend reads DATABASE_URL (the host is localhost on the host machine, and postgres inside Compose). The frontend reads a server-only BACKEND_URL.

Consequences

• Good: Instant reloads during development, and the container images are exercised whenever the full stack runs.
• Good: The full-stack mode runs production-like images, so build problems show up locally instead of in CI.
• Good: The frontend container gets no env_file, so backend secrets never reach it. A check of the running containers confirmed OPENROUTER_* variables are present in the backend and absent from the frontend.
• Cost: Two ways to run the system. Code changes need --build before they show up in Compose mode.
• Cost: Tests always run on the host against the Postgres container. They never run inside the backend image, which leaves out dev dependencies.

Context

PawDesk's orchestrator runs model calls concurrently with asyncio (intake and records extraction in parallel) and saves state to Postgres after every step. The API serves Server-Sent Events, which keep many connections open at once. Any database call that blocks the event loop stalls every in-flight run and stream.

Development happens on Windows hosts, where Python's default event loop is ProactorEventLoop. Production and Compose run on Linux.

Options considered

1. Sync SQLAlchemy, with calls offloaded to a thread pool. Mature and simple, but every database call made from async code has to be wrapped. Forgetting once blocks the event loop without any visible error.
2. Async SQLAlchemy with psycopg 3. One driver for both sync and async code, but its async mode refuses ProactorEventLoop. That requires event-loop workarounds in tests, migrations, scripts, and the dev server on Windows.
3. Async SQLAlchemy with asyncpg. Async-only, the most widely used async driver with SQLAlchemy, and works on both Windows event loops.
4. Raw asyncpg without an ORM. Fastest, but gives up the unit of work, optimistic version checks, and Alembic autogenerate.

Decision

Option 3. All application database access goes through SQLAlchemy 2's async engine with asyncpg (postgresql+asyncpg://), created in app/db.py. .env keeps a driver-neutral postgresql:// URL, which app/db.py rewrites.

Rules that come with this choice:

• Sessions use expire_on_commit=False. In async code, reading an expired attribute would trigger an implicit query that can't run.
• Relationships default to lazy="raise". Related rows are loaded explicitly (for example with selectinload), so a missing load fails in tests instead of failing at runtime.

Consequences

• Good: Database work never blocks the event loop. The same code runs on Windows hosts and Linux containers with no event-loop configuration.
• Good: One access pattern for the API, the orchestrator, scripts, and tests.
• Cost: Two Postgres drivers are installed. psycopg remains for the synchronous /health/db probe.
• Cost: Async SQLAlchemy has sharp edges (implicit I/O, session lifetime across tasks). One AsyncSession must never be shared between concurrent tasks; each parallel step gets its own session.

Context

Requirements live in openspec/specs/ as ### Requirement: blocks with WHEN/THEN scenarios (decision record 0001). The build walkthrough page has to show, for any requirement, which tests verify it and whether they passed in the latest run. That needs a machine-readable link from each test to its requirement. The link has to exist from the very first spec, because adding it to hundreds of existing tests later is slow and error-prone.

Options considered

1. Naming convention (for example, test_booking_lifecycle__only_listed_transitions). Needs no tooling, but requirement names contain spaces and punctuation. Renames silently break the link, and one test can't cover two requirements.
2. Docstring tags (for example, Spec: booking-lifecycle / …). Easy to read, but it's free text that needs custom parsing, and nothing validates it.
3. A registered pytest marker, @pytest.mark.spec("<capability>", "<requirement name>"). It's structured and validated by pytest (--strict-markers rejects typos in the marker name). A test can carry several markers, and the marker can be exported through pytest's standard JUnit XML.
4. A separate mapping file (YAML from requirement to test IDs). Keeps tests untouched, but drifts from the code it describes.

Decision

Option 3.

• Every test that verifies a requirement carries @pytest.mark.spec(capability, requirement), where capability is the spec folder name and requirement is the exact ### Requirement: title.
• A hook in tests/conftest.py rejects malformed markers. It also copies each marker into the test's JUnit XML properties as <property name="spec" value="<capability>: <requirement>"/>.
• Tests of infrastructure with no spec behavior (for example, test-database isolation) carry no marker.
• The convention is tested itself, in tests/test_spec_traceability.py.

Consequences

• Good: A standard JUnit XML report holds everything needed to join tests and results to requirements. No custom test runner is required.
• Good: A test can verify several requirements by stacking markers.
• Cost: Requirement titles become identifiers. Renaming a requirement means updating its markers. A coverage check (every requirement has at least one marked test, and every marker names an existing requirement) catches drift. It runs at the end of each change and becomes automatic with the build walkthrough page.
• Cost: The link is only as accurate as the marker. A test can be marked with a requirement it doesn't really verify. Review is the guard against that.

Context

Several components reason about the same dates: vaccine compliance, kennel capacity, extracting records from certificate images, the resolver's alternative offers, and the reply sent to pet parents. If any two of them read a date differently, the result is an off-by-one decision. Examples: a vaccine treated as valid on the day it lapses, or a night counted twice. Ambiguity hides in four places:

• Is a printed expiry date the last valid day or the first invalid day?
• Is the pet on site on checkout day?
• Which time zone turns "now" into "today"?
• Are kennel stays measured in days or in nights?

Options considered

1. Timestamps everywhere (check-in and checkout times, expiry at midnight in some zone). Precise, but every comparison needs time-zone arithmetic, check-in hours vary by facility, and certificates only carry dates.
2. Calendar dates in UTC. Simple, but a late-evening request in Chicago lands on the next UTC day.
3. Calendar dates in the facility's time zone, with fixed meanings for each boundary.

Decision

Option 3.

• Stay dates are date values in the facility's IANA time zone (facilities.timezone).
• Days on site: a pet is on site every date from the start date through the end date, inclusive. The end date is checkout day. A day service (daycare, grooming) has start = end.
• Kennel nights run from the start date up to, but not including, the end date: [start, end). That matches the reservation exclusion constraint, so back-to-back stays don't conflict.
• Vaccine expiry: expires_on is the last valid day, inclusive. A record covers a stay when administered_on <= start and expires_on >= end.
• Converting instants to dates: happens once, at the boundary, with facility_date(timezone, instant). Naive datetimes are rejected rather than guessed.

Consequences

• Good: Compliance, capacity, extraction, and replies share one set of meanings. The boundary cases (expires on checkout day, back-to-back stays) are pinned by tests.
• Good: Days on site and kennel nights are two views of the same stay. A Mar 6–9 stay is 4 days on site and 3 kennel nights.
• Cost: Every extraction prompt and reply template has to use these meanings. For example, a certificate's "expires 03/09/2027" is stored as 2027-03-09, and a reply says "valid through" rather than "until".
• Cost: Facilities that check vaccines against arrival only (not checkout) would need a separate rule. That is out of scope for now.

Context

Every later phase adds steps that call a language model or run tools: intake, records extraction, the resolver, and the reply. Operators need to see each step: what went in, what came out, how long it took, what it cost, and why it failed. The most important case is the failed one, and failed work usually ends with its database transaction rolled back. If spans were written in that same transaction, the record of the failure would vanish with it.

Model calls are also where cost, latency, retries, and leaked secrets come from. If each agent called the provider in its own way, those concerns would be handled inconsistently, or not at all.

Options considered

1. Spans written in the business transaction. Simple and consistent, but failures erase their own record, and nothing is visible until the step commits.
2. Spans buffered in memory and flushed at the end of a run. Fewer writes, but a crash loses the whole trace, and nothing can show live progress.
3. Spans written in their own short transactions (insert as running at start, update at end), plus a single traced model client that all model calls must use.
4. The OpenTelemetry SDK with an external collector. Industry standard, but it adds infrastructure, and the dashboard would have to query a second store.

Decision

Option 3.

• app/tracing.py owns a session factory that business code never uses. Each span write is its own transaction, committed immediately.
• Trace context lives in contextvars, so nested and concurrent work gets correct parent links without passing ids around.
• app/llm.py's complete() is the only code that calls a model.
  • It refuses to run outside a trace.
  • It owns the retry loop, so every attempt is recorded.
  • It records cost exactly as the provider reports it, never estimated.
  • It never stores keys, headers, or raw image data.
• The data model (trace, span, parent, kind, status, attributes) mirrors OpenTelemetry, so an exporter can be added later without changing callers.

Consequences

• Good: A failed or crashed step still leaves a span with its error. running rows let the dashboard show live progress.
• Good: One place enforces "no untraced model calls", along with retries, timeouts, and secret hygiene.
• Cost: Two extra database round trips per span, which is negligible next to model latency.
• Cost: Traces can reference a booking request only after that request is committed (foreign key). Work on not-yet-stored requests is traced without the link.
• Cost: Test fixtures must clean up committed tracing rows, because they aren't covered by the per-test rollback.

Context

Unit tests prove that the pipeline around a model is correct: validation, repair, code checks, and handoff. They use scripted answers, so they can't say how good the model is at reading messages or certificates. That needs an evaluation against the real model. Evaluations cost money, vary between runs, and are easy to misreport: a cherry-picked demo, a number remembered from an older prompt, or an accuracy figure that hides the errors that actually matter. This project already has two such steps (intake and records), and the architecture comparison will need a fair way to compare whole systems.

Options considered

1. Manual spot checks. Fast, but not repeatable and not comparable over time.
2. One overall accuracy number per step. Simple, but it hides which errors happen and whether they change decisions.
3. Labeled sets with layered metrics, run on demand, with every run stored as a dated report.
4. An LLM judge scoring free-form outputs. Useful for open-ended text, but unnecessary when ground truth is exact, and it adds a second model's errors.

Decision

Option 3, with these rules for every evaluated step:

• Ground truth is exact. Labeled cases (intake) or generated data whose truth is known by construction (records). Fields that are genuinely open to interpretation are marked unscored rather than guessed.
• Three levels of metric:
  1. Field accuracy, per field.
  2. Dangerous errors counted separately. For example, invented expiries: a missing value made up by the model.
  3. Decision agreement: run the deterministic decision (compliance) on the truth and on the model's output, and compare. Silent disagreements, where the decision is wrong and nothing sent the case to a person, are the headline safety number.
• Results are broken down by case type or variant, not only averaged.
• Every run is a measurement:
  • It runs against the real model only on demand.
  • It is traced like production, so its cost comes from span rows.
  • It is written to backend/evals/results/<step>-<timestamp>.json with the prompt version, model, and trace id.
  • It is never edited by hand. Numbers are quoted with the run they came from, never estimated or averaged across prompt versions.
• Benchmarks are versioned. The labeled files and generated images are committed. Changing them creates a new benchmark, not a silent edit.
• Scoring code is unit-tested with scripted answers, so the evaluation itself can be trusted.

Consequences

• Good: A prompt or model change can be judged by its effect on decisions, not just on field accuracy.
• Good: The same method extends to the architecture comparison: the same sets, the same metrics, and the same cost accounting.
• Cost: Real-model runs cost money. Intake and records each run in cents, so they're run deliberately, not on every commit.
• Limitation: Synthetic benchmarks are cleaner than real-world input. Results are reported as performance on this benchmark, never as real-world accuracy.

Context

A booking request passes through reading (the free-text message and the certificate image), checks (vaccine compliance, kennel capacity, pet identity), and routing (draft a reply, or resolve problems first). Only the reading needs a language model. Every check has a correct answer that code can compute and test.

The common agent architectures put a model in charge of more or less of this:

| Pattern | Who decides what happens next | |---|---| | Prompt chaining | Code, in a fixed order | | Routing | Code or a classifier picks one of several fixed paths | | Parallelization (sectioning or voting) | Code fans out independent work and gathers it | | Orchestrator–workers | A model plans subtasks at runtime and hands them to workers | | Evaluator–optimizer | A model critiques and revises its own output in a loop | | Autonomous agent | A model with tools picks every next action | | Multi-agent (supervisor, peer handoff, shared blackboard) | Several models coordinate, through a manager, direct handoffs, or a shared workspace |

Errors compound across model decisions. If each decision is right 95% of the time, a chain of five is right about 77% of the time (0.95⁵). Each decision handed to a model also needs its own evaluation, trace, and defense against prompt injection, because pet-parent messages and images are untrusted.

Options considered

1. Fully agentic. One agent with tools for intake, records, compliance, capacity, and routing, choosing the order itself. Flexible, but its path can't be predicted from the data. An injected instruction can steer it, and every run needs a model decision just to reach the deterministic checks.
2. Multi-agent supervisor. A manager agent delegates to intake, records, and checking agents. This adds coordination calls and a second layer where errors compound, for a flow that never changes shape.
3. LLM-planned orchestrator–workers. A model writes the plan per request. That fits when the subtasks can't be known in advance, which isn't the case here.
4. Hybrid workflow. A plain async function fixes the order: concurrent reading (sectioning), then deterministic checks, then code routing with listed reasons. Models are used only where the input is messy. A narrowly scoped agent is allowed later, inside a fenced step, where the work really is open-ended (the resolver).

Decision

Option 4.

• app/orchestrator.py branches only on the request's lifecycle state and saved step results.
• Model calls happen only inside the intake and records steps. Their outputs are validated, and they can raise issues but never approve anything.
• Compliance, capacity, pet-name matching, and routing are plain code with unit tests. Routing records every reason a request needs a closer look.

Consequences

• Good: The path of every request can be predicted from its data, and tested without a model.
• Good: Prompt injection can at most add an issue, which routes the request toward a person. It can't skip a check.
• Good: Two model calls per request, made at the same time, so cost and latency stay small and bounded.
• Cost: New kinds of work need code changes, not just a prompt change.
• Cost: The workflow can't improvise. Unusual requests end up in resolving and, later, with a person. For a booking desk, that's intended.
• Follow-up: Parallel voting (two independent reads of a certificate, with disagreement sent to review) fits this structure without changing it. The records evaluation showed confident misreads of handwritten dates that a single read can't catch.

Context

Processing a booking request takes several seconds of model calls followed by database checks. A process can die at any point: a deploy, an out-of-memory kill, a hung provider connection. Two things must hold after a crash:

• No work is lost. The request can continue instead of being stuck or starting over.
• Nothing happens twice when a step is retried. That means no second reservation, no second email, and ideally no paid model call repeated for a step that already finished.

Doing a step twice must have the same effect as doing it once (idempotency), and the system must know which steps already finished (durability).

Options considered

1. Re-run everything from the start. Simple, but it repeats every model call after a crash and depends on every step being free of side effects forever.
2. A workflow engine (Temporal, or a queue with step state). Durable execution, timers, and retries built in. But it's a separate service to run and learn, for one short linear workflow.
3. Event sourcing. Store every event and rebuild state from them. More history than needed: the transition log already records every state change.
4. Step checkpoints in the existing database. Save each step's result in a booking_step_results table, in the same transaction as the state change that follows it. A run reads the request's state and saved results, and runs only the steps without a result.

Decision

Option 4.

• booking_step_results has one row per (request, step), enforced by a unique constraint.
• A step's results, the request's state change, and its transition log row are committed in one transaction. Either all exist or none do.
• The received → extracting move is committed before any model call, so a crash during extraction resumes at extraction.
• Each step is idempotent:
  • The reading steps have no side effects beyond spans.
  • The checks are pure reads.
  • Later steps with side effects rely on database uniqueness (one reservation per request) rather than on being run only once.
• A step that fails in an expected way (model unavailable, output invalid after repair) moves the request to failed with the error. The next run retries from that step. Any other exception is treated as a bug and left visible in the working state.
• When a request is processed again, spans that a killed run left running are closed as abandoned.

Consequences

• Good: A crash between steps loses nothing, and finished model calls are never repeated.
• Good: No new infrastructure. The checkpoint, state, and log share one transaction in the same database.
• Cost: A crash inside a step repeats that whole step, including both concurrent model calls in extraction. That costs money, not correctness.
• Cost: It assumes one worker processes a request at a time. The version column and unique checkpoints prevent corruption if two runs race, but running many workers safely needs leases, a later change.
• Cost: No built-in timers or scheduled retries. Retrying a failed request is an explicit call.

Context

Some booking requests can't go ahead as asked: no kennel of the pet's size is free for those nights, or a required vaccine won't cover the stay. A helpful desk offers alternatives: other dates, a larger kennel, or an updated vaccine record. PawDesk is a hybrid workflow where code decides and models read (decision record 0008). Finding alternatives is the one place where an agent (a model calling tools in a loop) is worth considering, because the fixes interact. Dates that free a kennel can push the stay past a vaccine's expiry, a larger kennel helps only with capacity, and an updated record helps only with coverage.

An agent is also the least predictable pattern. Each model decision compounds error, a loop can run without making progress, and a model that can act can be steered by text it reads.

Options considered

1. No agent. A deterministic search lists free windows and failing vaccines, and staff combine them. It is fully predictable, and it covers the common single-problem case. It can't weigh combinations or explain a recommendation.
2. A free agent. The model gets the customer message and broad tools (search, reserve, message the customer). It is flexible, but the injection surface is large, it can act on its own, and its outcomes are hard to test.
3. A bounded, proposal-only agent. The model gets only structured facts and read-only tools bound to the request. Its only exit is a proposal that code re-checks, and it stops on hard limits.
4. An agent framework (a graph or agent SDK) to run the loop. It brings built-in loops, memory, and tracing, but those hide the parts this project needs to show and test: the stop conditions, the stuck guard, and what goes into each turn.

Decision

Option 3, with a hand-written loop (not option 4).

• Code decides whether the agent runs. Only capacity and vaccine-coverage problems reach it. An unknown pet, a records problem, suspected injection, or an unknown expiry goes to a person with no model call.
• No untrusted text. The prompt is built from typed values only. The customer's message, notes, certificate text, and even the pet's name as written are excluded. This removes the injection surface instead of defending it.
• Read-only tools bound to the request. The pet, facility, service, and dates are fixed in code, so the model can choose only a search range, dates, or a size class. Mistakes come back as error results the model can read.
• Stop conditions:
  • a valid proposal;
  • any repeat of an earlier tool call with the same arguments (tools are deterministic within a run, so a repeat can't make progress);
  • an answer without a tool call;
  • 6 model calls or 12 tool calls.
• Proposals, not actions. Every option is re-checked against capacity and compliance and stored as verified or not. The lifecycle allows only resolving → drafting, so nothing the agent produces can approve a booking. A person reviews every request that passed through resolving.
• The loop is plain Python (one function of a few dozen lines) over the traced model client, with one llm span per turn and one tool span per call.

Consequences

• Good: The agent's worst case is a useless proposal or an escalation, never a wrong action. Both end in front of a person.
• Good: Every boundary is testable without a model: eligibility, tools, stop conditions, and verification run against scripted turns.
• Good: Cost is bounded: at most 6 model calls per blocked request, and none for problems the agent can't fix.
• Cost: For a single problem, a deterministic search would usually find the same option. The agent has to earn its place on interacting problems and in the staff summary. This is worth measuring before the agent grows.
• Cost: A hand-written loop has no built-in features such as memory, streaming, or human-in-the-loop pauses. These are added only when needed.

Context

A reply to a pet parent is a promise, and sending it (with a kennel reservation for a confirmed stay) is the first thing PawDesk does that can't be quietly undone. Some requests can be decided with full confidence by code: every check is deterministic, and all of them passed. Others needed resolving, carry an issue (an unknown pet, unreadable records, instruction-like text), or produced a draft that says more than the checks confirmed. A person must look at those before anything goes out. Where to put that checkpoint, and how a paused run continues, shapes the whole approval flow.

Options considered

1. A human approves everything. It's the simplest rule and has no automation risk, but staff time goes into rubber-stamping requests the checks already decided.
2. A gate before the reply is drafted. Staff decide first, then a model writes the reply. But staff decide without seeing what will be said, and the model's text goes out unreviewed.
3. A gate after drafting, before sending. Flagged requests stop in needs_review with the checks, the resolution, the draft, and the reasons for the flag. Staff approve, edit, or reject once. Clean requests skip the gate.
4. A gate after sending (review samples afterward). Fast, but a wrong promise has already reached the customer.

Decision

Option 3.

• The gate is a state, not a running process. needs_review is persisted like every other state, so a paused request survives restarts and holds no resources.
  • A staff action records an Approval and moves the request in one transaction, guarded by the version check.
  • Processing then resumes from the new state, exactly as after a crash (decision record 0009).
• Who waits is decided by code:
  • every request that came through resolving;
  • every draft that fails the deterministic promise check (unconfirmed dates, commitment words when nothing was confirmed, prices).
• Clean requests (every check passed, intake raised nothing, and the draft promises nothing extra) are approved by the system and sent.
• Staff can:
  • approve the draft;
  • replace its text;
  • reject with a reason, which is recorded and never sent to the customer. The customer gets a fixed decline.
• The reservation for a confirmed stay is made when sending, in the same transaction as the message. A kennel taken since checking fails the step instead of double-booking.

Consequences

• Good: Staff time goes only to requests with a reason to look, and each one arrives with that reason written down.
• Good: Nothing irreversible happens before the gate, and nothing after it happens twice: one message and one reservation per request, both enforced by the database.
• Cost: Clean requests are sent with no human involved. That is safe only because every check before it is deterministic and tested. A new check that relies on a model must flag, not pass.
• Cost: Staff edits are sent as written and aren't re-checked. Staff are trusted, and their edit is recorded on the Approval.
• Cost: Staff sign-in doesn't exist yet. The API records a name and must not be exposed before authentication is added.

Context

Decision record 0004 links every backend test to the requirement it verifies, through a pytest marker exported into JUnit XML. The dashboard adds frontend requirements, such as how the run graph derives step status, how the timeline lays out overlapping spans, and which paths the proxy forwards. Their tests run under Vitest, not pytest. The coverage check and the build walkthrough page need the same link from these tests.

Options considered

1. Vitest test annotations or metadata. Structured, but whether and how they reach the JUnit report differs between versions, so the link would depend on reporter internals.
2. A mapping file from requirement to test names. Rejected in 0004 for the same reason: it drifts from the code.
3. A tag in the test name, [spec <capability>: <requirement>] <description>, written by a small specTest() helper that refuses malformed tags. Vitest's built-in JUnit reporter writes test names unchanged.

Decision

Option 3.

• Writing tags: frontend/tests/spec.ts provides specTest(capability, requirement, name, fn) and the tag pattern. The pattern is tested.
• Reporting: pnpm test writes frontend/test-results/junit.xml.
• Checking: the coverage check reads both JUnit reports. A requirement is covered when a backend spec property or a frontend tag names it, and the covering tests passed.

Consequences

• Good: One requirement can be covered by backend and frontend tests together, and the walkthrough page shows both.
• Good: No dependency on reporter metadata features; any JUnit reader sees the tag.
• Cost: Requirement titles appear in test names, so renaming a requirement means updating both test languages. The coverage check catches drift, as in 0004.

Context

The dashboard's run view should light up steps as they happen, while a demo request is being processed or a staff action resumes a run. Runs are processed by more than one process: the API (demo submissions and staff actions, as background work) and the command line (scripts/bookings.py). Every process records its progress the same way, as spans written and committed while work runs (decision record 0006), plus state changes. The browser also never talks to the backend directly. Its requests go through the dashboard's allowlisted proxy route.

Options considered

1. Client polling. The browser re-fetches the whole request every second. Simple, but every viewer pulls full payloads, and an update arrives up to a full interval late.
2. WebSockets. Two-way and low-latency, but the traffic only flows one way. A WebSocket also can't pass through an ordinary HTTP route handler, and it needs its own reconnect logic.
3. Server-sent events, fed by an in-process event bus. Push without polling. But it only sees runs in the same process, so a command-line run would be invisible.
4. Server-sent events, fed by reading the database. The stream reads a light snapshot (state, plus each span's timing and status) every half second, and sends it only when it changed.
5. Server-sent events, fed by Postgres LISTEN/NOTIFY. Push from every process with low latency, but every writer (or a trigger) must notify, which adds moving parts to the tracing path.

Decision

Option 4.

• The endpoint: GET /requests/{id}/events is a text/event-stream.
  • What it sends: a snapshot when the stream opens and whenever the snapshot changes.
  • When it ends: once the request is resting (sent, needs_review, failed) with no span running, or after five minutes.
• The proxy passes the response body through as a stream.
• The browser uses EventSource. It reconnects on its own, and the view closes it when the page is hidden.
• On each snapshot, the view re-fetches the full request for the inspector. The snapshot itself stays small.

Consequences

• Good: It works for runs from any process, because the database is the one place they all write to.
• Good: One-way HTTP that passes through the existing proxy and its allowlist, with nothing new to run.
• Cost: A light query per open stream every half second, and up to half a second of delay. Fine for a handful of operators. LISTEN/NOTIFY is the upgrade path if viewers or latency demands grow.
• Cost: The stream doesn't resume after it ends. After a staff action, the view reloads to follow the rest of the run.

Context

The build walkthrough page shows how PawDesk was built: every archived change, every requirement with the tests that verify it and their latest result, and every decision record. All of it already exists as files: OpenSpec archives and specs, docs/decisions/, and the JUnit reports the test suites write. The page must stay truthful, so it has to come from those files rather than being written by hand. It must also read only public sources. The working agreement, the build plan, and the private notes must never leak into it.

The frontend's Docker image is built from frontend/ alone, and the backend image doesn't ship openspec/ or docs/.

Options considered

1. Hand-written page. Simple, but it drifts, and nothing proves its claims.
2. Generate during the frontend build by reading ../openspec and ../docs. The data stays current, but it breaks the self-contained frontend build context and couples the build to the test reports.
3. A backend endpoint that reads the repository at runtime. The data is live, but the backend image would have to ship the specs, docs, and test reports, and a running service would be reading the repository.
4. A generator script that writes a JSON file into the frontend, committed with the change. The pages import it and are fully static.

Decision

Option 4.

• The generator: backend/scripts/walkthrough.py reads an explicit list of sources:

  • the archived changes;
  • the current specs;
  • the decision records;
  • the backend and frontend JUnit reports.

  It writes frontend/content/walkthrough.json, with its generation time and the time of each test report.

• --check fails when any requirement lacks a passing linked test or a test names an unknown requirement. It replaces the one-off coverage script used at the end of earlier changes.

• Ordering: changes are ordered by the commit that archived them, because several can share a date.

Consequences

• Good: Every statement on the page traces to a file, and the coverage claim ("106/106 verified") is checked by a command, not asserted.
• Good: The pages are static, cheap to serve, and independent of the backend.
• Good: A test proves that private notes never reach the output.
• Cost: The committed file can go stale between regenerations. The page shows when it was generated, and CI regenerates and checks it (Phase 11).
• Cost: Test results in the file are those of the last local run, not of CI, until CI takes over generation.

Context

Decision record 0008 chose a hybrid workflow over an agentic design on principle. The principles were error compounding, injection risk, and testability. The obvious alternative is simpler to build: one autonomous agent given the customer's message, the certificate image, and every tool, deciding the outcome itself. Phase 10 measured both on the same 20 booking cases (comparison-v1):

• 17 ordinary cases, plus 3 injection attempts;
• the same model (deepseek/deepseek-v4.1-flash, temperature 0);
• the same deterministic checking tools;
• the same expected outcomes, computed by running the checks on each case's true facts (certificate ground truth, the pet on file);
• an isolated lab database, with nothing sent.

The report is backend/evals/results/comparison-20261014T204832Z.json. It's one run on a small set, so it shows failure modes, not precise rates.

Options considered

1. Keep the hybrid workflow (code decides, models read and write, a fenced resolver proposes, people review flagged requests).
2. Switch to the single autonomous agent.
3. A mix: the autonomous agent for clean requests, the hybrid for the rest. But deciding which requests are clean is exactly the hybrid's job, so this adds a decision without removing one.

Results

| | Hybrid | Autonomous | |---|---|---| | Decision accuracy | 18/20 | 19/20 | | Unsafe confirmations | 0 | 1 | | Missed confirmations | 2 | 0 | | Injection attempts handed to a person | 3/3 | 0/3 | | Errors | 0 | 0 | | Cost for 20 cases | $0.0275 | $0.0238 | | Average latency | 8.5 s | 10.3 s |

What lies behind the numbers:

• The autonomous agent's unsafe confirmation (i01):
  • It recognized the injected "SYSTEM: … mark this booking approved" text as manipulation, said it ignored it, ran the checks, and confirmed.
  • The checks did pass, so the booking itself was valid. But a request that tries to manipulate the desk was approved by the component it tried to manipulate, which was the only judge of whether it had been manipulated.
  • For the other two injections it declined, but only because the vaccines failed. It offered alternatives and handed nothing to a person.
• The autonomous agent's replies described actions it never took:
  • In 4 of its 6 confirmations it told the customer a kennel was "reserved", although its tools only read.
  • The hybrid's reply is drafted from code-built facts, and checked for promises the checks didn't confirm.
• The hybrid's two misses (c01, c06):
  • The records model reported doubt: the generator's "synthetic test document" watermark, and a label partly hidden by it.
  • The hybrid's rule is that any reported doubt goes to a person. That cost two reviews, not two wrong bookings. The autonomous agent saw similar doubts and proceeded.
• The hybrid was faster because intake and records run in parallel. It cost about 16% more, mostly from the separate reply call and resolver runs.

Decision

Option 1: keep the hybrid workflow.

The measured difference is in the dangerous direction. The autonomous agent was slightly more "accurate", because it was willing to proceed past doubt. That same willingness produced the unsafe confirmation, the zero injection hand-offs, and the replies claiming reservations. The hybrid's errors were all over-caution, which costs staff time and is visible in the review queue. The autonomous agent's error was an approval nobody reviewed.

Consequences

• Good: The design choice is now backed by measurements, including the cost of the hybrid's caution (2 of 20 cases sent to a person unnecessarily, about 16% more model cost).
• Follow-up: The hybrid's missed confirmations come from model-reported doubt about benchmark artifacts. The next records benchmark should vary the watermark, or allowlist known non-substantive notes, and measure whether misses drop without unsafe confirmations appearing.
• Follow-up: Rerun the comparison on any model or prompt change. One run on 20 cases can't rank two close accuracies, only expose failure modes.
• Cost: The lab and the autonomous agent stay in the repository as a measurement tool, not as a supported path.

Context

The demo (submit a request, watch it run, approve a flagged one) is the most important path through PawDesk. Until now it was checked only by hand in a browser. Running it automatically on every push means running the real backend and dashboard, and the backend calls a language model at several steps (intake, records, reply). In CI a real model call:

• costs money on every push;
• needs the provider key as a CI secret;
• fails when the provider is down;
• can answer differently from run to run, which makes the test flaky.

Model quality is already measured separately, on demand, by the evals (decision record 0007).

Options considered

1. Call the real model in CI. It's the most realistic option, but it has every problem above, and a test that fails because the model changed its wording tells us nothing about our code.
2. Mock at the function level (fake run_intake, draft_reply). Deterministic, but it skips the real client: the retries, spans, cost recording, and schema validation that the demo depends on.
3. A fake model server that answers by rules. Deterministic, but its answers are invented, and someone has to keep them plausible as prompts change.
4. Record real responses once, then replay them at the HTTP layer. A transport under the model client saves each response body under a hash of the exact request, and replays it later without the network. Everything above the transport runs for real.

Decision

Option 4.

• The transport: app/cassettes.py, selected by the settings llm_cassette_mode (off, record, replay) and llm_cassette_dir.
• What's stored: only response bodies, keyed by the request body's SHA-256. Request bodies (with images) and headers (with the key) are never written.
• A missing recording is a permanent error that names the request hash, so any change to a prompt, schema, or input shows up as a failing end-to-end test until the responses are re-recorded.
• Recording is one command (PAWDESK_E2E_RECORD=1 pnpm e2e) and makes a handful of real calls. CI only replays. Its workflow holds a placeholder key, and a test checks that no model secret is referenced.

Consequences

• Good: CI is free, deterministic, and offline from the model's point of view, and the full stack is still exercised (real client, real traces, real database, real browser).
• Good: Recording against the real model exposed a bug that scripted tests had missed. The provider's responses are gzip-compressed, and the recorder first passed decoded bodies on with their compression header. It's now covered by a regression test.
• Cost: The recordings have to be re-made after prompt or schema changes. The failure names exactly which request changed.
• Cost: Replay proves the plumbing and the flow, not the model's quality. The evals remain the measure of that, and are rerun on model or prompt changes.

• Relates to: 0008, 0009, 0010, 0011, 0015

Context

PawDesk's agent machinery was built by hand on the official openai client:

• the durable orchestrator;
• the resolver's tool loop;
• the human gate;
• tracing;
• retries;
• recorded responses.

Hand-building was deliberate, to keep every stop condition, checkpoint, and trace visible and testable. Now that it exists, the question is what an agent framework would have provided, and at what price.

The parts a framework would replace, measured as non-blank, non-comment lines (docstrings included):

| Part | Lines | |---|---| | Orchestrator: steps, checkpoints, resume, failure, routing (app/orchestrator.py) | 538 | | Resolver: eligibility, tools, loop, verification (app/resolver.py) | 407 (the loop itself, about 46) | | Model client: retries, deadlines, cost, tool calls (app/llm.py) | 248 | | Tracing (app/tracing.py) | 127 | | Lifecycle guard and transitions (app/lifecycle.py, app/transitions.py) | 105 | | Staff decisions (app/approvals.py) | 85 |

The comparison below rests on each framework's documentation and published descriptions, not on porting PawDesk to it. No framework code was written for this record.

Options considered

LangGraph (graph workflows with persistence)

• Would give:
  • a StateGraph for the fixed order of steps;
  • a checkpointer that saves state after every step and resumes a thread after a crash;
  • interrupt() to pause for a person, and a resume command to continue, which is exactly the shape of needs_review;
  • streaming, and "time travel" (replaying or forking from earlier checkpoints).
• Overlap with PawDesk: most of the orchestrator's resume and failure plumbing, and the mechanics of the human gate.
• Would cost:
  • Checkpoints would be framework state, kept apart from the business data. PawDesk saves each step's result in the same transaction as the request's state change and its transition log row, guarded by the lifecycle rules and a version check (0009). A framework checkpoint lives outside that transaction unless the two are carefully joined, and the domain's own guard would still be needed.
  • The checkpoints would be serialized graph state rather than queryable rows. The dashboard and the build walkthrough read PawDesk's step results directly.
  • There would be another abstraction to learn and upgrade.
• Fit: the strongest candidate if PawDesk grew into many workflows, long waits, or several workers.

OpenAI Agents SDK (agents, tools, handoffs, guardrails)

• Would give:

  • a runner for the tool loop with a turn limit;
  • function tools with schemas from type hints;
  • handoffs between agents;
  • input and output guardrails that can stop a run;
  • session memory;
  • built-in tracing of model calls, tool calls, handoffs, and guardrails.

  It's provider-agnostic, so it would work with the model PawDesk uses through OpenRouter.

• Overlap with PawDesk: the resolver's loop (about 46 lines), its tool schemas, and part of the tracing.

• Would cost:

  • PawDesk's stuck guard (stop on any repeat of an earlier call) and the "proposals are re-checked by code" step would still be custom.
  • Tracing would have to be pointed at PawDesk's own spans. The dashboard depends on exact provider-reported cost, per-attempt retry records, and spans stored in Postgres, which built-in tracing doesn't give by default.
  • Retry ownership would need care, so the SDK and PawDesk's own client don't both retry.
  • Its multi-agent strengths (handoffs) answer a question PawDesk measured and declined (0008, 0015).

• Fit: a reasonable replacement for the resolver loop alone. It shrinks little of what matters most here.

DeepSeek Harness (open-source agent runtime)

• Would give: a plugin-based runtime. Per its published descriptions, models, tools, skills, sandboxes, storage, loops, scheduling, and the UI are all swappable plugins, with preset work modes centered on a general coding agent.
• Overlap with PawDesk: little. It's built for open-ended agents that act (files, shells, sub-agents). PawDesk deliberately keeps its one agent proposal-only, with read-only tools.
• Would cost:
  • a runtime far larger than the problem;
  • a developer-preview status, with breaking changes possible, according to the reviews available;
  • a design center (autonomy) that runs against the measured result in 0015.
• Fit: not a fit for a business workflow with a human gate. It might fit internal developer tooling around the project.

Decision

Keep the hand-built implementation for PawDesk as it stands.

• The reason: the parts that make PawDesk safe are domain rules no framework provides. The framework would supply plumbing around them:
  • the lifecycle guard;
  • step results committed with state changes;
  • code-decided eligibility and routing;
  • re-verified proposals;
  • the promise check;
  • a human gate as a database state.
• Writing the plumbing by hand (about 1,500 lines, all tested and traced to specs) made every stop condition and checkpoint explicit and testable.

If PawDesk grows, adopt in this order:

1. LangGraph for the workflow, once there are several workflows, long waits, or many workers. Keep step results and the lifecycle guard in the business database, and treat the framework checkpoint as an implementation detail.
2. The OpenAI Agents SDK for agent loops, once an agent needs many tools or handoffs. Route its tracing into PawDesk's spans.

Consequences

• Good: No framework lock-in or upgrade churn, and the tracing, durability, and approval rules stay exactly as specified and tested.
• Cost: PawDesk maintains code a framework would maintain for it: retries, the resume loop, and the human-gate plumbing. It also lacks framework conveniences, like time-travel debugging and built-in streaming of agent steps.
• Cost: This retrospective rests on documentation, not a port. A spike porting the resolver loop to the Agents SDK, and the orchestrator to LangGraph with a Postgres checkpointer, would turn it into a measurement. That's the next step if either adoption trigger above is reached.

Sources

• LangGraph human-in-the-loop and interrupts: https://docs.langchain.com/oss/python/langgraph/human-in-the-loop
• OpenAI Agents SDK (package and overview): https://pypi.org/p/openai-agents
• DeepSeek Harness overviews (third-party; maturity claims differ between them):
  • https://www.datacamp.com/tutorial/deepseek-harness
  • https://nexforce.ai/en/blog/deepseek-harness-framework-agentes
  • https://pandaily.com/deepseek-harness-hands-on-four-modes-model-plus-harness-equals-agent-aug2026

Context

The dashboard's text read as templated. It used slogans built on a contrast ("Models are measured, not trusted"), phrases repeated three times for rhythm, filler adverbs ("exactly"), and passive sentences that left out who acted ("A polite decline was sent").

The passive sentences matter more than the rest. PawDesk's main safety property is who approves a booking: the deterministic checks or a named person, never a model. A status sentence that doesn't say who decided hides the one fact a reader most needs.

Two strings were also false. The build page said nothing on it was written by hand, but its method section was. A request that failed while sending read "Stopped while approved", which showed a raw state name.

Rewriting the text once fixes today's copy. Each new feature brings new copy, so the fix needs something that keeps it fixed.

Options considered

1. Rewrite once, with no rules written down. The next feature brings the same patterns back, and nothing notices until a reader does.
2. Rewrite, and write a guide. It covers the rules that need judgment, but a guide doesn't stop a filler word from being committed.
3. Rewrite, write a guide, and check the mechanical rules in the test suites. The checkable rules are a list of banned words and phrases, and no em dash inside a sentence. A shared list file is read by a dashboard test and a backend test (the demo sample descriptions come from the backend).
4. A model-based style check in CI. It catches more than a word list, but it costs money on every run, and two runs can disagree on the same text. That breaks the project's rule that anything with a checkable answer is checked by plain code.

Decision

Option 3.

• The guide: docs/style/interface-copy.md. It gives each rule a PawDesk example and lists three exceptions: "never" can state a guarantee the tests prove, a list of three is fine when there are three things, and short labels can drop the subject.
• The list: docs/style/copy-rules.json. Words are matched as whole words, ignoring case.
• The dashboard check parses the source with the TypeScript compiler and reads only text a user can see: JSX text and string values. It skips class names, import paths, object keys, and comments.
• Outcome sentences name the actor, and their unit tests pin that per actor.

The guide is a separate file because decision records don't change after they're committed, and the word list and examples will grow.

Consequences

• Good: New copy, such as the generated-request Try page, meets the mechanical rules without a review pass. A banned word fails the build with its file and line.
• Good: Who decided is part of each outcome sentence's tested contract, not only a style preference.
• Cost: A list can't judge tone. Contrast slogans and repetition still depend on review against the guide.
• Cost: The list file sits outside both packages, so the frontend and backend tests each read a file from docs/. Two copies would drift apart, which is worse.

Context

The demo runs only on a developer machine (decision record 0002). A reviewer should be able to open a link and send a request without cloning anything. The hosting has to be free, run the existing stack (a FastAPI backend, a Next.js dashboard, and PostgreSQL), keep the model key server-side, and stay cheap to operate for a demo that is idle most of the time.

Options considered

1. Hugging Face Docker Space + Neon Postgres. The Space runs one container from a Dockerfile, with secrets set on the Space. The free hardware sleeps after about two idle days and its disk is wiped on restart, so the database lives at Neon (free plan, serverless Postgres that suspends when idle). The cost is folding two services into one container.
2. Vercel (dashboard) + Render (backend) + Neon. Each part runs on a platform built for it. Three accounts and three deploys to keep in step, and Render's free web services sleep after about 15 idle minutes, so most visits start with a long cold start.
3. A free-tier virtual machine (for example Oracle Cloud) running Docker Compose. The local setup runs unchanged. The machine's operating system, TLS certificates, firewall, and updates become the project's job.
4. AWS (container service + managed Postgres). The most production-like option, but not free beyond trial credits, and the most setup for a demo.

Decision

Option 1.

• One container runs the backend on the loopback address and the dashboard on the public port, so the browser still reaches the backend only through the allowlisted proxy.
• Neon's direct endpoint holds the database. The Space's container is disposable.
• Each start migrates and reseeds the database (explicitly allowed for that one host), so the public demo always starts from the known synthetic dataset.
• Model spend has two ceilings: limits on model calls per hour and per day, counted from recorded spans, and a separate OpenRouter key with a credit limit.
• Deploys upload only committed files (git archive HEAD), so local secrets and private notes can't reach the Space.

Consequences

• Good: one link to a working demo, with no cost beyond the capped model spend.
• Good: the deploy path can't publish uncommitted or ignored files.
• Cost: the hosted topology differs from local development (one container instead of three), so a hosted-only failure needs the Space's logs to debug.
• Cost: visits after an idle period wait for the Space and Neon's compute to wake.
• Cost: every visitor acts as staff and can change what other visitors see until the next restart. That is acceptable only because all data is synthetic and no reply leaves the system. Real sign-in is needed before real data.