Evidenced

A QA tool for agents. It checks a user outcome in a real browser and returns a passed, failed or inconclusive verdict with evidence the agent can loop on.

Status: In progress
Role: Full Stack Developer
Team: Solo

The problem

Coding agents can write code but cannot reliably tell whether the result works for a user.

Who it is for

Coding agents, through an MCP server, and the people who review their work, through a dashboard.

What I built

A QA primitive. It drives a real browser toward a goal and returns a structured verdict (passed, failed, or inconclusive) with evidence, instead of a raw action log the agent has to interpret. The MCP server and the dashboard share one execution path, so both produce the same verdict. It is open-core and self-hostable under Apache-2.0. This project is in progress.

Built with

Frontend: Next.js 16
React 19
TanStack Query
Tailwind CSS
next-themes
Backend: Fastify 5
Node.js 22
Better Auth
Zod
Engine: Puppeteer
OpenRouter
MCP (stdio)
Data & Queue: PostgreSQL
Drizzle
Redis
BullMQ
Infra: Turborepo
pnpm
Docker
S3 / R2
Billing: Dodo Payments

In detail

Verdict states: 3
Deterministic guards: 5
Stage engine pipeline: 4
Entry points, one path: 2

How it works

Trigger a run
An agent (evidenced goal:"...") or a human (dashboard "Run") hits the backend, the single front door. It authenticates the caller, enforces tenancy and the app's URL allowlist, gates on credits (hosted only), decrypts the run's secrets, and enqueues a validated job on BullMQ.
Execute in isolation
The worker pulls the job, re-checks the allowlist, launches an isolated browser, and runs the engine. The only difference between an MCP run and a dashboard run is the recorded source.
Plan, execute, judge, assemble
The engine turns the goal and starting URL into an ordered journey, drives each step in the browser while capturing evidence, runs five deterministic guards, then consults the LLM only if no guard fired.
Return a verdict, loop
The verdict and evidence references land in Postgres and the object store. The agent reads the verdict and loops (fix, re-run); the human reviews it in the dashboard as the lifecycle streams back.

The engine is a pure pipeline with no transport, storage, or queue concerns; everything external is supplied by the worker as an adapter:

packages/core
  planner/    goal + URL -> ordered journey (one model call)
  executor/   step execution, intent resolution, evidence capture
  judge/      guards.ts (the five guards) + verdict.ts (LLM judge)
  evidence/   evidence collector
  assemble.ts verdict assembly + schema validation
  adapters/   interfaces (types.ts) + concrete impls per subdirectory

Key features

Outcome-level verdicts

A verdict is relative to a goal, so the caller never has to parse a transcript. failed means the app is wrong; inconclusive means the environment (unreachable target, timeout) prevented a verdict. An agent loops only on real defects.

Deterministic guards first

Five guards (HTTP status, URL assertion, element presence, console error, page load) run on every step before any LLM is consulted. A guard failure is conclusive and short-circuits the model, which can never override it.

Machine-readable failures

Every failed verdict carries a dossier: the failing step, the reason, a root-cause hypothesis, evidence references, and a suggested next action, so an agent has a plan to act on.

Context economy

Evidence (screenshots, DOM snapshots, console, network) is stored and returned by reference. Agents receive compact verdicts instead of megabytes of artifacts.

One execution path

MCP and dashboard runs share the same job, queue, worker, and verdict. No functional gap between hosted and self-hosted; only the commercial layer differs.

Sealed secrets

Model keys and app credentials are sealed with a per-org data key wrapped by a boot-time KEK, decrypted only at enqueue, carried through Redis, and scrubbed from every error before it reaches a client.

Allowlist cannot be widened

The target URL is checked at enqueue in the backend and re-checked in the worker before navigation. An agent has no path to expand its own reach.

Pluggable adapters

Browser (Puppeteer), model (OpenRouter), resolution (DOM ladder), and storage (disk / S3 / R2) are all adapters behind interfaces. Swapping one is a config change in the composition root, not an engine change.

Architecture

A pnpm and Turborepo monorepo (Node 22, TypeScript, ESM throughout): five apps and four packages behind a single backend front door.

apps/backend     Fastify API: auth, tenancy, allowlist, secrets, enqueue, reads
apps/worker      BullMQ consumer: executes runs, persists verdicts + evidence
apps/mcp         Thin stdio MCP server an agent runs locally
apps/dashboard   Next.js app: runs, apps, keys, credentials, billing
apps/web         Public Next.js marketing + auth + SEO surface

packages/contracts  Every cross-boundary shape (verdict, tool I/O, jobs). Pure Zod.
packages/core       The engine: planner, executor, judge + guards, adapters
packages/db         Postgres schema (Drizzle), org-scoped DAL, sealed secrets
ee/                 Commercial hosted tier: billing, metering, gating

Decision

Open-core, runtime switch

The commercial ee/ package is an optional dependency reached only through a guarded dynamic import, gated by BILLING_ENABLED / REQUIRE_BILLING. The switch is runtime, not compile-time, so the OSS build and the hosted build run the identical verdict loop; with billing off the seams resolve to no-ops and runs are unlimited.

Tradeoff

failed is not inconclusive

An unreachable target (DNS failure, connection refused, timeout) yields inconclusive, not failed: the environment prevented a verdict, the app was never proven wrong. Collapsing these two would make an agent chase phantom defects every time the network hiccuped.

Note

Purity keeps SDKs off consumers

packages/core exports the engine and adapter interfaces only. Concrete adapters live behind subpath exports (@evidenced/core/adapters/model/openrouter, etc.), so importing the package for its types never drags Puppeteer or the model SDK into a consumer that does not touch them.

Outcome

Evidenced is an open-core agent-QA system: a pure four-stage browser verification engine, five deterministic guards gating the LLM, a single execution path shared by an MCP server and a human dashboard, per-org envelope-encrypted secrets, and server-side tenant isolation. The agent receives a compact, schema-validated verdict with an evidence-backed failure dossier it can loop on.

Challenges

Distinguishing failed (the app is wrong) from inconclusive (the environment prevented a verdict) so an agent loops on real defects, not on DNS failures or timeouts
Five deterministic guards that rule before any LLM judgment, with a guard failure short-circuiting the model and the LLM never allowed to override a guard
One hosted execution path shared by the MCP server and the dashboard, so both produce the same verdict from the same job, queue, worker, and code
Envelope-encrypted per-org secrets decrypted only at enqueue, carried through Redis on the job, scrubbed from errors, and never logged or written into evidence
URL allowlist checked at enqueue and re-checked in the worker before navigation, so an agent cannot widen its own target
Open-core split where the commercial ee/ tier is reached only through a guarded dynamic import, keeping the OSS verdict loop functionally identical and unmetered

What I learned

Returning evidence (screenshots, DOM snapshots, console, network) by reference keeps verdicts compact so agents get a decision, not megabytes of artifacts
A pure plan to execute to judge to assemble engine with no transport, storage, or queue imports, everything external supplied as an adapter
Subpath exports keep concrete adapters (Puppeteer, OpenRouter, S3) out of the barrel so a types-only consumer never drags in an SDK
Tenant isolation by construction: every row carries org_id, all access goes through forOrg(orgId), and orgId is resolved server-side, never from the request body
Verdict schema validated on write, on read, and on receipt, with an additive-only schemaVersion
Runtime open-core switch (BILLING_ENABLED / REQUIRE_BILLING) rather than a compile-time fork, so the seams resolve to no-ops when billing is off

Have a role in mind?

Send me a message on X with the role and a link to the posting. My resume has the rest.

Send me a message Book a call

Building a product instead? See how I work with founders.