Manual Test Reports — Agent & Founder Runbook

What this system is

The test-reports system lives at https://churchwiseai.com/test-reports?token=cw-testers-2026.

Who uses it: John Moelker (founder) and Lexa Moelker record manual QA observations here. When they hit a bug or a broken flow, they file a report with a screenshot, tester name, property, and details.
How it flows downstream: A Claude agent picks up open reports, diagnoses the root cause, implements a fix on a feature branch, and marks the report as fixed. John or Lexa then verify in production and close it.
Key tables: manual_test_reports, manual_test_comments
As of 2026-05-12: ~49 open reports awaiting agent attention.

Listing open reports an agent should work

HTTP (preferred for agent server-side calls — token in header, not URL)

curl -s "https://churchwiseai.com/api/test-reports?status=reported,reopened&limit=50" \
  -H "x-test-token: cw-testers-2026" | jq '.data[] | {id, test_scenario, property, priority, details}'

HTTP (query-param form — fine for testing, avoid in server logs)

GET https://churchwiseai.com/api/test-reports?status=reported,reopened&token=cw-testers-2026

Useful filters

Filter	Example	Notes
`status`	`reported,reopened`	Comma-separated; these are the two states that need agent work
`priority`	`critical`	Filter by priority: `low`, `medium`, `high`, `critical`
`property`	`churchwiseai-web`	Scope to one product
`category`	`billing`	Filter by test category
`page` + `limit`	`page=2&limit=25`	Default: page 1, 50 per page

Status lifecycle

reported
  └─► agent_reviewing      ← agent claims the report
        └─► fix_proposed   ← optional: agent has a fix plan but not yet applied
              └─► fix_applied   ← PR is open/merged, fix deployed
                    └─► human_verified  ← John or Lexa confirmed it's fixed in prod
                          └─► closed   ← archive
                    └─► reopened        ← human says fix didn't work → back to work
wont_fix                               ← working as intended / out of scope

Transitions an agent performs:

reported → agent_reviewing (with agent_notes = diagnosis)
agent_reviewing → fix_applied (with agent_fix_details = PR link + file:line)

Transitions a human performs:

fix_applied → human_verified (via "Approve Fix" button on test-reports page)
fix_applied → reopened (via "Reopen" button — fix didn't work)
human_verified → closed (explicit close, or auto-closed by cron)

How an agent works a report — step by step

1. Claim the report

curl -s -X PATCH \
  "https://churchwiseai.com/api/test-reports/{REPORT_ID}?token=cw-testers-2026" \
  -H "Content-Type: application/json" \
  -d '{
    "status": "agent_reviewing",
    "agent_notes": "Root cause: the billing tab shows an empty state because the Stripe customer ID is null for legacy demo churches. The query at src/app/api/admin/billing/route.ts:42 fails silently when stripe_customer_id is null."
  }'

Using x-test-token header (preferred for server-side calls):

curl -s -X PATCH \
  "https://churchwiseai.com/api/test-reports/{REPORT_ID}" \
  -H "x-test-token: cw-testers-2026" \
  -H "Content-Type: application/json" \
  -d '{"status": "agent_reviewing", "agent_notes": "..."}'

2. Implement the fix

Follow the repo's normal git workflow:

git fetch origin && git checkout main && git pull
git checkout -b fix/test-report-{short-slug}
# ... make changes ...
git push -u origin fix/test-report-{short-slug}
gh pr create --title "fix: ..." --body "Fixes test report #{REPORT_ID}..."

3. Mark as fixed

curl -s -X PATCH \
  "https://churchwiseai.com/api/test-reports/{REPORT_ID}" \
  -H "x-test-token: cw-testers-2026" \
  -H "Content-Type: application/json" \
  -d '{
    "status": "fix_applied",
    "agent_fix_details": "PR #427 — src/app/api/admin/billing/route.ts:42 now guards for null stripe_customer_id and returns an empty-state response instead of erroring. Deployed to prod via Vercel auto-deploy on main merge."
  }'

4. Leave it for human verification

John or Lexa will get notified (or see it on the test-reports dashboard), click through the fixed flow in production, and either approve or reopen.

Do NOT close reports yourself — human verification is the quality gate.

PATCH body reference

{
  "status":           "agent_reviewing | fix_proposed | fix_applied | human_verified | reopened | closed | wont_fix",
  "priority":         "low | medium | high | critical",
  "agent_notes":      "Free-text diagnosis / notes from the agent",
  "agent_fix_details": "PR link + file:line + 1-line description"
}

Only include fields you're changing. The route whitelists these four fields and ignores everything else (no arbitrary spread).

When to use `wont_fix` vs escalate

Use `wont_fix` when:

The reported behavior is working as intended (document why in agent_notes)
The scenario is out of scope for the property being tested
The report cannot be reproduced after 2 genuine attempts on production
The "bug" is a known, documented limitation

Always leave a clear agent_notes explanation — the tester needs to know why you closed it without fixing it.

curl -s -X PATCH "https://churchwiseai.com/api/test-reports/{ID}" \
  -H "x-test-token: cw-testers-2026" \
  -H "Content-Type: application/json" \
  -d '{"status": "wont_fix", "agent_notes": "Working as intended: the billing tab is hidden for demo churches (admin_token starts with demo-). Verified against tier-config.ts:isPaidPlan()."}'

Escalate to the founder when:

The bug touches billing (Stripe subscriptions, plan changes, checkout)
The bug touches auth (sign-in, sign-up, session, RBAC)
The bug touches life-safety (chatbot crisis detection, voice agent crisis flow, HEAR protocol)
The bug involves customer data (any write to non-demo rows)
You're unsure whether the intended behavior is actually working as intended

For escalations: add a detailed agent_notes comment, set status to agent_reviewing (not wont_fix), and file a P0/P1 in FOUNDER_ACTIONS.md.

Constraint-sync gotcha

The DB CHECK constraints on manual_test_reports and the UI scenario lists in src/lib/test-scenarios.ts must stay in sync. When they drift, testers get raw Postgres errors.

Source of truth for allowed values:

DB: migrations/2026-05-12-test-reports-constraint-sync.sql
API: src/app/api/test-reports/route.ts (ALLOWED_PROPERTIES, ALLOWED_CATEGORIES etc.)
UI: src/lib/test-scenarios.ts (PROPERTIES, JOURNEY_CATEGORIES arrays)

If a tester gets a 400 Invalid property: "..." or a DB constraint violation, add the new value to all three places and run the ALTER TABLE migration.

The "Have Agent Run This Test" button — what it is NOT

The test-reports page has a "Have Agent Run This Test" button that triggers POST /api/test-reports/agent-run. This is a smoke-test runner — it makes HTTP requests to production endpoints and chatbot calls against the demo church to verify basic availability.

It is NOT the bug-fixing workflow described in this runbook. Do not confuse the two:

	Smoke-test runner	Bug-fixing workflow
Triggered by	"Have Agent Run This Test" button	Agent picks up `status=reported` reports
What it does	Checks if pages/endpoints load	Diagnoses root cause, writes code fix, opens PR
Output	StepResult pass/fail list	Status PATCH + PR link
Side effects	Read-only (chatbot calls against demo church)	Code changes + DB status update

Adding comments to a report

curl -s -X POST \
  "https://churchwiseai.com/api/test-reports/{REPORT_ID}/comments" \
  -H "x-test-token: cw-testers-2026" \
  -H "Content-Type: application/json" \
  -d '{"author": "Claude Agent", "comment": "Confirmed root cause in billing query. Fix in PR #427."}'

src/app/api/test-reports/route.ts — GET list + POST create
src/app/api/test-reports/[id]/route.ts — PATCH status/notes
src/app/api/test-reports/[id]/comments/route.ts — GET + POST comments
src/app/api/test-reports/upload/route.ts — screenshot upload
src/app/api/test-reports/agent-run/route.ts — smoke-test runner
src/app/api/test-reports/auth.ts — token verification
src/app/test-reports/page.tsx — the UI (do not edit; owned by UI agent)
src/lib/test-scenarios.ts — scenario definitions (do not edit; owned by UI agent)
migrations/2026-05-12-test-reports-constraint-sync.sql — DB constraint history

What this system is​

Listing open reports an agent should work​

HTTP (preferred for agent server-side calls — token in header, not URL)​

HTTP (query-param form — fine for testing, avoid in server logs)​

Useful filters​

Status lifecycle​

How an agent works a report — step by step​

1. Claim the report​

2. Implement the fix​

3. Mark as fixed​

4. Leave it for human verification​

PATCH body reference​

When to use wont_fix vs escalate​

Use wont_fix when:​

Escalate to the founder when:​

Constraint-sync gotcha​

The "Have Agent Run This Test" button — what it is NOT​

Adding comments to a report​

Related files​