Test Finding Post-Op Procedure

The Problem This Solves

Testing agents find issues and fix code. But the root cause often exists in multiple places — journey specs, persona YAMLs, CLAUDE.md, product_knowledge, marketing copy, acceptance specs. If you only fix the code, the NEXT audit finds the same "bug" again because the specs are still wrong.

Example: SermonWise journey tests reported "FAIL: /sermons/dashboard returns 404." Three agents spent time investigating. We built redirect pages. But the real issue was that journey YAMLs pointed to /sermons/dashboard (doesn't exist) instead of /sermons/app (the real route). The code was fine. The specs were wrong.

The Procedure

After EVERY significant test finding (SPEC VIOLATION, GOAL BLOCKED, or any fix that changes behavior), run through this checklist:

Step 1: Classify the Finding

Was it...	Then...
A real code bug	Fix code → then check if docs/specs describe the old (buggy) behavior and need updating
A spec/doc error	Fix the spec → then verify the code actually works correctly at the real URL/behavior
A gap (feature missing)	Build the feature → then add it to all relevant specs, journeys, personas, product_knowledge
A stale reference	Fix the reference → then search for the same stale reference everywhere else

Step 2: Ripple Search

Search for the same incorrect information across ALL knowledge sources:

grep -r "the wrong thing" across:
  C:\dev\knowledge\tests\journeys\*.yaml       # Journey specs
  C:\dev\knowledge\tests\personas\*.yaml        # Persona definitions
  C:\dev\knowledge\acceptance\*.md              # Acceptance specs
  C:\dev\knowledge\**\*.md                      # All knowledge docs
  C:\dev\*\CLAUDE.md                            # All project CLAUDE.md files
  C:\dev\PRICING.md                             # Pricing source of truth
  C:\dev\FEATURE_REGISTRY.md                    # Feature registry

Fix EVERY instance, not just the one the test found.

Step 3: Source of Truth Verification

Ask: "Where should the correct value live canonically?"

What changed	Canonical source
Route/URL	The actual src/app/*/page.tsx files (code is truth)
Pricing	C:\dev\PRICING.md + knowledge/data/pricing.yaml
Features	C:\dev\FEATURE_REGISTRY.md + knowledge/data/features.yaml
Product behavior	The code + acceptance specs in knowledge/acceptance/
Denomination/tradition list	src/lib/theolenses.ts (code is truth)
Church count	Database query with directory_visible=true (DB is truth)

If the canonical source is wrong, fix it FIRST. Then propagate to all downstream docs.

Step 4: Regression Prevention

Add a test or check that catches this specific issue automatically:

Finding type	Prevention
Wrong URL in spec	Add URL validation: for every url: field in journey/persona YAMLs, verify the route exists in the codebase
Stale pricing	Cross-source consistency test (already exists in knowledge/tests/)
Missing feature	Add to FEATURE_REGISTRY.md
Broken link	Add to Playwright smoke test
Wrong church count	Already caught by feedback_church_count_filter.md

Step 5: Document the Learning

If the finding reveals something non-obvious about the system:

• Update the relevant CLAUDE.md
• Add a memory if it's a pattern agents keep getting wrong
• Update the DECISION_LOG.md if a design decision was involved

When to Skip This

• Visual baseline drift (just update baselines)
• Flaky timeouts (fix the test, not the docs)
• Rate limiting masking tests (environmental, not systemic)

When This is MANDATORY

• Any SPEC VIOLATION or GOAL BLOCKED finding
• Any fix that changes a URL, route, price, feature name, or product behavior
• Any finding where the code was correct but the spec was wrong
• Any finding that affected 2+ agents (indicates a systemic documentation gap)