Skip to main content

Knowledge > Runbooks > Agent Ops > How to Run the QA Checklist

How to Run the QA Checklist

A senior QA engineer's approach to self-reviewing your own work before shipping. The goal is to find bugs others miss, not to rubber-stamp your own implementation.

Prerequisites​

  • Implementation complete and committed to feature branch
  • C:\dev\QA_CHECKLIST.md open (9 sections)
  • C:\dev\AGENT_QUALITY_PRINCIPLES.md open (35 principles)

Steps​

1. Switch Mental Model​

Before starting the QA review, explicitly shift perspective:

"I am now a senior QA engineer who does not know or care how this code was implemented. I am trying to find any way this could fail, confuse a user, or produce wrong output."

This mental shift is essential. As the implementer, you have blind spots. A QA engineer has none — they only see behavior.

2. Read Through QA_CHECKLIST.md (All 9 Sections)​

The 9 sections cover:

  1. Database queries and safety
  2. Security and auth
  3. Client/server boundary (Next.js)
  4. SEO and metadata
  5. Content accuracy and marketing claims
  6. UI/UX and branding
  7. API and integration correctness
  8. Performance and scalability
  9. Error handling and edge cases

Work through each section systematically. For each checklist item, actively look for the problem — don't just read the question and assume you're fine.

3. Read Through AGENT_QUALITY_PRINCIPLES.md​

These 35 principles are derived from 165 real bugs that shipped. For each principle, ask: "Did I make this exact mistake in what I just built?"

Common failure modes agents repeatedly hit:

  • Missing directory_visible=true filter on churches queries
  • Using 261K (total) instead of 218K (visible) in any church count
  • Importing server-only modules into 'use client' components
  • Hardcoding pricing that's already stale vs. pricing.yaml
  • Claiming features that don't exist yet (unapproved features in copy)
  • Forgetting to paginate large table queries
  • Not updating product_knowledge when flows change

4. Persona-Based Testing​

Test your implementation as each of these personas. If you can't test the actual UI, trace through the code path as if you were this user:

The Skeptical Pastor

  • Has been burned by tech before
  • Will read every word on the pricing page
  • Will test the chatbot with hard theological questions
  • Asks: "Is this price really what it says? What happens if I cancel?"

The Tiny Rural Church

  • Single staff member, 40 members, very limited budget
  • May have no tech knowledge base set up
  • What does the chatbot say when there's no KB content?
  • Does the starter plan actually work without customization?

The Committee Buyer

  • Must present this to a board of 5 people
  • Will screenshot the pricing page and feature list
  • Every claim must be accurate — one wrong claim loses the sale

The New Visitor

  • Found the church online, wants to learn more
  • Will use the widget on the church's site without knowing it's AI
  • Should get helpful, accurate, warm responses

For each persona: does the experience make sense? Is the data correct? Are any claims wrong?

5. Check Data Sanity​

Review any data displayed in the UI or returned by the API:

  • Are counts plausible? (Churches: ~218K. unified_rag_content: ~327K. Premium churches: 4 currently)
  • Are prices correct? (Always check against PRICING.md or pricing.yaml)
  • Are dates formatted correctly and in the right timezone?
  • Are empty states handled gracefully? (No content, no calls, new account)
  • Are error states handled? (API down, invalid token, expired session)

6. Test Edge Cases​

For every input field or action in your implementation, ask:

  • What happens with empty input?
  • What happens with extremely long input?
  • What happens when the user is unauthenticated?
  • What happens when a required dependency (Supabase, Stripe, external API) is down?
  • What happens for a brand-new account with zero data?

7. Fix ALL Issues Found — Never Defer​

If you find anything — critical, important, OR minor — fix it before declaring done.

Do NOT:

  • Add it to a "future PR" list
  • Note it as "known issue"
  • Skip it because "it's unlikely to happen"

Every deferred issue is a bug that will eventually reach a real customer or cause a real failure.

8. Verify on the Deployed URL​

After deploying, verify the actual production URL — not localhost:

# Check the deployed page actually works
# Use WebFetch tool or curl to read the deployed response
curl -s https://churchwiseai.com/[changed-path] | head -50

Common gaps between local and production:

  • Env vars missing in Vercel
  • Build-time vs runtime behavior differences
  • Tailwind v4 CSS that worked in dev but not in Vercel build

Verification​

  • Every checklist item in QA_CHECKLIST.md has been reviewed
  • All issues found are fixed — no deferrals
  • At least one persona-based test has been traced through
  • Deployed URL confirms expected behavior

See Also​