Production Testing Runbook
The Rule
Never use WebFetch to verify React CSR pages. WebFetch returns static HTML — it sees "Loading admin panel..." instead of the rendered dashboard. Always use Playwright for production verification.
Running Playwright Against Production
All 3 codebases support BASE_URL override. When set, the dev server is NOT started.
ChurchWiseAI (churchwiseai.com)
cd C:\dev\churchwiseai-web
BASE_URL=https://churchwiseai.com npx playwright test --project=desktop
PewSearch (pewsearch.com)
cd C:\dev\pewsearch\web
BASE_URL=https://pewsearch.com npx playwright test --project=desktop
IllustrateTheWord (illustratetheword.com)
cd C:\dev\sermon-illustrations
BASE_URL=https://illustratetheword.com npx playwright test --project=desktop
SermonWise (sermonwise.ai)
cd C:\dev\churchwiseai-web
BASE_URL=https://sermonwise.ai npx playwright test sermonwise --project=desktop
Running Specific Test Files
BASE_URL=https://churchwiseai.com npx playwright test smoke.spec.ts --project=desktop
BASE_URL=https://churchwiseai.com npx playwright test e2e/delivers/ --project=desktop
Production Smoke Test (Quick Verification)
# Run just smoke tests across all 3 codebases against production
cd C:\dev\churchwiseai-web && BASE_URL=https://churchwiseai.com npx playwright test smoke.spec.ts --project=desktop
cd C:\dev\pewsearch\web && BASE_URL=https://pewsearch.com npx playwright test smoke.spec.ts --project=desktop
cd C:\dev\sermon-illustrations && BASE_URL=https://illustratetheword.com npx playwright test smoke.spec.ts --project=desktop
When to Use Production Testing
| Scenario | Use localhost | Use production |
|---|---|---|
| Writing new tests | Yes | No |
| Pre-commit validation | Yes | No |
| Post-deploy verification | No | Yes |
| QA audits / go-live readiness | No | Yes |
| Persona / journey testing | No | Yes |
| Visual regression baselines | No | Yes (production is the source of truth) |
Agent Instructions
When running QA audits, go-live reviews, or persona testing:
- NEVER use WebFetch on admin dashboards or any React CSR page — you'll only see a loading spinner
- Use
npx playwright testwithBASE_URL=for any page that needs real browser rendering - For quick checks of static marketing pages, WebFetch is acceptable
- For interactive testing (forms, chatbot, tabs), always use Playwright
Taking Screenshots Against Production
# Run a specific test that takes screenshots
BASE_URL=https://churchwiseai.com npx playwright test visual.spec.ts --project=desktop --update-snapshots
Limitations
- Tests that write to the database will write to PRODUCTION — be careful
- Tests that trigger Stripe checkouts will use real Stripe (test mode keys should still work)
- Rate limiting may affect tests run repeatedly against production
- Auth-dependent tests need valid credentials in .env.local or .env.test