Persona Test & TAG Consistency Results — 2026-03-31
Session: Full handoff execution (Phases 1-4)
Agents dispatched: 6 TAG + 11 persona = 17 total
Products tested: PewSearch, IllustrateTheWord, SermonWise, CWA Dashboard
Personas used: Tired Pastor, Justice-Minded Fact Checker, Anxious Board Member, Budget Treasurer, Catholic Secretary
Fixes Applied This Session
Phase 1: Branch Merges + Hardening
| Fix | Codebase | Commit |
|---|
Force currency: 'usd' on ITW checkout | sermon-illustrations | fc53bc4 |
| Plain-text URL fallback in welcome email | pewsearch | 0a373e5 |
| Duplicate embed widget, role 7->9, agent 2->4, illustration 30K->50K | churchwiseai-web | 6b6f114d |
adaptive_pricing: { enabled: false } on ITW checkout | sermon-illustrations | 5b48a8c |
adaptive_pricing: { enabled: false } on PewSearch checkout + pre-checkout | pewsearch | 3647cdc |
Phase 2: TAG Fixes
| Fix | Codebase | Commit |
|---|
| Church count 261K->218K on about page (2 instances) | pewsearch | e35aff3 |
| Church count 261K->218K in cross-promo card | sermon-illustrations | 77d8fc1 |
| Demo page $34.95->$9.95, Premium description fix | pewsearch | 9de849b |
| Chatbot "14-day"->48-hour preview | pewsearch | (in 9de849b) |
| Church page overstated Premium features | pewsearch | (in 9de849b) |
| Chatbot sales line->demo numbers for voice agent | pewsearch | 98511d4 |
| PDF generator: 33 tools->39, 26K->50K, 240K->218K | churchwiseai-web | 12175a88 |
| Demo page 290K->327K resources + chatbot framing->directory framing | pewsearch | bfec28e |
Other Fixes
| Fix | Codebase | Status |
|---|
| Duplicate church idempotency bug (provisionNewChurch) | churchwiseai-web | Agent dispatched, in progress |
Critical Findings (Blocks CEO Testing)
C1. No AI Disclosure on Church Chatbot
Product: PewSearch (church chatbot at /chat/[slug])
Source: Board Member persona
Issue: Visitors see "Chat with [Church Name]" and "Powered by PewSearch" with NO indication they're talking to AI. FTC guidance requires clear AI disclosure. A grieving widow could share a prayer request thinking she's reaching the pastor.
Action: Add "AI Assistant" label to chat interface header. Update privacy policy to mention AI/OpenAI.
Owner: Founder decision (legal/policy)
C2. Terms of Service Silent on AI
Product: PewSearch
Source: Board Member persona
Issue: ToS contains zero mention of AI, chatbot accuracy, or responsibility for AI-generated responses. No disclaimer that AI may be inaccurate. No carve-out for AI errors.
Action: Add AI-specific section to ToS covering accuracy disclaimer, theological content, liability.
Owner: Founder decision (legal)
C3. Privacy Policy Doesn't Mention OpenAI
Product: PewSearch
Source: Board Member persona
Issue: Visitor messages go to Anthropic (Claude Haiku 4.5, primary) and OpenAI (gpt-4o-mini, fallback + embeddings) but this is not disclosed in the privacy policy. The policy names Stripe, Supabase, Vercel, Resend, PostHog — but not the AI providers processing conversation content.
Action: Add Anthropic and OpenAI to the service providers section of the privacy policy.
Owner: Founder decision (legal)
C4. Dashboard Getting Started Checklist Expires After 7 Days
Product: CWA Dashboard
Source: Tired Pastor persona
Issue: The only onboarding guide auto-hides after 7 days. Pastors who delay setup (extremely common) return to zero guidance. Most likely cause of trial churn.
Action: Make checklist persistent until all steps are complete, or add a permanent "Setup Guide" link.
C5. Feature Matrix Shows Disabled/Undeployed Features as Available
Product: CWA Dashboard (UpgradeTab)
Source: Fact Checker persona
Issue: "Congregation Care Messaging" (broadcasting disabled) and "AI Social Media (ShareWiseAI)" (not deployed) both show green checkmarks for Pro tier. Pastors who upgrade expecting these will feel deceived.
Action: Remove or add "Coming Soon" badges to these rows in TIER_FEATURE_MATRIX.
C6. Dashboard Says "Pro Website Included" but Spec Says Pro Gets Premium Listing
Product: CWA Dashboard
Source: Fact Checker persona
Issue: The Pro Website CTA card says "Pro Website Included" for Pro-tier customers, but features.yaml says Pro gets "Premium Listing ($9.95 value)" — Suite gets "Pro Website ($19.95 value)". Wrong tier benefit displayed.
Action: Verify which PewSearch product is provisioned for Pro subscribers and fix the dashboard label to match.
Important Findings (CEO Will Notice)
PewSearch
| ID | Finding | Source |
|---|
| P1 | "Admin panel" used 4+ times with no plain-English explanation | Tired Pastor |
| P2 | No explanation of what happens after "Continue to Payment" | Tired Pastor |
| P3 | "Claim Your Church" CTA lands on unsignposted directory search page | Tired Pastor |
| P4 | Pricing page doesn't list chatbot on Premium plan; claim page does | Tired Pastor + Fact Checker |
| P5 | 48-hour preview not mentioned on pricing page | Tired Pastor + Fact Checker |
| P6 | Terms of Service no-refund policy doesn't mention preview exception | Fact Checker |
| P7 | Liability capped at ~$240 (12 months of fees) — church bears AI risk | Board Member |
| P8 | Governing law is Ontario, Canada — US churches have limited recourse | Board Member |
| P9 | "SEO-optimized with structured data" jargon on Pro Website plan | Tired Pastor |
| P10 | Basic chatbot not listed in Premium features on pricing page | Fact Checker |
IllustrateTheWord
| ID | Finding | Source |
|---|
| I1 | Three-tier access system (public/free-signup/premium) never explained upfront | Tired Pastor |
| I2 | Illustration count: homepage says 53K+ dynamically, 13+ hardcoded strings say 50K+ | Fact Checker |
| I3 | No pricing link in main navigation | Tired Pastor |
| I4 | "LensLines" jargon with no explanation | Tired Pastor |
| I5 | Signup page says "thousands" instead of "53,000+" | Tired Pastor |
| I6 | Post-email-confirmation redirects to homepage, not the illustration that triggered signup | Tired Pastor |
| I7 | Content type labels shown as developer text ("movie_analogy", "vivid_retelling") | Tired Pastor |
SermonWise
| ID | Finding | Source |
|---|
| S1 | AI persona hardcoded as "pastor" for ALL traditions including Catholic (sermon-prompt.ts:179) | Catholic Secretary |
| S2 | App UI uses "sermon" throughout — no adaptive label for Catholic/Orthodox ("homily") | Catholic Secretary |
| S3 | Homepage tradition strip includes "Universal" and "Reformed Baptist" — neither maps to a distinct lens | Fact Checker |
| S4 | Homepage lists 5 sermon styles (with Homily); pricing page lists only 4 (without Homily) | Fact Checker |
| S5 | Tradition selector marked "(optional)" but triggers mandatory onboarding redirect if skipped | Tired Pastor |
| S6 | No auth redirect on /sermons/app — blank flash then forced login for logged-out users | Tired Pastor |
| S7 | Signup page missing "no credit card needed" reassurance | Tired Pastor |
| S8 | "Compare to Generic AI" link in hero is 40% opacity — nearly invisible | Tired Pastor |
CWA Dashboard
| ID | Finding | Source |
|---|
| D1 | FAQ management not discoverable from Overview — requires Training > FAQs (3 clicks) | Tired Pastor |
| D2 | Document upload buried 4 clicks deep with no signposting | Tired Pastor |
| D3 | "Hero Photo URL" field label is jargon — should be "Church Banner Photo" | Tired Pastor |
| D4 | "Short URL" field label — should be "Custom Web Address" | Tired Pastor |
| D5 | Integration fields expose raw developer labels: "API Key", "Personal Access Token", "Application ID" | Tired Pastor |
| D6 | Agent tool counts (e.g., "8/11") shown without explaining what tools are | Tired Pastor + Fact Checker |
| D7 | Agent card denominators include Suite-locked tools — ambiguous for Pro users | Fact Checker |
| D8 | No persistent "Share your chatbot" prompt on Overview | Tired Pastor |
| D9 | Pro Website card links to PewSearch without explaining the relationship | Tired Pastor |
| D10 | Red safety banner ("Crisis detections...") is alarming as first thing a pastor sees | Tired Pastor |
Minor Findings (Polish)
PewSearch
- "Stripe" named in trust signals but unexplained
- "WiseAI Ecosystem" in footer unexplained
- "Cinematic templates" — unclear meaning for pastors
- "Priority support" undefined
- "Authorized representative" legal language on claim checkbox
- "Apostles' Creed" requirement may confuse some evangelical pastors
IllustrateTheWord
- Annual pricing math shows % but not dollar savings
- "AI-crafted by ChurchWiseAI" badge may create authenticity doubts
- "Surprise Me" button with no explanation
- LensLines trademark symbol feels corporate
- support@churchwiseai.com cross-brand leak on ITW site
- No formal free trial, but free tier compensates
SermonWise
- "Sign In" alongside "Get Started Free" creates decision paralysis
- Showcase page proves capability but lacks pastor testimonials
- "My Sermons" nav label confusing for new users with zero sermons
- "Church of Christ" denomination ambiguous (multiple denominations share name)
- "Saved preferences" marketing card doesn't clarify Free/Pro distinction
- PDF export shown as general feature but is Pro-only
CWA Dashboard
- "Simulator" tab label should be "Test Your AI"
- "Care Hub" label unclear
- Embed code section mixes pastor and webmaster audiences
- Channel badges ("Chat" vs "Chat + Voice") unexplained
- "0 of 50 calls used" shows without explaining what happens at limit
- Tab order gap when Calls tab is hidden on chat-only plans
Jargon Violations Found (13 instances — FA-016 scope)
| Term | File | Severity |
|---|
| "ministry tools" (no tooltip) | PewSearch UpgradeTab.tsx:205 | High |
| "Hero Photo URL" | CWA SettingsTab.tsx:190 | High |
| "Hero Photo URL" | PewSearch SettingsTab.tsx:132 | High |
| "Short URL" | CWA SettingsTab.tsx:192 | High |
| "Short URL" | PewSearch SettingsTab.tsx:141 | High |
| "API Key" | CWA SettingsTab.tsx:511 | High |
| "API Key" | CWA TrainingTab.tsx:649 | High |
| "Personal Access Token" | CWA SettingsTab.tsx:482, TrainingTab.tsx:602 | High |
| "Agent Personalities" + "handoff rules" | CWA TrainingTab.tsx:882 | Medium |
| "Agent Tool Settings" | CWA AgentToolSettings.tsx:296 | Medium |
| "Theological Lens" (no tooltip) | CWA TheologySettings.tsx:243 | Medium |
| "personas" count badge | CWA AgentConfig.tsx:271 | High |
| "ElevenLabs API key" error msg | CWA VoicePicker.tsx:38 | Medium |
TAG Registry Updates Needed
| Tag | Current Canonical | Should Be | Reason |
|---|
| illustration_count | 30,000+ | 53,000+ | DB has 53,115; homepage renders dynamically; TAG registry and 13+ hardcoded strings are stale |
Recommendations for Founder
Fix Now (before next CEO test session)
- C5 — Remove Congregation Care + ShareWiseAI checkmarks from feature matrix
- C6 — Fix "Pro Website Included" label to match actual tier benefit
- C4 — Make Getting Started checklist persistent (not 7-day expiry)
- D8 — Add "Share your chatbot" with one-click copy link to Overview
Fix Before Launch
- C1-C3 — AI disclosure, ToS AI section, privacy policy OpenAI mention (legal)
- S1 — Make AI persona tradition-aware ("priest" for Catholic, not "pastor")
- P4 — Align Premium feature lists between pricing page and claim page
- I1 — Explain three-tier access system before signup
Backlog (Important but not blocking)
- All 13 jargon violations (FA-016 label pass)
- ITW illustration count: update 13 hardcoded "50,000+" strings to "53,000+"
- SermonWise tradition strip: remove fake "Universal" and "Reformed Baptist" labels
- PewSearch 48-hour preview: mention on pricing page
- All integration field labels: add step-by-step walkthroughs
- Internal docs: update 5 occurrences of 261K in CWA internal doc pages