Skip to main content

QA Consistency Report -- Acceptance Specs (Phase 2)

Date: 2026-03-28 Reviewer: QA + Consistency Review Lead Scope: All 12 acceptance specs, PRICING.md, features.yaml, interview-action-items.md


Summary

MetricCount
Total specs reviewed12
Critical issues8
Warnings12
Notes9

Critical Issues (must fix before tests)

C1. [features.yaml] Pro agents list says 2, specs say 4

features.yaml defines Pro tier agents as ["Care Agent", "Coordinator Agent"] (2 agents). Every CWA acceptance spec for Pro and Suite tiers says Pro has 4 agents (Care, Coordinator, Discipleship, Stewardship). Suite has the same in features.yaml -- also listed as only 2. The canonical features.yaml is wrong or was never updated when the 4-agent architecture was established.

  • pro-suite-chat.md User State Definition: "Agents: Care + Coordinator + Discipleship + Stewardship (all 4)"
  • pro-both.md User State Definition: "Agents: Care + Coordinator + Discipleship + Stewardship (all 4)"
  • suite-both.md User State Definition: "Agents: Care + Coordinator + Discipleship + Stewardship + custom agents"
  • features.yaml pro.agents: ["Care Agent", "Coordinator Agent"] -- only 2
  • features.yaml suite.agents: ["Care Agent", "Coordinator Agent"] -- only 2

Fix: Update features.yaml Pro agents to ["Care Agent", "Coordinator Agent", "Discipleship Agent", "Stewardship Agent"] and Suite agents to the same plus custom agents note.


C2. [features.yaml] RBAC roles count: 7 vs 9

features.yaml lists rbac_roles.count: 7 with 7 roles defined. Multiple acceptance specs reference "9 roles" for team members (e.g., starter-chat.md Touchpoint 42B, starter-voice.md Touchpoint 42B, starter-both.md Touchpoint 42B). The features.yaml itself notes this discrepancy: "Suite feature says '9 roles' in pricing.ts, but AI Governance doc defines 7."

  • All acceptance specs say "Unlimited team members, 9 roles" at all tiers (not just Suite)
  • features.yaml says 7 roles and marks it as Suite-only feature (role-based team access)
  • Starter specs say 9 roles available, contradicting features.yaml saying RBAC is Suite-only

Fix: Reconcile the actual role count (7 or 9?), update features.yaml, and clarify whether team management is available at all tiers (as specs say) or Suite-only (as features.yaml implies).


C3. [starter-chat.md] Training sub-tabs: 6 vs 5

starter-chat.md Touchpoint 30 says Starter Chat has 6 sub-tabs: Church Knowledge, This Week, Theology, Agents, Safety, Training Progress.

starter-voice.md Touchpoint 30 says Starter Voice has 5 sub-tabs: Church Knowledge, This Week, Theology, Agents, Safety.

starter-both.md Touchpoint 30 says Starter Both has 5 sub-tabs: Church Knowledge, This Week, Theology, Agents, Safety.

The inconsistency is between Starter Chat (6 with Training Progress) and Starter Voice/Both (5 without Training Progress). However, Starter Both includes the chatbot channel and should logically match Starter Chat's sub-tab count. Further, Starter Both Touchpoint 38 includes a "Training Progress" section describing a 9-item checklist, implying it exists as a visible touchpoint even though Touchpoint 30 says only 5 sub-tabs.

Fix: Decide: Does Starter Both have 5 or 6 Training sub-tabs? If Training Progress is a sub-tab for Starter Chat, it should also be one for Starter Both (which includes chat). Update starter-both.md Touchpoint 30 to 6 sub-tabs if Training Progress applies. Alternatively, if the intent is 5 sub-tabs for all Starter tiers and Training Progress is embedded elsewhere, update starter-chat.md to match.


C4. [starter-chat.md] Getting Started checklist: 3 items vs email says 3 but other specs say different counts

There is inconsistency in the Getting Started checklist item counts across specs:

SpecChecklist Items
starter-chat.md3 (complete profile, train chatbot, share chat page)
starter-voice.md5 (profile, denomination, voice personality, forward phone, test call)
starter-both.md6 (phone line ready, profile, denomination, agent personalities, share chat, forward phone + test call)
pro-suite-chat.md4 + 1 bonus (profile, train chatbot, share/embed, theological lens, bonus: PewSearch)
pro-both.md6 + 1 bonus (phone line ready, profile, train agents, theological lens, share/embed, forward phone + test call, bonus: PewSearch)
suite-both.md6 + 1 bonus (same as pro-both but bonus mentions Pro Website)

This is internally consistent per spec and reflects channel differences. However, the starter-chat.md email (Touchpoint 18) says "3 things to do in your first 5 minutes" while pro-suite-chat.md says "4 things to do." The problem is that starter-chat has 3 checklist items but the pro-suite-chat spec has 4 -- and the 4th is "Set your theological lens." Should Starter Chat also have this step? The theological lens selector is described as "visible" at Starter in the User State Definition.

Fix: Confirm whether "Set your denomination/theological tradition" should be a Getting Started item for Starter Chat (making it 4 items, matching Pro but without the PewSearch bonus). Currently Starter Chat has only 3 and omits the theological lens step entirely from the checklist.


C5. [features.yaml] Pro persona_label says "Full Voice + Care Automation" -- wrong for chat-only

features.yaml line: persona_label: "Full Voice + Care Automation" for the Pro tier. This is flagged as a BUG in the pro-suite-chat.md spec (Touchpoint 2) where it says the Pro Chat card headline incorrectly says "Full Voice + Care Automation" when it should say "Your Full AI Ministry Team." The features.yaml is the source that propagates this error.

Fix: Change pro.persona_label in features.yaml from "Full Voice + Care Automation" to "Your Full AI Ministry Team" or a channel-agnostic label.


C6. [features.yaml] Pro feature list includes "Pro Website included" -- incorrect for Pro Chat standalone

features.yaml Pro tier features include "Pro Website included." However, the Pro Chat spec and Pro Both spec both say Pro gets Premium Listing only (not Pro Website). Pro Website is a Suite bonus. The current features.yaml is misleading.

  • pro-suite-chat.md: "Pro: Premium Listing ($9.95 value)" vs "Suite: Premium Listing + Pro Website ($19.95 value)"
  • pro-both.md: "Pro gets Premium Listing only, not Pro Website. Pro Website is Suite bonus."

Fix: Change "Pro Website included" in features.yaml Pro features to "Premium Listing included" or "PewSearch Premium Listing (free)."


C7. [starter-chat.md vs pro-suite-chat.md] Tab count for Starter Chat: 6 tabs with no Website tab, but pro-suite-chat says 7 tabs with Website tab

Starter Chat has 6 tabs: Overview, Requests, Care, Training, Settings, Upgrade. Pro/Suite Chat has 7 tabs: Overview, Requests, Training, Care, Website, Settings, Upgrade.

The tab order differs between these specs:

  • Starter Chat order: Overview, Requests, Care, Training, Settings, Upgrade
  • Pro/Suite Chat order: Overview, Requests, Training, Care, Website, Settings, Upgrade

Note that "Care" and "Training" swap positions between Starter and Pro/Suite. This may be intentional (to put Training more prominently for Pro users) or an error.

Fix: Verify the intended tab order is consistent. If tabs reorder by tier, document this explicitly. If it should be the same order at all tiers (with some tabs hidden), align the specs.


C8. [pro-both.md] Tab order differs from pro-suite-chat.md

pro-both.md Touchpoint 27 lists 8 tabs: Overview, Calls, Requests, Training, Care, Website, Settings, Upgrade.

pro-suite-chat.md Touchpoint 27 lists 7 tabs: Overview, Requests, Training, Care, Website, Settings, Upgrade.

Pro Both correctly adds "Calls" (voice channel) but the position of Calls between Overview and Requests differs from how Starter Voice places it (Starter Voice: Overview, Calls, Requests, Training, Settings, Upgrade). The positions are consistent for voice-inclusive plans (Calls comes after Overview), so this is actually consistent. However, within pro-both the order is: Overview, Calls, Requests, Training, Care, Website, Settings, Upgrade -- where Training comes before Care. In starter-both, the order is: Overview, Calls, Requests, Care, Training, Settings, Upgrade -- where Care comes before Training.

Fix: Verify tab order for both plans. Currently starter-both has Care before Training, pro-both has Training before Care. Decide on one consistent order.


Warnings (should fix)

W1. [features.yaml] Starter chat_limits.conversations_per_month: null but specs say 200

All Starter specs define a 200 monthly message limit (e.g., starter-chat.md Touchpoint 55, starter-both.md Touchpoint 55). But features.yaml shows starter.chat_limits.conversations_per_month: null. Similarly, Pro has null in features.yaml but specs say 500, and Suite says 2000 in features.yaml matching the specs.

Fix: Update features.yaml Starter conversations_per_month to 200 and Pro to 500.


W2. [features.yaml] Starter chat_limits.faqs: null and documents: null -- ambiguous

These are listed as null for Starter, but the specs clearly state FAQs and Documents are "NOT included (HIDDEN -- Pro+ feature)." The null could be interpreted as "no limit" rather than "not available." Other tiers use numeric values or "unlimited" for these fields.

Fix: Change Starter faqs and documents to 0 or "not_available" to clearly indicate the feature is absent, not unlimited.


W3. [sermonwise-pro.md] Annual pricing math: $19.95 x 10 = $199.50

The spec says $199.50/yr for SermonWise Pro annual. The formula (monthly x 10) gives $19.95 x 10 = $199.50. PRICING.md confirms $199.50/yr. This is correct. No issue.


W4. [itw-premium.md] Annual pricing: $99.50/yr

ITW Premium: $9.95/mo, annual = $9.95 x 10 = $99.50. PRICING.md confirms $99.50/yr. Correct.


W5. [pro-suite-chat.md] Touchpoint 13 says Starter gets no PewSearch bonus but starter-chat.md Touchpoint 13 describes a "Free Premium Listing" bonus

starter-chat.md Touchpoint 13 (redesigned) describes a "Bonus: Free Premium Listing!" post-checkout page. But the starter-both.md explicitly says "NOTHING -- Starter gets no PewSearch bonus." And starter-voice.md also says "Starter Voice gets no bonus."

Looking more carefully, starter-chat.md Touchpoint 13 is written with the NEW design (post-checkout bonus) but then pro-suite-chat.md Touchpoint 13 says "This touchpoint is REMOVED for Starter Chat -- no PewSearch bonus offered." These are contradictory.

Fix: The starter-chat.md Touchpoint 13 redesign section describes a bonus that was originally intended to be for all tiers but was later scoped to Pro+ only. The text in starter-chat.md should be updated to clearly say "NOTHING" for Starter, matching pro-suite-chat.md's note and all voice/bundle Starter specs.


W6. [cancelled.md] Does not explicitly mention voice number retention timing for bundle plans

cancelled.md Touchpoint CX-9 covers voice number retention (30 days) but only mentions "Voice/Bundle Plans" generically. The spec handles this adequately at a high level, but does not call out the bundle-specific nuance that when a bundle customer cancels, both the phone number AND chatbot go offline simultaneously. The spec does cover this in CX-7 (channel-aware banner copy) and CX-5/CX-6 (service during remaining period).

This is a minor gap, not a contradiction. The spec is logically complete but could be more explicit for bundle scenarios.


W7. [trial-expired.md] Does not mention voice plans at all in scope

trial-expired.md correctly says "Does NOT apply to: Voice plans, bundle plans (no trial)." This is correct and complete. However, there is no explicit spec for what happens when a voice/bundle customer's payment fails repeatedly (leading to an "expired-like" state). This is covered in cancelled.md Touchpoint CX-22 (failed payment leading to cancellation). No gap -- just confirming the lifecycle specs cover this.


W8. [features.yaml] Missing care_broadcasts limits per tier

The acceptance specs define care broadcast limits:

  • Starter: Subscribe page + member management only (no broadcasting)
  • Pro: 4 broadcasts/mo, 250 recipients max, 1,000 emails/mo cap
  • Suite: 12 broadcasts/mo, unlimited recipients, 5,000 emails/mo soft cap

features.yaml has a congregation_care add-on section that says broadcasting is DISABLED, but does not include the per-tier broadcast limits. These limits exist only in the acceptance specs.

Fix: Add care_broadcast_limits to features.yaml per tier to match the acceptance specs.


W9. [starter-both.md] Touchpoint 47 has an ambiguous note about embed widget

Starter Both Touchpoint 47 says "HIDDEN at Starter tier (per current tier-config; note that Pro Chat interview moved embed to all tiers -- if that change ships, this becomes visible)." This is an internal note about a potential future change that creates ambiguity. The current state should be definitive.

Fix: Remove the parenthetical about a potential future change. If the embed is hidden at Starter, say so definitively. If the decision to move embed to all tiers has been made, update all Starter specs consistently.


W10. [pro-suite-chat.md] Touchpoint 2 states embed widget moved to ALL tiers including Starter

pro-suite-chat.md says "Embed widget moved to ALL tiers (was wrongly Pro+ only on pricing card)." This contradicts starter-chat.md which lists embed widget as "NOT included (HIDDEN -- Pro+ feature)" in the User State Definition.

Fix: Decide definitively: Is embed widget available at Starter or not? Currently specs contradict each other. If the Pro Chat interview decision to move embed to all tiers is approved, update starter-chat.md, starter-voice.md, and starter-both.md User State Definitions. If not, remove the "moved to ALL tiers" note from pro-suite-chat.md.


W11. [features.yaml] channels.suite.voice: false but Suite Voice standalone is mentioned nowhere

features.yaml says suite.voice: false (Suite Voice standalone does not exist). This is consistent with PRICING.md which only lists Suite as chat-only or bundle. However, the note says "Suite is available as chat-only or full bundle (voice+chat)" which is correct. No issue -- just confirming this is intentional.


W12. [pro-both.md] Savings calculation may confuse customers

Pro Both says "Saves $24.95/mo vs separate." Let me verify: Pro Chat ($34.95) + Pro Voice ($69.95) = $104.90. Pro Both = $79.95. Savings = $24.95. Correct.

Suite Both says "Saves $19.95/mo vs separate." Suite Chat ($59.95) + (no Suite Voice standalone). Since Suite Voice doesn't exist as standalone, this savings claim compares Suite Both ($99.95) to what? Suite Chat ($59.95) + Pro Voice ($69.95) = $129.90. Savings = $29.95. But the spec says $19.95. Alternatively: Suite Chat ($59.95) + Starter Voice ($39.95) = $99.90. Then savings = -$0.05 (Suite Both costs more!).

The $19.95 savings claim seems to be calculated as: Suite Chat ($59.95) + Voice component priced at $39.95 (Starter Voice) = $99.90 vs Suite Both at $99.95 -- which is actually $0.05 MORE expensive, not saving $19.95. The savings claim appears wrong.

Fix: Recalculate Suite Both savings claim. If Suite Voice standalone doesn't exist, the "saves vs separate" comparison needs to be clarified or the savings claim needs to be recalculated against a sensible baseline (perhaps Pro Voice at $69.95: $59.95 + $69.95 = $129.90, savings = $29.95).


Notes (minor, nice-to-have)

N1. [all CWA specs] Touchpoint numbering is consistent at 62

All 8 CWA acceptance specs (starter-chat, starter-voice, starter-both, pro-suite-chat, pro-both, suite-both, trial-expired, cancelled) use the same 62-touchpoint framework (A through H) plus lifecycle-specific touchpoints. Cross-property specs (ITW, PewSearch, SermonWise) use their own numbering appropriate to their simpler journeys. This is correct and consistent.


N2. [all specs] "Hidden not locked" principle is consistently applied

All specs consistently describe unavailable features as "HIDDEN" not "locked" or "grayed out." No spec mentions gray overlays, "Upgrade to unlock" buttons within content areas, or disabled states for tier-gated features. The Upgrade tab is the sole location for tier comparison and upgrade CTAs. Principle compliance is excellent.


N3. [all specs] "Did you know?" educational moments used correctly

Starter specs use "Did you know?" text with links (not buttons) to the Upgrade tab. No spec puts upsell buttons in the main dashboard content. Pro/Suite specs correctly omit upgrade CTAs from the main content area. Principle compliance is good.


N4. [all specs] Soft delete mentioned consistently

Call logs, prayer requests, visitor contacts, and callbacks all mention soft delete/hide functionality. No spec references hard deletion of customer data. Consistent.


N5. [all specs] Team members listed as "Unlimited, 9 roles" at all tiers

Every CWA spec says unlimited team members with 9 roles. This is consistent across all specs but contradicts features.yaml (see C2 above).


N6. [cancelled.md] Covers voice scenarios adequately

The cancelled spec covers voice-specific scenarios including: what callers hear (CX-9), phone number retention (30 days, CX-9), channel-aware banner copy (CX-7), and bundle cancellation (CX-21). The interview action item #3 (voice-specific cancellation scenarios) appears to be addressed.


N7. [trial-expired.md] Correctly excludes voice plans

The trial-expired spec correctly states it applies only to chat plans and explicitly excludes voice and bundle plans (which have no trial). Correct.


N8. [pewsearch-premium.md and pewsearch-pro-website.md] PewSearch team has 7 roles, not 9

PewSearch Settings tab (Touchpoint 12) lists 7 team roles: admin, office_admin, prayer_team, care_team, treasurer, volunteer_coordinator, worship_leader. This matches features.yaml's 7-role list. CWA specs say 9 roles. The discrepancy may be because CWA adds 2 additional roles not in PewSearch (possibly deacon and elder, or the distinction between admin_token owner and viewer). This needs clarification (see C2).


N9. [sermonwise-pro.md] Gaps flagged are real and need attention

The SermonWise spec flags several real gaps: sermon series planning listed but not built, no settings/billing page, no welcome email, no payment failure handling, no account deletion mechanism. These are accurately identified in the spec. They are product gaps, not spec inconsistencies.


Price Verification

CheckResultNotes
Starter Chat $14.95/moPASSMatches PRICING.md
Starter Chat Annual $149.50/yrPASS$14.95 x 10 = $149.50
Pro Chat $34.95/moPASSMatches PRICING.md
Pro Chat Annual $349.50/yrPASS$34.95 x 10 = $349.50
Suite Chat $59.95/moPASSMatches PRICING.md
Suite Chat Annual $599.50/yrPASS$59.95 x 10 = $599.50
Starter Voice $39.95/moPASSMatches PRICING.md
Pro Voice $69.95/moPASSMatches PRICING.md
Starter Both $49.95/moPASSMatches PRICING.md
Pro Both $79.95/moPASSMatches PRICING.md
Suite Both $99.95/moPASSMatches PRICING.md
PewSearch Premium $9.95/moPASSMatches PRICING.md
PewSearch Pro Website $19.95/moPASSMatches PRICING.md
ITW Premium $9.95/moPASSMatches PRICING.md
ITW Premium Annual $99.50/yrPASS$9.95 x 10 = $99.50
SermonWise Pro $19.95/moPASSMatches PRICING.md
SermonWise Pro Annual $199.50/yrPASS$19.95 x 10 = $199.50
AI Starter Kit $4.95PASSMatches PRICING.md
Annual math (monthly x 10)PASSAll annual prices = monthly x 10
Trial: 14 days for chat plansPASSAll chat specs say 14-day trial
Trial: None for voice/bundlePASSAll voice/bundle specs say no trial
Trial: None for PewSearchPASSPreview mode instead
Trial: None for ITW/SermonWisePASSFree tier serves as try-before-buy
Starter Both savings $4.95/moPASS($14.95 + $39.95) - $49.95 = $4.95
Pro Both savings $24.95/moPASS($34.95 + $69.95) - $79.95 = $24.95
Suite Both savings $19.95/moFAILSee W12. No Suite Voice standalone exists; baseline for comparison is unclear

Cross-Spec Consistency Matrix

CheckResultDetails
Tab counts consistent per tier+channelWARNStarter Chat=6, Starter Voice=6, Starter Both=7, Pro/Suite Chat=7, Pro/Suite Both=8. Counts are correct per channel but tab ORDER differs (C7, C8)
Feature access consistent across same-tier specsWARNEmbed widget claimed "moved to all tiers" in pro-suite-chat but hidden at Starter elsewhere (W10)
Training sub-tabs: Starter=5 or 6, Pro/Suite=7FAILStarter Chat says 6 (with Training Progress), Starter Voice/Both say 5 (without it) -- inconsistent (C3)
Agent visibility: Starter=2, Pro/Suite=4PASSAll specs agree: Starter=Care+Coordinator, Pro/Suite=all 4
Agent count in features.yamlFAILfeatures.yaml says 2 agents for all tiers including Pro/Suite (C1)
Stat cards consistentPASSChat-only=5 (conversations, prayers, visitors, callbacks, care members), Voice-only=5 (calls, prayers, visitors, callbacks, avg duration), Both=5 (calls, conversations, prayers combined, visitors combined, callbacks combined)
Getting Started auto-dismiss: 7 daysPASSAll specs say auto-dismiss after 7 days or all complete
Prices match PRICING.mdPASSAll prices verified (see table above)
Annual math correctPASSAll annual = monthly x 10
Trial durations correctPASSChat=14 days, Voice/Bundle=none
"Hidden not locked" principlePASSNo spec mentions gray overlays or "Upgrade to unlock" in dashboard
No upsell buttons in dashboardPASSOnly Upgrade tab has upgrade CTAs; main content uses text links only
Soft delete mentionedPASSAll request types and call logs reference soft delete
Team members unlimited at all tiersPASSAll specs say unlimited team members
Role count (7 vs 9)FAILSpecs say 9, features.yaml says 7 (C2)
PewSearch bonus correct per tierPASSStarter=none, Pro=Premium Listing, Suite=Premium Listing + Pro Website
Bundle no annual billingPASSAll bundle and voice specs hide annual toggle
Bundle no trialPASSAll bundle and voice specs say immediate charge
Cancelled spec covers voicePASSChannel-aware copy, phone number retention, caller experience
Trial-expired excludes voicePASSExplicitly scoped to chat-only plans
Care broadcasts per tier definedWARNDefined in specs but missing from features.yaml (W8)
Suite Both savings claimFAIL$19.95 savings claim has no valid baseline (W12)

Interview Action Items Verification

Action ItemAddressed in Specs?Notes
#1: Product Knowledge FAQ (no trial/annual for voice)NOT IN SPECSThis is a product_knowledge DB insert, not a spec item. Needs to be executed separately.
#2: Voice welcome email two-email flowYESAll voice and bundle specs describe Email 1 (immediate) + Email 2 (after provisioning). Consistent.
#3: Cancelled interview must cover voice scenariosYEScancelled.md covers voice offline (CX-9), phone number retention, caller experience, bundle scenarios (CX-21).
#4a: Phone provider field in onboardingYESstarter-voice.md (TP 12), starter-both.md (TP 12), pro-both.md (TP 12), suite-both.md (TP 12) all include phone provider dropdown.
#4b: Carrier-specific forwarding instructionsYESAll voice spec Email 2 (18B) describes carrier-specific instructions based on phone provider.
#5: Soft delete for call logs and requestsYESAll specs reference soft delete/hide for calls and requests.
#6: BACKLOG: Care tab for voice-only (outbound calling)ACKNOWLEDGEDstarter-voice.md explicitly says Care tab is hidden, with note about backlog for outbound calling.

Recommendations

Immediate (before Playwright test generation)

  1. Fix features.yaml agent counts (C1) -- Pro and Suite should list 4 agents, not 2
  2. Fix features.yaml persona_label (C5) -- Remove voice reference from Pro label
  3. Fix features.yaml Pro Website inclusion (C6) -- Pro gets Premium Listing, not Pro Website
  4. Resolve training sub-tab count for Starter (C3) -- Decide 5 or 6 for all Starter tiers
  5. Resolve embed widget tier availability (W10) -- Is it Starter or Pro+?
  6. Resolve role count: 7 or 9 (C2) -- Count actual roles, update features.yaml and specs
  7. Fix Suite Both savings claim (W12) -- Recalculate or reframe the baseline
  8. Resolve tab order inconsistency (C7, C8) -- Standardize Care vs Training position

Soon (before launch)

  1. Add conversations_per_month to features.yaml for Starter (200) and Pro (500)
  2. Add care broadcast limits to features.yaml
  3. Clarify Starter faqs/documents as "not_available" instead of null
  4. Clean up starter-chat.md Touchpoint 13 to clearly say no PewSearch bonus for Starter
  5. Remove ambiguous "if that change ships" note from starter-both.md Touchpoint 47
  6. Execute product_knowledge INSERT for voice trial/annual FAQ (action item #1)

Backlog

  1. Monitor Suite Both savings claim once Suite Voice standalone is considered
  2. Consider adding Getting Started theological lens step to Starter Chat (C4)