Product Knowledge Alignment — Chatbot/Voice Info

  Property: ChurchWiseAI   Category: Chatbot Tier: any-chat Persona: pastor-assistant Touchpoint: /chat/[slug], voice agent

Preconditions

• product_knowledge table populated and validated
• Chatbot injected with product_knowledge via system prompt
• Voice agent uses product_knowledge at session initialization

Steps

#	Action	Expected Result
1	Ask chatbot: 'What is ChurchWiseAI?'	Bot returns product_knowledge entry for 'What is ChurchWiseAI'. Answer matches marketing description.
2	Ask: 'How much does the Pro plan cost?'	Bot returns price from product_knowledge. Price matches PRICING.md ($59.95/mo for chat).
3	Ask: 'How many tools are available on Starter?'	Bot returns '12 tools' (from product_knowledge). Matches features.yaml.
4	Ask: 'What is included in voice setup?'	Bot answers from product_knowledge. Includes phone number, SIP, cost ($49.95 setup). Accurate.
5	Ask: 'Can I cancel anytime?'	Bot returns cancellation policy from product_knowledge. Matches PRICING.md (month-to-month, no lock-in).
6	Ask: 'What denominations do you support?'	Bot lists supported traditions from product_knowledge. All 5+ traditions mentioned.
7	Call voice agent and ask for pricing	Voice agent responds with pricing info from product_knowledge. Same answer as chatbot.
8	Check for outdated product_knowledge	If features added/removed recently, product_knowledge updated to match. No stale info.

Known Failure Modes

• Chatbot pricing different from website — knowledge stale
• Bot claims feature not in product_knowledge — answer generation hallucinating
• product_knowledge entry missing for common question — bot invents answer
• Voice and chat answer different — inconsistent knowledge

References

• Playwright spec: e2e/delivers/consistency/product-knowledge-alignment.spec.ts
• Code files:
  • src/app/api/chatbot/stream/route.ts
  • voice-agent-livekit/session.py
  • src/lib/product-knowledge-injection.ts (if exists)

Notes

Tests that chatbot/voice agent use accurate product_knowledge from DB. Both channels must give consistent answers to pricing, features, policy questions. product_knowledge is source of truth for bot knowledge. If pricing.yaml or PRICING.md changes, product_knowledge must update first. See CLAUDE.md rule: "Verify product_knowledge after EVERY update."