Multi-Vertical Tenant Content Refactor — Pre-Phase-2 Baseline
Captured: 2026-05-14T13:08:50Z
Methodology: Chat — live HTTP POST to https://churchwiseai.com/api/chatbot/stream (20 real responses). Voice — prompt reconstructed from verticals/church/prompts.py + production DB; no live call made.
Purpose: Frozen reference for regression testing in Phases 3 and 4. Compare flag-off vs flag-on responses against this baseline.
Churches Under Test
| Field | Medhanialem (Zewdei) | Melvindale (Deb) |
|---|---|---|
| church_id | 96f5b89e-b238-4811-8d76-777f84f4241c | 9eeb9d71-a658-4958-b17f-d55112ff5e8e |
| premium_id | 92eef012-3ded-4285-b3ca-066f48492762 | 67024145-5443-4c92-86c1-ff2817501942 |
| Plan | cwa_starter_both | cwa_pro_both |
| Denomination | Evangelical | Church of God in Christ |
| Tradition key | nondenominational_evangelical | cogic |
| Pastor | Pastor Solomon Mebratu | Pastor Deb Moelker |
| Voice phone | +14144007103 | +17473897673 |
| URC rows (churches.id) | 8 | 2 |
| church_knowledge_base rows | 8 | 2 |
Chat Baseline Answers — Medhanialem Ethiopian Evangelical Church
Q1: What denomination is this church?
Source: structured_data | Time: 1894ms
Medhanialem Ethiopian Evangelical Church Milwaukee is a Evangelical church. Would you like to know more about what to expect on a visit?
Note: "a Evangelical" is grammatically awkward — should be "an Evangelical". Pre-existing issue.
Q2: Who is the pastor?
Source: structured_data | Time: 1077ms
Here's the team at Medhanialem Ethiopian Evangelical Church Milwaukee: Pastor Solomon Mebratu — Pastor
Note: Chat returns only custom_staff (1 entry). Voice injects the full 'Church Leadership and Elders' FAQ which includes 5 elder names with phone numbers. This is the church_knowledge_base gap for chat.
Q3: What time are services?
Source: structured_data | Time: 921ms
Here are Medhanialem Ethiopian Evangelical Church Milwaukee's service times: Friday: 6:30 PM - 7:30 PM (Zoom + in person) Sunday: 10:00 AM - 1:00 PM (Schedules may change — we recommend confirming with the church office before your visit.)
Correct. Matches custom_hours in DB.
Q4: Where are you located?
Source: structured_data | Time: 1431ms
Medhanialem Ethiopian Evangelical Church Milwaukee is located at 3301 S 76th St, Milwaukee, WI 53219. We'd love to see you!
Correct.
Q5: How do I give online?
Source: llm | Time: 6430ms
I'd be happy to help you support Medhanialem Ethiopian Evangelical Church Milwaukee! [...] We don't currently have online giving set up through our system, but I'd recommend reaching out to the church office directly — they can walk you through the best way to give. You can call us at +1-262-293-6676 or visit our website at https://www.eecmke.org/
Correct — giving_url is null. Fallback to phone/website is accurate.
Q6: Do you have a children's ministry?
Source: structured_data | Time: 1139ms
Children's Programs: Nursery for ages 0-3, children's church for ages 4-12. We'd love to welcome your family!
Correct. From what_to_expect.children.
Q7: I'm new — what should I expect?
Source: llm | Time: 7517ms
Welcome! I'm so glad you're thinking about visiting [...] Sunday service starts at 10:00 AM and runs about 75 minutes. Contemporary worship with a full band [...] zero pressure to give or sign up [...] free lot behind the building [...] nursery care for ages 0-3 and children's church for ages 4-12.
Correct. Synthesizes what_to_expect fields accurately.
Q8: Can I speak to the pastor?
Source: llm | Time: 2872ms
I'd love to help connect you with Pastor Solomon Mebratu! To make sure he can reach you, may I get your name and the best way to contact you — either a phone number or email?
Correct. Initiates callback capture flow.
Q9: What does your church believe?
Source: llm | Time: 4107ms
We believe the Bible is the inspired, infallible, and authoritative word of God [...] At the center of everything is Jesus Christ [...] Salvation comes by grace alone through faith alone in Christ alone [...] water baptism and the Lord's Supper as holy ordinances [...] hope of Christ's return. For our full statement of faith, you can visit eecmke.org/our-beliefs.
Correct and high quality. The response matches Medhanialem's 'Statement of Faith and Beliefs' entry in church_knowledge_base (8 core points). This answer was synthesized from URC/RAG or LLM + KB. Post-refactor this should remain substantially identical.
Q10: What do you do for visitors?
Source: llm | Time: 4847ms
Welcome! [...] Answer your questions about our church [...] Connect you with our team [...] Pray with you [...] Help you get involved [...] whether you want to talk with Pastor Solomon Mebratu, join a small group, volunteer...
Correct. Generic visitor welcome + correct pastor name.
Chat Baseline Answers — Melvindale Church of God
Q1: What denomination is this church?
Source: structured_data | Time: 1555ms
Melvindale Church of God is a Church of God in Christ church. Would you like to know more about what to expect on a visit?
Correct.
Q2: Who is the pastor?
Source: structured_data | Time: 1070ms
Here's the team at Melvindale Church of God: Pastor Deb Moelker — Senior Pastor Erin Moelker — Church Administrator
Correct. Both custom_staff entries shown.
Q3: What time are services?
Source: llm | Time: 4830ms
Our main Sunday morning worship service is at 10:00 AM in the sanctuary. We also have a Tuesday Bible study at 7:00 PM [...] we worship until the Spirit leads us (no strict time limit!).
Correct. Sunday 10AM from custom_ministries. "Spirit leads us" from service_length: "We go as long as we go". Melvindale doesn't have Sunday in custom_hours — the agent correctly inferred service time from ministries data.
Q4: Where are you located?
Source: structured_data | Time: 1234ms
Melvindale Church of God is located at 869 Melvindale Road, Melvindale, NB 45781. We'd love to see you!
Note: "NB 45781" is internally inconsistent — NB is New Brunswick (Canada) but 45781 is a US zip code format. Pre-existing data quality issue in premium_churches.address. Not a refactor issue.
Q5: How do I give online?
Source: llm | Time: 4159ms
We don't have online giving set up through the chat yet, but it's easy to reach our office! [...] call at (548) 642-7781 during office hours (Monday–Friday, 8:00 AM–4:00 PM), and Erin Moelker, our church administrator, can walk you through giving options.
Correct — giving_url is null. Fallback to office hours + admin name is accurate.
Q6: Do you have a children's ministry?
Source: structured_data | Time: 953ms
Children's Programs: Nursery for ages 0-3, children's church for ages 4-12. We'd love to welcome your family!
Correct.
Q7: I'm new — what should I expect?
Source: llm | Time: 5150ms
You'll feel welcomed [...] Come as you are [...] Parking is easy [...] The worship is alive [...] we worship until the Spirit leads us — we don't watch the clock [...] No obligation.
Correct. Captures COGIC/Pentecostal character ("until the Spirit leads us", "alive"). Pulls accurately from what_to_expect fields.
Q8: Can I speak to the pastor?
Source: llm | Time: 2529ms
I'd love to help connect you with Pastor Deb Moelker! To make sure she gets your message and reaches out to you personally, could I get your name and the best way to contact you — your phone number or email?
Correct. Initiates callback. Uses "she" correctly.
Q9: What does your church believe?
Source: llm | Time: 5527ms
At Melvindale Church of God, we're rooted in the Pentecostal tradition [...] baptism of the Holy Spirit as a distinct, empowering experience after salvation, often evidenced by speaking in tongues [...] all the spiritual gifts — prophecy, healing, miracles, tongues — are alive and active today [...] spiritual warfare, deliverance, and breakthrough seriously.
High quality. Correctly identifies COGIC as Pentecostal/Holiness tradition. Melvindale has no statement of faith in KB or beliefs JSONB — the LLM drew on COGIC tradition context. This is correct behavior for this church's data state. Post-refactor: once denominations.beliefs is populated (Phase 7), this answer should improve with curated content rather than LLM inference.
Q10: What do you do for visitors?
Source: llm | Time: 5701ms
If you're thinking about visiting [...] If you're going through something [...] connect you with our prayer team, help set up a callback with Pastor Deb Moelker [...] If you're looking to get involved [...] questions about baptism, the Holy Spirit, or anything else on your heart.
Correct and comprehensive. Correctly names pastor. Holy Spirit/baptism references appropriate for COGIC tradition.
Voice Baseline — Content Inventory
Voice answers are NOT captured as live call transcripts (cannot initiate real calls from this session). Instead, the complete component inventory of each church's voice system prompt is frozen in the JSON baseline under voice_prompt_sources.
Medhanialem Voice Prompt Components
| Component | Value |
|---|---|
| Identity | AI receptionist for Medhanialem Ethiopian Evangelical Church Milwaukee |
| Tradition context | nondenominational_evangelical (from denomination "Evangelical") |
| Pastor name | Pastor Solomon Mebratu |
| Welcome greeting | "Thank you for calling Medhanialem Ethiopian Evangelical Church. How can I help you today?" |
| Custom hours | Sunday 10AM–1PM, Friday 6:30–7:30PM (Zoom + in-person) |
| Custom staff | Pastor Solomon Mebratu — Pastor |
| Custom ministries | Youth Group (teens gr. 6-12) |
| Inline FAQs injected | 8 rows (About, Leadership+Elders with 5 elder names/phones, Ministries, Contact, Sermons, Statement of Faith, Events, Service Times) |
| URC rows available | 8 (churches.id keyed) |
| Pastor's Pulse | None (sermon_topic = null) |
| Crisis message | Default (none custom) |
| Giving | No giving_url; etransfer_email null |
| Pastor availability | None configured |
Melvindale Voice Prompt Components
| Component | Value |
|---|---|
| Identity | AI receptionist for Melvindale Church of God |
| Tradition context | cogic (from denomination "Church of God in Christ") |
| Pastor name | Pastor Deb Moelker |
| Welcome greeting | "How can I help you today?" |
| Custom hours | Mon–Fri 8AM–4PM (office hours) |
| Custom staff | Pastor Deb Moelker (Senior Pastor), Erin Moelker (Church Administrator) |
| Custom ministries | Sunday Worship 10AM, Tuesday Bible study 7pm |
| Events | Easter Service 2026-04-05 10:30AM (past — no date filtering in voice events block) |
| Inline FAQs injected | 2 rows ("Do women have to wear hats?" → No; involvement optional) |
| URC rows available | 2 (churches.id keyed) |
| Pastor's Pulse | "This week's sermon: 'Jesus Died for Us' / Theme verse: Isaiah 53" |
| Crisis message | Custom (explicit 988 message) |
| Human request message | Custom (forward to Pastor Deb by email) |
| Pastor availability | Tue/Thu 10AM–3PM, Fri 12–4PM |
| Escalation | Pastor Deb Moelker — 226-883-2030 |
Source Attribution Analysis
Chat sources observed
| Source type | Questions | Notes |
|---|---|---|
structured_data | Q1, Q2, Q3, Q4, Q6 (both churches) | Fast (under 1.5s), no LLM tokens. Uses structured data fast-path in stream/route.ts |
llm | Q5, Q7, Q8, Q9, Q10 (both churches); Q3 (Melvindale) | Slower (4–8s), full LLM synthesis. HEAR protocol active. |
The structured_data path is used for simple factual lookups (denomination, pastor list, hours, address, children's programs). More complex questions requiring synthesis, empathy, or multi-source assembly go to the LLM path.
Voice sources (reconstructed)
For both churches, voice content at call time comes from:
premium_churchesstructured fields — custom_hours, custom_staff, custom_ministries, what_to_expect, events, custom_description, beliefs (null for both)church_voice_agentsfields — pastor_name, welcome_greeting, sermon_topic, theme_verse, crisis/escalation config, giving_url (null for both)church_knowledge_baseinline FAQs — 8 rows for Medhanialem (rich content), 2 rows for Melvindale (minimal)tradition_care_context— nondenominational_evangelical for Medhanialem, cogic for Melvindaleproduct_knowledge— platform-wide FAQ (shared)- URC vector RAG — 8 rows available for Medhanialem, 2 for Melvindale (via
search_church_knowledgeRPC, churches.id keyed)
Pre-Existing Quality Issues (Not Caused by This Refactor)
These issues exist today and should NOT be counted as regressions caused by the refactor:
-
Denomination grammar — "a Evangelical church" should be "an Evangelical church". (Medhanialem Q1,
structured_datapath.) -
Melvindale address inconsistency — "Melvindale, NB 45781" — NB = New Brunswick (Canada) but zip code format is US. Data quality issue in
premium_churches.address. The founder should correct this. -
No giving_url for either church — Both Zewdei and Melvindale have
giving_url = null. The chatbot correctly falls back to phone/website, but the experience is suboptimal. The founder or church admins should add their giving URLs. -
Medhanialem chat missing elder leadership detail — The
church_knowledge_base'Church Leadership and Elders' FAQ has 5 elder names with direct phone numbers. Chat cannot see this table; voice can. This is the Phase 5 gap (chat readingchurch_knowledge_base). -
Melvindale Easter event is past — The single
eventsentry ("Easter Service 2026-04-05") is 6 weeks past. Voice doesn't date-filter events in the prompt builder (prompts.py: "_format_events — No date filtering — that is done at runtime"). Runtime filtering likely prevents this from appearing in voice responses, but the data should be cleaned. -
Melvindale
beliefsJSONB = null — The agent infers beliefs from COGIC tradition context. For a Pro tier customer, explicit beliefs would be better. The beliefs field should be populated.
What Correct Post-Refactor Responses Should Look Like
When the refactor is complete (Phases 2–5), the following improvements should be detectable vs this baseline:
Phase 2 (additive RPC only): Zero change to response content. This baseline should be bit-for-bit identical to post-Phase-2 responses when CHAT_USE_GENERIC_TENANT_KNOWLEDGE_RPC=false and VOICE_USE_GENERIC_TENANT_KNOWLEDGE_RPC=false.
Phase 3 (voice migrates to generic RPC): Voice responses for Zewdei + Melvindale should remain identical to this baseline's voice_prompt_sources component inventory — same inline FAQs, same structured fields, same tradition context. Any difference is a regression.
Phase 4 (chat migrates to generic RPC): Chat responses should be identical to this baseline (flag-off). With flag on, if church_knowledge_base is added as a chat source (Phase 5), then:
- Medhanialem Q2 ("Who is the pastor?") should expand from 1 staff entry to include 5 elders with phone numbers.
- Medhanialem Q9 ("What does your church believe?") should cite the Statement of Faith FAQ verbatim.
- Both church answers should be richer on ministry details.
Phase 7 (beliefs/denomination data wired): Melvindale Q9 ("What does your church believe?") should cite curated COGIC theological content from denominations.beliefs rather than LLM inference.
Regression Test Assertions (for Phases 3 and 4)
For each of the 10 baseline questions, post-refactor assertions:
| Q | Medhanialem assertion | Melvindale assertion |
|---|---|---|
| 1 | Response contains "Evangelical" | Response contains "Church of God in Christ" |
| 2 | Response contains "Solomon Mebratu" | Response contains "Deb Moelker" |
| 3 | Response contains "10:00 AM" and "Sunday" | Response contains "10:00 AM" |
| 4 | Response contains "3301 S 76th St" and "Milwaukee" | Response contains "869 Melvindale" |
| 5 | Response contains "eecmke.org" or phone number | Response contains "(548) 642-7781" or "Erin Moelker" |
| 6 | Response contains "children" | Response contains "children" |
| 7 | Response contains "greeters" OR "75 minutes" | Response contains "greeters" OR "Spirit" |
| 8 | Response contains "Solomon Mebratu" and callback initiation | Response contains "Deb Moelker" and callback initiation |
| 9 | Response contains "Bible" and "Jesus" and "salvation" | Response contains "Holy Spirit" OR "Pentecostal" |
| 10 | Response contains church name and welcoming language | Response contains church name and welcoming language |
Hard failures (any of these in post-refactor = regression):
- Wrong pastor name (e.g., "Pastor Deb" for Medhanialem, or "Solomon" for Melvindale)
- Wrong address
- Wrong denomination
- Empty response (0 chars)
- HTTP non-200 status