Skip to main content

Knowledge > Runbooks > Chatbot Ops > Debug Chatbot Issues

Debug Chatbot Issues

Diagnose and fix chatbot problems: wrong or irrelevant responses, tools not working, prayer requests not saving, LLM errors, or the widget not loading.

Prerequisites​

  • Vercel CLI authenticated (vercel logs)
  • Supabase MCP or direct DB access
  • Access to churchwiseai-web codebase for code-level debugging

Reproduce the Issue First​

Before diving into logs, reproduce the exact failure:

# Test the chat API directly
curl -X POST https://churchwiseai.com/api/chatbot/stream \
  -H "Content-Type: application/json" \
  -d '{"message": "[failing message]", "churchId": "[uuid]", "conversationId": "[new-uuid]"}'

Or test via the admin dashboard preview at https://churchwiseai.com/admin/[token]/.

Steps​

Step 1: Check Vercel logs for server errors​

vercel logs --project churchwiseai-web --tail

Look for:

  • 500 errors from /api/chatbot/stream
  • ANTHROPIC_API_KEY or OPENAI_API_KEY errors (LLM routing failures)
  • SUPABASE_SERVICE_ROLE_KEY errors
  • Timeouts (Vercel functions have a 30s limit)

Step 2: Verify church config in organization_settings​

SELECT
  church_id, enabled, tier, theological_lens_id,
  greeting_message, chatbot_name, pro_website_mode
FROM organization_settings
WHERE church_id = '[uuid]';
  • enabled = false → chatbot is disabled. See disable-chatbot.md.
  • tier mismatch → tools may be restricted. Re-check premium_churches.plan.

Step 3: Verify knowledge base content exists​

SELECT COUNT(*) as kb_entries
FROM unified_rag_content
WHERE church_id = '[uuid]'
  AND category = 'church_kb'
  AND is_public = true;

If count is 0, the chatbot has no church-specific KB to draw from. Add content via update-kb-content.md.

Step 4: Check the LLM routing chain​

The chat route uses this fallback chain:

  1. Claude Haiku 4.5 (primary)
  2. Claude Sonnet 4.6 (if escalate: true returned)
  3. gpt-4o-mini (if Anthropic returns a 5xx error)

If all three are failing, check:

  • ANTHROPIC_API_KEY env var in Vercel (vercel env ls)
  • OPENAI_API_KEY env var in Vercel
  • Anthropic service status: status.anthropic.com

Step 5: Check tool failures (prayer requests, callbacks, visitor capture)​

Tools write to shared tables. Test RLS:

-- Check that service role can see recent inserts
SELECT id, church_id, created_at, request_text
FROM voice_prayer_requests
WHERE church_id = '[uuid]'
ORDER BY created_at DESC
LIMIT 3;

If tool inserts are failing silently, the most common cause is SUPABASE_SERVICE_ROLE_KEY misconfigured in Vercel. Verify:

vercel env ls --project churchwiseai-web | grep SUPABASE

Row Level Security must allow service role inserts on voice_prayer_requests, voice_callback_requests, and voice_visitor_contacts. These tables should have NO FORCE ROW LEVEL SECURITY for the service role.

Step 6: Check ops_error_reports for recent chatbot errors​

SELECT severity, component, summary, created_at
FROM ops_error_reports
WHERE component LIKE 'chatbot%'
ORDER BY created_at DESC
LIMIT 10;

Step 7: Check Pro Website mode restrictions​

Pro Website chatbot has a simplified toolset (Q&A + prayer request only):

SELECT pro_website_mode FROM organization_settings WHERE church_id = '[uuid]';

If pro_website_mode = true, RAG search, KB queries, and advanced tools are disabled by design.

Common Issues and Fixes​

SymptomMost Likely CauseFix
Widget not loadingJS error or wrong embed URLCheck browser console, verify widget URL
Generic responsesNo KB content for churchAdd KB via update-kb-content.md
Prayer request not savingRLS or service role key issueCheck SUPABASE_SERVICE_ROLE_KEY
500 error on all messagesLLM API key invalidCheck ANTHROPIC_API_KEY in Vercel
Timeout errorsLong RAG queryCheck pgvector index on unified_rag_content

Verification​

  • Test message returns a relevant, church-specific response.
  • Prayer request test saves a row in voice_prayer_requests.
  • No 5xx errors in Vercel logs during test.

See Also​