Skip to main content

Knowledge > Runbooks > Technical Ops > Respond to a Security Incident

Respond to a Security Incident

Contain, assess, and recover from a security incident such as an exposed API key, unauthorized database access, or credential compromise.

Prerequisites​

  • Access to all CLIs: vercel, stripe, Supabase dashboard
  • STRIPE_LIVE_SECRET_KEY from knowledge/.env for live mode operations
  • Ability to rotate keys in each service's dashboard

Incident Types​

TypeImmediate Action
Exposed API key (committed to git, logged, etc.)Rotate the key immediately — do NOT wait
Unauthorized admin sessionInvalidate sessions in church_admin_sessions
Suspected SQL injectionCheck Supabase logs, assess data exposure
Exposed PII (church or parishioner data)Assess scope, notify founder, prepare notification
Vercel env var exposed in logs/errorsRotate the exposed secret

Steps​

Phase 1: Contain (Do Immediately)​

  1. Identify which secret or credential is exposed

    Common secrets across the portfolio:

    • SUPABASE_SERVICE_ROLE_KEY — widest access, rotate first
    • STRIPE_SECRET_KEY (test or live)
    • STRIPE_WEBHOOK_SECRET
    • ANTHROPIC_API_KEY
    • OPENAI_API_KEY
    • RESEND_API_KEY
    • TWILIO_AUTH_TOKEN
    • NEXT_PUBLIC_SUPABASE_ANON_KEY — lower risk (public), but still rotate

  2. Rotate the exposed key immediately

    Supabase service role key:

    • Supabase Dashboard → Project Settings → API → Regenerate service role key
    • Update Vercel: echo "new-key" | vercel env add SUPABASE_SERVICE_ROLE_KEY production
    • Repeat for all three Vercel projects that use it

    Stripe secret key:

    • Stripe Dashboard → Developers → API keys → Roll key
    • Update Vercel: echo "sk_live_..." | vercel env add STRIPE_SECRET_KEY production

    Webhook secret:

    • Stripe Dashboard → Developers → Webhooks → endpoint → Reveal signing secret → Roll
    • Update Vercel: echo "whsec_..." | vercel env add STRIPE_WEBHOOK_SECRET production

    Other API keys:

    • Regenerate in the respective service dashboard
    • Update in Vercel immediately after

  3. Redeploy affected projects to pick up rotated keys

    cd /c/dev/churchwiseai-web && vercel --prod
    cd /c/dev/pewsearch/web && vercel --prod
    cd /c/dev/sermon-illustrations && vercel --prod

Phase 2: Assess Scope​

  1. Check for unauthorized admin sessions

    SELECT id, church_id, created_at, last_active, ip_address
    FROM church_admin_sessions
    WHERE created_at > now() - interval '7 days'
    ORDER BY created_at DESC;

    If any sessions appear unauthorized (unfamiliar IP, unexpected church_id):

    -- Invalidate specific suspicious session (get founder approval first)
    DELETE FROM church_admin_sessions WHERE id = 'suspicious-session-id';

  2. Check Supabase logs for suspicious queries

    Supabase Dashboard → Logs → API logs → filter by timeframe and look for:

    • Unexpected high volume from a single IP
    • Queries targeting sensitive tables (premium_churches, church_admin_sessions)
    • SQL patterns that suggest injection attempts

  3. Assess data exposure

    If unauthorized access may have read sensitive data, identify which tables were accessible with the exposed key:

    • SUPABASE_SERVICE_ROLE_KEY → all tables (RLS bypassed)
    • SUPABASE_ANON_KEY → tables with public RLS policies only
    • Application API key → only what the API routes expose

Phase 3: Remediate and Notify​

  1. Fix the root cause

    If a key was committed to git:

    # Remove from git history if recently committed
    git log --oneline -10  # find the commit
    # Inform founder — git history rewriting on shared branches requires coordination

    If a key appeared in logs:

    • Find the code path that logged it
    • Fix the logging statement
    • Deploy the fix

  2. Notify affected customers if PII was exposed

    If any parishioner PII (prayer requests, contact info, visitor data) may have been accessed:

    • Add an item to FOUNDER_ACTIONS.md with severity P0
    • Draft customer notification email for founder review
    • Do NOT send notifications without founder approval

  3. Document in DECISION_LOG.md

    - 2026-03-25: Security incident — [type]. Rotated [key names]. [Brief scope assessment]. [Customers affected: Y/N].

Verification​

  • New keys are active in all Vercel projects (test with a lightweight API call)
  • No new unauthorized sessions appear in church_admin_sessions
  • ops_error_reports shows no auth-related errors from the rotated keys
  • Old keys are fully revoked in service dashboards (test that old key returns 401)

See Also​