Skip to main content

Knowledge > Runbooks > Deployment > Rollback

Rollback a Bad Deployment

Revert a production deployment that is causing errors, crashes, or wrong behavior.

Prerequisites​

  • vercel CLI logged in (vercel whoami → churchwiseai-5386)
  • GitHub write access (for code reverts)
  • WSL access (for voice agent rollback only)

Web App Rollback (churchwiseai-web / pewsearch / ITW)​

Option A — Vercel Instant Rollback (Fastest)​

Vercel keeps all previous deployments. Promoting a previous deploy to production takes ~10 seconds.

  1. Find the last known-good deployment ID

    vercel ls --project <project-name>

    Where <project-name> is one of: churchwiseai-web, pewsearch-web, sermon-illustrations. Look for the deployment before your broken one. Copy its deployment URL or ID.

  2. Promote the previous deployment to production In the Vercel Dashboard:

    • Go to the project → Deployments tab
    • Find the last known-good deployment
    • Click "..." → "Promote to Production"
    • Confirm

    Or, if the Vercel CLI supports it in your version:

    vercel promote <deployment-url> --scope churchwiseai-5386

  3. Verify production is restored

    • Visit the production URL and confirm the site is working
    • Check vercel logs --tail --project <project-name> for any remaining errors

Option B — Git Revert (Preferred for Code Tracking)​

  1. Identify the bad commit

    cd /c/dev/<codebase>
    git log --oneline -10

  2. Revert the bad commit

    git revert <bad-commit-sha>

    This creates a new commit that undoes the change — safe and preserves history.

  3. Push to the deploy branch

    git push

    Vercel picks up the push and deploys the reverted code.

  4. Verify production is restored

    vercel logs --tail --project <project-name>

Voice Agent Rollback​

The LiveKit voice agent does not have a dashboard rollback. You must revert the code and redeploy via the LiveKit CLI.

  1. Revert the code change

    cd /c/dev/churchwiseai-web
    git log --oneline -5
    git revert HEAD
    git push origin main

  2. Redeploy the voice agent from the voice-agent-livekit/ directory:

    C:\dev\lk.exe agent deploy --project cwa-voice --silent

  3. Test by calling a configured church number and verifying normal behavior.

Database Migration Rollback​

Database migrations are the most dangerous rollback scenario. There is no automatic undo.

  1. Assess the impact first. Determine exactly which rows or schema changes were made.
  2. Get explicit founder approval before running any rollback query.
  3. Write the inverse migration manually:
    • If you added a column: ALTER TABLE x DROP COLUMN y;
    • If you updated rows: restore from the values you captured before the change
    • If you inserted rows: DELETE FROM x WHERE id IN (...); — only with founder approval
  4. NEVER delete rows from unified_rag_content — this table has 327K irreplaceable records.
  5. NEVER run a rollback that affects churches in bulk without founder approval.

Decision: When to Rollback vs. Forward-Fix​

SituationAction
Site is down / 500 errors on all pagesRollback immediately (Option A)
One page is brokenForward-fix — push a patch
Build failed on deploy branchFix and push; Vercel won't deploy a failed build
Voice agent crashing on all callsRollback via WSL immediately
DB migration caused data issuesStop, assess, get founder approval

See Also​