church-website/infra/runbooks/production-migration.md

Runbook: Migrate to the production server (hl-mutter-teresa-chemnitz.de)

One-off procedure to provision the new production VPS (4 vCPU / 8 GB, no Forgejo), copy the staging data (DB + uploads) onto it, and cut DNS over from the old server (178.104.35.59).

Shell variables used throughout (run on your dev machine):

OLD=root@178.104.35.59
NEW=root@217.154.211.139

---

Phase 0 — Prep (days before)

1. Lower the DNS TTL for hl-mutter-teresa-chemnitz.de and www.hl-mutter-teresa-chemnitz.de to 300s.
2. Set the production secrets in the vault:

   cd infra/ansible
   ansible-vault edit inventory/group_vars/all/vault.yml --vault-password-file ~/.config/chemnitz-vault-pass
   

   • vault_db_password: new random value (e.g. openssl rand -hex 24)
   • vault_payload_secret: copy the value of vault_payload_secret_staging. The migrated database may contain fields encrypted with the staging secret (Payload API-key fields); a different secret would silently break them.
3. Fill the production VPS IP into inventory/production.yml (ansible_host).
4. Make sure your SSH public key is on the new VPS root account (ssh $NEW true works without password).

Phase 1 — Provision + initial deploy

cd infra/ansible
ansible-playbook playbooks/setup.yml -i inventory/production.yml --vault-password-file ~/.config/chemnitz-vault-pass

• The first run fails at "Clone or update repository": the server's freshly generated SSH key isn't known to Forgejo yet. Copy the public key printed by the "Show SSH public key" task and add it in Forgejo (git.skick.app → church-website repo → Settings → Deploy Keys, read-only).
• Re-run the same command; it now completes clone → build → migrate → start app-production.

Sanity checks:

ssh $NEW "curl -sI http://127.0.0.1:3001 | head -1"          # HTTP 200/30x
ssh $NEW "docker exec postgres psql -U postgres -c 'SHOW shared_buffers;'"   # 512MB
ssh $NEW "swapon --show"                                     # 4G /swapfile

Caddy will log ACME errors for the domain until DNS moves — expected, ignore.

Phase 2 — Data sync (repeatable; rehearse once before cutover)

2a. Database

# Dump staging on the old server
ssh $OLD "docker exec postgres pg_dump -U postgres --no-owner --no-acl church_website_staging | gzip" > staging.sql.gz

# Recreate the production DB empty
ssh $NEW "docker stop app-production"
ssh $NEW "docker exec postgres psql -U postgres -c \"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'church_website' AND pid <> pg_backend_pid();\""
ssh $NEW "docker exec postgres psql -U postgres -c 'DROP DATABASE IF EXISTS church_website;'"
ssh $NEW "docker exec postgres psql -U postgres -c 'CREATE DATABASE church_website OWNER church_website;'"
ssh $NEW "docker exec postgres psql -U postgres -d church_website -c 'CREATE EXTENSION IF NOT EXISTS postgis;'"

# Restore (spatial_ref_sys errors are harmless — that table is owned by postgis)
gunzip -c staging.sql.gz | ssh $NEW "docker exec -i postgres psql -U postgres -d church_website"

The drop/recreate is required even though Phase 1 already created a schema: the dump's payload_migrations table must be authoritative, not merged into a freshly migrated schema.

Reassign ownership to the app user (same block as copy-staging-to-test.yml):

ssh $NEW "docker exec -i postgres psql -U postgres -d church_website" <<'SQL'
DO $$
DECLARE
  r RECORD;
BEGIN
  FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = 'public' LOOP
    EXECUTE 'ALTER TABLE public.' || quote_ident(r.tablename) || ' OWNER TO church_website';
  END LOOP;
  FOR r IN SELECT sequencename FROM pg_sequences WHERE schemaname = 'public' LOOP
    EXECUTE 'ALTER SEQUENCE public.' || quote_ident(r.sequencename) || ' OWNER TO church_website';
  END LOOP;
  FOR r IN SELECT typname FROM pg_type t
    WHERE t.typnamespace = 'public'::regnamespace
      AND t.typtype = 'e'
      AND NOT EXISTS (SELECT 1 FROM pg_depend d WHERE d.objid = t.oid AND d.deptype = 'e')
  LOOP
    EXECUTE 'ALTER TYPE public.' || quote_ident(r.typname) || ' OWNER TO church_website';
  END LOOP;
END
$$;
SQL

2b. Upload volumes

ssh $OLD "docker run --rm -v uploads-staging-media:/src:ro alpine tar -C /src -czf - ." \
  | ssh $NEW "docker run --rm -i -v uploads-production-media:/dst alpine sh -c 'rm -rf /dst/* && tar -xzf - -C /dst'"

ssh $OLD "docker run --rm -v uploads-staging-documents:/src:ro alpine tar -C /src -czf - ." \
  | ssh $NEW "docker run --rm -i -v uploads-production-documents:/dst alpine sh -c 'rm -rf /dst/* && tar -xzf - -C /dst'"

2c. Restart + verify

ssh $NEW "docker start app-production"
ssh $NEW "docker exec -u 0 app-production chown -R 1001:1001 /app/media /app/documents"
ssh $NEW "curl -sI http://127.0.0.1:3001 | head -1"          # 200
ssh $NEW "curl -sI http://127.0.0.1:3001/admin | head -1"    # 200/30x

Optionally browse the site through an SSH tunnel before cutover: ssh -L 3001:127.0.0.1:3001 $NEW, then open http://localhost:3001 — content, images, and admin login (staging credentials) must all work.

Phase 3 — Cutover

1. Content freeze: ask the editors to stop editing (~30 min window).
2. Re-run all of Phase 2 (final sync).
3. Switch the DNS A records for apex + www to the production IP.
4. Kick Caddy so it retries ACME immediately, and watch until certs issue:

   ssh $NEW "systemctl restart caddy && journalctl -u caddy -f"
   

5. Verify from outside:

   curl -sI https://hl-mutter-teresa-chemnitz.de | head -1        # 200
   curl -sI https://www.hl-mutter-teresa-chemnitz.de | head -3    # 308 → apex
   

   Spot-check pages, images, admin login, and (if feasible) a contact form email.

Expected downtime ≈ DNS TTL (≤5 min) plus seconds of cert issuance. Visitors still resolving the old IP keep getting the identical just-synced content from the old server, so the window is soft.

Phase 4 — Post-cutover

1. Stop the old server from answering/renewing the domain — the countdown config is already removed from the repo, so re-rendering its Caddyfile drops the domain:

   cd infra/ansible
   ansible-playbook playbooks/setup.yml -i inventory/test.yml --vault-password-file ~/.config/chemnitz-vault-pass
   ssh $OLD "rm -rf /var/www/countdown"
   

   Do not run this before the DNS switch — it takes the live domain off the old server.

2. Restore the DNS TTL to its normal value.
3. Confirm the deployed build has the correct canonical URL: page source / sitemap should reference hl-mutter-teresa-chemnitz.de, not skick.app.

Optional — minimal daily backup (recommended)

On the production server, as root (crontab -e):

30 3 * * * mkdir -p /opt/backups && docker exec postgres pg_dump -U postgres -Fc church_website > /opt/backups/db-$(date +\%F).dump && docker run --rm -v uploads-production-media:/m:ro -v uploads-production-documents:/d:ro -v /opt/backups:/b alpine tar -czf /b/uploads-$(date +\%F).tar.gz -C / m d && find /opt/backups -mtime +7 -delete

Keeps 7 days of DB dumps + upload tarballs in /opt/backups. Off-site copies (e.g. rsync to the old server or your machine) are a possible next step.

Ongoing: deploying to production

cd infra/ansible
ansible-playbook playbooks/deploy-production.yml -i inventory/production.yml --vault-password-file ~/.config/chemnitz-vault-pass

Production tracks the staging branch on git.skick.app.