church-website/infra/runbooks/production-migration.md

# Runbook: Migrate to the production server (hl-mutter-teresa-chemnitz.de)

One-off procedure to provision the new production VPS (4 vCPU / 8 GB, no Forgejo),
copy the staging data (DB + uploads) onto it, and cut DNS over from the old
server (178.104.35.59).

Shell variables used throughout (run on your dev machine):

```bash
OLD=root@178.104.35.59
NEW=root@217.154.211.139
```

---

## Phase 0 — Prep (days before)

1. Lower the DNS TTL for `hl-mutter-teresa-chemnitz.de` and `www.hl-mutter-teresa-chemnitz.de` to 300s.
2. Set the production secrets in the vault:
   ```bash
   cd infra/ansible
   ansible-vault edit inventory/group_vars/all/vault.yml --vault-password-file ~/.config/chemnitz-vault-pass
   ```
   - `vault_db_password`: new random value (e.g. `openssl rand -hex 24`)
   - `vault_payload_secret`: **copy the value of `vault_payload_secret_staging`**.
     The migrated database may contain fields encrypted with the staging secret
     (Payload API-key fields); a different secret would silently break them.
3. Fill the production VPS IP into `inventory/production.yml` (`ansible_host`).
4. Make sure your SSH public key is on the new VPS root account (`ssh $NEW true` works without password).

## Phase 1 — Provision + initial deploy

```bash
cd infra/ansible
ansible-playbook playbooks/setup.yml -i inventory/production.yml --vault-password-file ~/.config/chemnitz-vault-pass
```

- The **first run fails at "Clone or update repository"**: the server's freshly
  generated SSH key isn't known to Forgejo yet. Copy the public key printed by
  the "Show SSH public key" task and add it in Forgejo
  (git.skick.app → church-website repo → Settings → Deploy Keys, **read-only**).
- Re-run the same command; it now completes clone → build → migrate → start `app-production`.

Sanity checks:

```bash
ssh $NEW "curl -sI http://127.0.0.1:3001 | head -1"          # HTTP 200/30x
ssh $NEW "docker exec postgres psql -U postgres -c 'SHOW shared_buffers;'"   # 256MB
ssh $NEW "swapon --show"                                     # 4G /swapfile
```

Caddy will log ACME errors for the domain until DNS moves — expected, ignore.

## Phase 2 — Data sync (repeatable; rehearse once before cutover)

### 2a. Database

```bash
# Dump staging on the old server
ssh $OLD "docker exec postgres pg_dump -U postgres --no-owner --no-acl church_website_staging | gzip" > staging.sql.gz

# Recreate the production DB empty
ssh $NEW "docker stop app-production"
ssh $NEW "docker exec postgres psql -U postgres -c \"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'church_website' AND pid <> pg_backend_pid();\""
ssh $NEW "docker exec postgres psql -U postgres -c 'DROP DATABASE IF EXISTS church_website;'"
ssh $NEW "docker exec postgres psql -U postgres -c 'CREATE DATABASE church_website OWNER church_website;'"
ssh $NEW "docker exec postgres psql -U postgres -d church_website -c 'CREATE EXTENSION IF NOT EXISTS postgis;'"

# Restore (spatial_ref_sys errors are harmless — that table is owned by postgis)
gunzip -c staging.sql.gz | ssh $NEW "docker exec -i postgres psql -U postgres -d church_website"
```

The drop/recreate is required even though Phase 1 already created a schema:
the dump's `payload_migrations` table must be authoritative, not merged into a
freshly migrated schema.

Reassign ownership to the app user (same block as `copy-staging-to-test.yml`):

```bash
ssh $NEW "docker exec -i postgres psql -U postgres -d church_website" <<'SQL'
DO $$
DECLARE
  r RECORD;
BEGIN
  FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = 'public' LOOP
    EXECUTE 'ALTER TABLE public.' || quote_ident(r.tablename) || ' OWNER TO church_website';
  END LOOP;
  FOR r IN SELECT sequencename FROM pg_sequences WHERE schemaname = 'public' LOOP
    EXECUTE 'ALTER SEQUENCE public.' || quote_ident(r.sequencename) || ' OWNER TO church_website';
  END LOOP;
  FOR r IN SELECT typname FROM pg_type t
    WHERE t.typnamespace = 'public'::regnamespace
      AND t.typtype = 'e'
      AND NOT EXISTS (SELECT 1 FROM pg_depend d WHERE d.objid = t.oid AND d.deptype = 'e')
  LOOP
    EXECUTE 'ALTER TYPE public.' || quote_ident(r.typname) || ' OWNER TO church_website';
  END LOOP;
END
$$;
SQL
```

### 2b. Upload volumes

```bash
ssh $OLD "docker run --rm -v uploads-staging-media:/src:ro alpine tar -C /src -czf - ." \
  | ssh $NEW "docker run --rm -i -v uploads-production-media:/dst alpine sh -c 'rm -rf /dst/* && tar -xzf - -C /dst'"

ssh $OLD "docker run --rm -v uploads-staging-documents:/src:ro alpine tar -C /src -czf - ." \
  | ssh $NEW "docker run --rm -i -v uploads-production-documents:/dst alpine sh -c 'rm -rf /dst/* && tar -xzf - -C /dst'"
```

### 2c. Restart + verify

Recreate the container rather than just starting it — Next.js caches rendered
pages in the container's filesystem, and a plain `docker start` serves stale
pages (e.g. a cached 404 for `/`) from before the restore:

```bash
ssh $NEW "docker rm -f app-production && docker run -d --name app-production --restart unless-stopped --network church-website-net --env-file /opt/church-website/envs/production/.env -v uploads-production-media:/app/media -v uploads-production-documents:/app/documents -p 127.0.0.1:3001:3000 church-website:production"
ssh $NEW "docker exec -u 0 app-production chown -R 1001:1001 /app/media /app/documents"
ssh $NEW "curl -sI http://127.0.0.1:3001 | head -1"          # 200
ssh $NEW "curl -sI http://127.0.0.1:3001/admin | head -1"    # 200/30x
```

Optionally browse the site through an SSH tunnel before cutover:
`ssh -L 3001:127.0.0.1:3001 $NEW`, then open http://localhost:3001 — content,
images, and admin login (staging credentials) must all work.

## Phase 3 — Cutover

1. **Content freeze**: ask the editors to stop editing (~30 min window).
2. Re-run all of Phase 2 (final sync).
3. Switch the DNS A records for apex + `www` to the production IP.
4. Kick Caddy so it retries ACME immediately, and watch until certs issue:
   ```bash
   ssh $NEW "systemctl restart caddy && journalctl -u caddy -f"
   ```
5. Verify from outside:
   ```bash
   curl -sI https://hl-mutter-teresa-chemnitz.de | head -1        # 200
   curl -sI https://www.hl-mutter-teresa-chemnitz.de | head -3    # 308 → apex
   ```
   Spot-check pages, images, admin login, and (if feasible) a contact form email.

Expected downtime ≈ DNS TTL (≤5 min) plus seconds of cert issuance. Visitors
still resolving the old IP keep getting the identical just-synced content from
the old server, so the window is soft.

## Phase 4 — Post-cutover

1. Stop the old server from answering/renewing the domain — the countdown
   config is already removed from the repo, so re-rendering its Caddyfile drops
   the domain:
   ```bash
   cd infra/ansible
   ansible-playbook playbooks/setup.yml -i inventory/test.yml --vault-password-file ~/.config/chemnitz-vault-pass
   ssh $OLD "rm -rf /var/www/countdown"
   ```
   > Do **not** run this before the DNS switch — it takes the live domain off
   > the old server.
2. Restore the DNS TTL to its normal value.
3. Confirm the deployed build has the correct canonical URL: page source /
   sitemap should reference `hl-mutter-teresa-chemnitz.de`, not `skick.app`.

## Optional — minimal daily backup (recommended)

On the production server, as root (`crontab -e`):

```cron
30 3 * * * mkdir -p /opt/backups && docker exec postgres pg_dump -U postgres -Fc church_website > /opt/backups/db-$(date +\%F).dump && docker run --rm -v uploads-production-media:/m:ro -v uploads-production-documents:/d:ro -v /opt/backups:/b alpine tar -czf /b/uploads-$(date +\%F).tar.gz -C / m d && find /opt/backups -mtime +7 -delete
```

Keeps 7 days of DB dumps + upload tarballs in `/opt/backups`. Off-site copies
(e.g. rsync to the old server or your machine) are a possible next step.

## Ongoing: deploying to production

```bash
cd infra/ansible
ansible-playbook playbooks/deploy-production.yml -i inventory/production.yml --vault-password-file ~/.config/chemnitz-vault-pass
```

Production tracks the `staging` branch on git.skick.app.