180 lines
7.4 KiB
Markdown
180 lines
7.4 KiB
Markdown
# Runbook: Migrate to the production server (hl-mutter-teresa-chemnitz.de)
|
|
|
|
One-off procedure to provision the new production VPS (4 vCPU / 8 GB, no Forgejo),
|
|
copy the staging data (DB + uploads) onto it, and cut DNS over from the old
|
|
server (178.104.35.59).
|
|
|
|
Shell variables used throughout (run on your dev machine):
|
|
|
|
```bash
|
|
OLD=root@178.104.35.59
|
|
NEW=root@217.154.211.139
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 0 — Prep (days before)
|
|
|
|
1. Lower the DNS TTL for `hl-mutter-teresa-chemnitz.de` and `www.hl-mutter-teresa-chemnitz.de` to 300s.
|
|
2. Set the production secrets in the vault:
|
|
```bash
|
|
cd infra/ansible
|
|
ansible-vault edit inventory/group_vars/all/vault.yml --vault-password-file ~/.config/chemnitz-vault-pass
|
|
```
|
|
- `vault_db_password`: new random value (e.g. `openssl rand -hex 24`)
|
|
- `vault_payload_secret`: **copy the value of `vault_payload_secret_staging`**.
|
|
The migrated database may contain fields encrypted with the staging secret
|
|
(Payload API-key fields); a different secret would silently break them.
|
|
3. Fill the production VPS IP into `inventory/production.yml` (`ansible_host`).
|
|
4. Make sure your SSH public key is on the new VPS root account (`ssh $NEW true` works without password).
|
|
|
|
## Phase 1 — Provision + initial deploy
|
|
|
|
```bash
|
|
cd infra/ansible
|
|
ansible-playbook playbooks/setup.yml -i inventory/production.yml --vault-password-file ~/.config/chemnitz-vault-pass
|
|
```
|
|
|
|
- The **first run fails at "Clone or update repository"**: the server's freshly
|
|
generated SSH key isn't known to Forgejo yet. Copy the public key printed by
|
|
the "Show SSH public key" task and add it in Forgejo
|
|
(git.skick.app → church-website repo → Settings → Deploy Keys, **read-only**).
|
|
- Re-run the same command; it now completes clone → build → migrate → start `app-production`.
|
|
|
|
Sanity checks:
|
|
|
|
```bash
|
|
ssh $NEW "curl -sI http://127.0.0.1:3001 | head -1" # HTTP 200/30x
|
|
ssh $NEW "docker exec postgres psql -U postgres -c 'SHOW shared_buffers;'" # 512MB
|
|
ssh $NEW "swapon --show" # 4G /swapfile
|
|
```
|
|
|
|
Caddy will log ACME errors for the domain until DNS moves — expected, ignore.
|
|
|
|
## Phase 2 — Data sync (repeatable; rehearse once before cutover)
|
|
|
|
### 2a. Database
|
|
|
|
```bash
|
|
# Dump staging on the old server
|
|
ssh $OLD "docker exec postgres pg_dump -U postgres --no-owner --no-acl church_website_staging | gzip" > staging.sql.gz
|
|
|
|
# Recreate the production DB empty
|
|
ssh $NEW "docker stop app-production"
|
|
ssh $NEW "docker exec postgres psql -U postgres -c \"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'church_website' AND pid <> pg_backend_pid();\""
|
|
ssh $NEW "docker exec postgres psql -U postgres -c 'DROP DATABASE IF EXISTS church_website;'"
|
|
ssh $NEW "docker exec postgres psql -U postgres -c 'CREATE DATABASE church_website OWNER church_website;'"
|
|
ssh $NEW "docker exec postgres psql -U postgres -d church_website -c 'CREATE EXTENSION IF NOT EXISTS postgis;'"
|
|
|
|
# Restore (spatial_ref_sys errors are harmless — that table is owned by postgis)
|
|
gunzip -c staging.sql.gz | ssh $NEW "docker exec -i postgres psql -U postgres -d church_website"
|
|
```
|
|
|
|
The drop/recreate is required even though Phase 1 already created a schema:
|
|
the dump's `payload_migrations` table must be authoritative, not merged into a
|
|
freshly migrated schema.
|
|
|
|
Reassign ownership to the app user (same block as `copy-staging-to-test.yml`):
|
|
|
|
```bash
|
|
ssh $NEW "docker exec -i postgres psql -U postgres -d church_website" <<'SQL'
|
|
DO $$
|
|
DECLARE
|
|
r RECORD;
|
|
BEGIN
|
|
FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = 'public' LOOP
|
|
EXECUTE 'ALTER TABLE public.' || quote_ident(r.tablename) || ' OWNER TO church_website';
|
|
END LOOP;
|
|
FOR r IN SELECT sequencename FROM pg_sequences WHERE schemaname = 'public' LOOP
|
|
EXECUTE 'ALTER SEQUENCE public.' || quote_ident(r.sequencename) || ' OWNER TO church_website';
|
|
END LOOP;
|
|
FOR r IN SELECT typname FROM pg_type t
|
|
WHERE t.typnamespace = 'public'::regnamespace
|
|
AND t.typtype = 'e'
|
|
AND NOT EXISTS (SELECT 1 FROM pg_depend d WHERE d.objid = t.oid AND d.deptype = 'e')
|
|
LOOP
|
|
EXECUTE 'ALTER TYPE public.' || quote_ident(r.typname) || ' OWNER TO church_website';
|
|
END LOOP;
|
|
END
|
|
$$;
|
|
SQL
|
|
```
|
|
|
|
### 2b. Upload volumes
|
|
|
|
```bash
|
|
ssh $OLD "docker run --rm -v uploads-staging-media:/src:ro alpine tar -C /src -czf - ." \
|
|
| ssh $NEW "docker run --rm -i -v uploads-production-media:/dst alpine sh -c 'rm -rf /dst/* && tar -xzf - -C /dst'"
|
|
|
|
ssh $OLD "docker run --rm -v uploads-staging-documents:/src:ro alpine tar -C /src -czf - ." \
|
|
| ssh $NEW "docker run --rm -i -v uploads-production-documents:/dst alpine sh -c 'rm -rf /dst/* && tar -xzf - -C /dst'"
|
|
```
|
|
|
|
### 2c. Restart + verify
|
|
|
|
```bash
|
|
ssh $NEW "docker start app-production"
|
|
ssh $NEW "docker exec -u 0 app-production chown -R 1001:1001 /app/media /app/documents"
|
|
ssh $NEW "curl -sI http://127.0.0.1:3001 | head -1" # 200
|
|
ssh $NEW "curl -sI http://127.0.0.1:3001/admin | head -1" # 200/30x
|
|
```
|
|
|
|
Optionally browse the site through an SSH tunnel before cutover:
|
|
`ssh -L 3001:127.0.0.1:3001 $NEW`, then open http://localhost:3001 — content,
|
|
images, and admin login (staging credentials) must all work.
|
|
|
|
## Phase 3 — Cutover
|
|
|
|
1. **Content freeze**: ask the editors to stop editing (~30 min window).
|
|
2. Re-run all of Phase 2 (final sync).
|
|
3. Switch the DNS A records for apex + `www` to the production IP.
|
|
4. Kick Caddy so it retries ACME immediately, and watch until certs issue:
|
|
```bash
|
|
ssh $NEW "systemctl restart caddy && journalctl -u caddy -f"
|
|
```
|
|
5. Verify from outside:
|
|
```bash
|
|
curl -sI https://hl-mutter-teresa-chemnitz.de | head -1 # 200
|
|
curl -sI https://www.hl-mutter-teresa-chemnitz.de | head -3 # 308 → apex
|
|
```
|
|
Spot-check pages, images, admin login, and (if feasible) a contact form email.
|
|
|
|
Expected downtime ≈ DNS TTL (≤5 min) plus seconds of cert issuance. Visitors
|
|
still resolving the old IP keep getting the identical just-synced content from
|
|
the old server, so the window is soft.
|
|
|
|
## Phase 4 — Post-cutover
|
|
|
|
1. Stop the old server from answering/renewing the domain — the countdown
|
|
config is already removed from the repo, so re-rendering its Caddyfile drops
|
|
the domain:
|
|
```bash
|
|
cd infra/ansible
|
|
ansible-playbook playbooks/setup.yml -i inventory/test.yml --vault-password-file ~/.config/chemnitz-vault-pass
|
|
ssh $OLD "rm -rf /var/www/countdown"
|
|
```
|
|
> Do **not** run this before the DNS switch — it takes the live domain off
|
|
> the old server.
|
|
2. Restore the DNS TTL to its normal value.
|
|
3. Confirm the deployed build has the correct canonical URL: page source /
|
|
sitemap should reference `hl-mutter-teresa-chemnitz.de`, not `skick.app`.
|
|
|
|
## Optional — minimal daily backup (recommended)
|
|
|
|
On the production server, as root (`crontab -e`):
|
|
|
|
```cron
|
|
30 3 * * * mkdir -p /opt/backups && docker exec postgres pg_dump -U postgres -Fc church_website > /opt/backups/db-$(date +\%F).dump && docker run --rm -v uploads-production-media:/m:ro -v uploads-production-documents:/d:ro -v /opt/backups:/b alpine tar -czf /b/uploads-$(date +\%F).tar.gz -C / m d && find /opt/backups -mtime +7 -delete
|
|
```
|
|
|
|
Keeps 7 days of DB dumps + upload tarballs in `/opt/backups`. Off-site copies
|
|
(e.g. rsync to the old server or your machine) are a possible next step.
|
|
|
|
## Ongoing: deploying to production
|
|
|
|
```bash
|
|
cd infra/ansible
|
|
ansible-playbook playbooks/deploy-production.yml -i inventory/production.yml --vault-password-file ~/.config/chemnitz-vault-pass
|
|
```
|
|
|
|
Production tracks the `staging` branch on git.skick.app.
|