church-website/infra

Infrastructure & Deployment

Architecture

VPS (Ubuntu 24, 8 GB RAM)
├── Caddy          — reverse proxy + auto SSL (native)
├── PostgreSQL     — postgis/postgis:16-3.4 (Docker)
├── Forgejo        — git server + CI/CD (Docker)
├── Forgejo Runner — executes CI/CD jobs (Docker)
├── app-staging    — Next.js + Payload CMS (Docker)
└── app-test       — Next.js + Payload CMS (Docker)

URL	Port	Purpose
mutter-teresa.skick.app	3001	Client demo (staging)
mutter-teresa-test.skick.app	3002	Developer testing
git.skick.app	3003	Forgejo git server

All app and database containers share the Docker network church-website-net.

---

Prerequisites

• Ansible installed locally (pip install ansible or brew install ansible)
• SSH access to the VPS (root or sudo user)
• DNS records pointing to the VPS IP:
  • mutter-teresa.skick.app → VPS IP
  • mutter-teresa-test.skick.app → VPS IP
  • git.skick.app → VPS IP

---

Quick Start: First-Time Server Setup

1. Configure secrets

Create an encrypted vault from the example template:

cd infra/ansible
cp inventory/group_vars/all/vault.yml.example inventory/group_vars/all/vault.yml
ansible-vault encrypt inventory/group_vars/all/vault.yml
ansible-vault edit inventory/group_vars/all/vault.yml

Fill in all CHANGE_ME values:

• vault_ansible_become_pass — VPS root password
• vault_postgres_root_password — PostgreSQL root password
• vault_db_password_staging / vault_db_password_test — database passwords
• vault_payload_secret_staging / vault_payload_secret_test — Payload CMS secrets
• vault_google_bucket — Google Cloud Storage bucket name
• vault_resend_api_key — Resend email API key
• vault_repo_url — Forgejo repository URL (e.g., ssh://git@git.skick.app:2222/org/church-website.git)

2. Configure inventory

Edit infra/ansible/inventory/test.yml:

• Set ansible_host to your VPS IP address
• Adjust ansible_user and SSH key path if needed

3. Run the playbook

cd infra/ansible
ansible-playbook playbooks/setup.yml -i inventory/test.yml --ask-vault-pass

This will:

1. Install Docker, configure firewall
2. Start PostgreSQL with both databases
3. Install and configure Caddy with SSL
4. Start Forgejo and the CI/CD runner
5. Clone the repo, build, and deploy both environments

4. Set up Forgejo

After the playbook completes:

1. Visit https://git.skick.app and complete the initial Forgejo setup
2. Create an organization and repository
3. Add the VPS SSH key to the repository for pull access
4. Register the Forgejo Runner:

   ssh root@YOUR_VPS_IP
   docker exec -it forgejo-runner forgejo-runner register \
     --instance https://git.skick.app \
     --token YOUR_RUNNER_TOKEN \
     --name local-runner \
     --labels ubuntu-latest:docker://node:22
   

5. Push to the staging branch — CI/CD will deploy automatically

---

Environment Variables

Variable	Description	Build-time?
DATABASE_URI	PostgreSQL connection string	No
PAYLOAD_SECRET	Payload CMS encryption secret	No
NEXT_PUBLIC_SERVER_URL	Public URL of the app	Yes
NEXT_PUBLIC_SITE_ID	Site identifier (e.g., chemnitz)	Yes
GOOGLE_BUCKET	GCS bucket for media storage	No
RESEND_API_KEY	Resend API key for emails	No

Variables marked "Build-time" are baked into the Docker image during docker build (via --build-arg). Changes to these require a rebuild.

---

Manual Operations

Check container logs

docker logs app-staging
docker logs app-test
docker logs postgres
docker logs forgejo

Redeploy manually (without CI/CD)

cd /opt/church-website/repo
git pull origin staging
/opt/church-website/scripts/deploy.sh staging 3001
/opt/church-website/scripts/deploy.sh test 3002

Run migrations manually

docker exec app-staging npx payload migrate
docker exec app-test npx payload migrate

Database backup

# Backup staging database
docker exec postgres pg_dump -U church_website_staging church_website_staging > backup_staging_$(date +%Y%m%d).sql

# Backup test database
docker exec postgres pg_dump -U church_website_test church_website_test > backup_test_$(date +%Y%m%d).sql

# Backup all databases
docker exec postgres pg_dumpall -U postgres > backup_all_$(date +%Y%m%d).sql

Database restore

# Restore staging database
cat backup_staging.sql | docker exec -i postgres psql -U church_website_staging church_website_staging

Restart a single service

docker restart app-staging
docker restart app-test
docker restart postgres

---

Deploy via Ansible (without CI/CD)

Use the deploy.yml playbook to deploy from your local machine — no Forgejo runner or CI/CD pipeline needed. This is useful for hotfixes, CI outages, or production servers without Forgejo.

cd infra/ansible

# Deploy to test/staging VPS
ansible-playbook playbooks/deploy.yml -i inventory/test.yml --ask-vault-pass

# Deploy to production
ansible-playbook playbooks/deploy.yml -i inventory/production.yml --ask-vault-pass

What it does:

1. Pulls the latest code from the configured branch (repo_branch in inventory)
2. Runs deploy.sh for each environment (sequentially to save RAM), which:
   • Builds the Docker app image with build-time env vars
   • Builds a migration image and runs npx payload migrate
   • Stops the old container, starts the new one
   • Prunes old Docker images

Deploy a specific branch:

ansible-playbook playbooks/deploy.yml -i inventory/test.yml --ask-vault-pass \
  -e repo_branch=feature/my-branch

Deploy only one environment (e.g., just staging):

ansible-playbook playbooks/deploy.yml -i inventory/test.yml --ask-vault-pass \
  -e '{"app_environments": [{"name": "staging", "port": 3001}]}'

Note: The server must already be provisioned with setup.yml before using deploy.yml. The deploy playbook only pulls code and rebuilds containers — it does not install Docker, Caddy, or PostgreSQL.

---

CI/CD

The Forgejo Actions workflow (.forgejo/workflows/deploy.yml) triggers on push to the staging branch. It:

1. Pulls the latest code on the VPS
2. Builds a new Docker image for staging
3. Stops the old container, starts the new one
4. Runs database migrations
5. Repeats for the test environment (sequentially, to save RAM)

---

Adding a New Environment

1. Add a new entry to app_environments in the inventory file
2. Add a new entry to caddy_domains with the new domain
3. Add a new database entry to databases
4. Run the playbook: ansible-playbook playbooks/setup.yml -i inventory/test.yml
5. Update the deploy workflow to include the new environment

---

Production Setup

1. Copy and edit the production inventory:

   cp infra/ansible/inventory/production.yml infra/ansible/inventory/my-production.yml
   

2. Fill in the production VPS IP, domain, and secrets
3. Run the playbook (skip Forgejo):

   ansible-playbook playbooks/setup.yml -i inventory/my-production.yml --ask-vault-pass
   

4. Set up a deploy workflow for production (triggered on tags/releases)

---

Troubleshooting

Build fails with OOM

The VPS has 4 GB RAM + 2 GB swap. Docker builds can peak at ~1.5 GB. If builds fail:

• Ensure only one build runs at a time (deploy script is sequential)
• Check swap: free -h
• Increase swap: edit swap_size_mb in inventory and re-run playbook

SSL certificate not working

• Ensure DNS records point to the VPS IP: dig mutter-teresa.skick.app
• Check Caddy logs: journalctl -u caddy
• Caddy auto-renews certificates — if stuck, restart: systemctl restart caddy

Database connection refused

• Check PostgreSQL is running: docker ps | grep postgres
• Check the container is on the right network: docker network inspect church-website-net
• Test connection: docker exec postgres psql -U postgres -l

Container won't start

• Check logs: docker logs app-staging
• Check if port is in use: ss -tlnp | grep 3001
• Check .env file: cat /opt/church-website/envs/staging/.env

---

Local Development

For local development with PostgreSQL:

# Start PostgreSQL (from project root)
docker compose up -d

# Configure .env
DATABASE_URI=postgres://postgres:password@localhost:5432/church_website_dev

# Start dev server
npm run dev