Session Handoff: Appwrite Stack Setup & Infrastructure Hardening

Date: 2026-03-14 Session Duration: ~3 hours Session Focus: Bring Appwrite stack to production-ready state on bab1.mgmt.toal.ca — env templating, systemd, secrets, networking, monitoring, ansible-navigator fixes Context Usage at Handoff: ~64%

What Was Accomplished

Created Appwrite .env Jinja2 template → playbooks/templates/appwrite.env.j2
Created systemd unit template → playbooks/templates/appwrite.service.j2
Created Prometheus node exporter playbook → playbooks/install_node_exporter.yml
Moved all Appwrite vars to inventory → ~/Dev/inventories/bab-inventory/host_vars/bab1.mgmt.toal.ca/appwrite.yml
Created monitoring vars → ~/Dev/inventories/bab-inventory/host_vars/bab1.mgmt.toal.ca/monitoring.yml
Created secrets file using HashiCorp Vault lookups → ~/Dev/inventories/bab-inventory/host_vars/bab1.mgmt.toal.ca/secrets.yml
Rewrote playbooks/install_appwrite.yml — added .env deploy, systemd, tags (deps/image/configure), handler, production compose URL
Heavily extended playbooks/tasks/patch_appwrite_compose.yml — Traefik pin, image fix, forwardedHeaders, proxyProtocol, handler notifications
Added docker prune after upgrade → playbooks/upgrade_appwrite.yml
Added community.hashi_vault to requirements.yml
Fixed ansible-navigator pipelining — moved config to environment-variables.set in ~/.ansible-navigator.yml; also added SSH multiplexing, fact caching, retry file suppression, profile_tasks via CALLBACKS_ENABLED
Deleted secrets.yml.example — contained plaintext secrets (security risk)

Exact State of Work in Progress

503 from appwrite.toal.ca: proxyProtocol patch added to patch_appwrite_compose.yml but not yet re-run against the host. The appwrite stack on bab1 is still running the old compose without proxyProtocol.trustedIPs. Next action: run ansible-navigator run playbooks/install_appwrite.yml --mode stdout --skip-tags deps,image to apply patches and restart.
Vault secret not populated: kv/oys/bab-appwrite in HashiCorp Vault (http://nas.lan.toal.ca:8200) has not been populated. The secrets.yml will fail lookups until this is done.

Decisions Made This Session

DECISION: HashiCorp Vault over ansible-vault for secrets BECAUSE AAP and dev workflows both need access; 1Password-based ansible-vault is local-only. Vault path: kv/data/oys/bab-appwrite. All secrets stored as fields in one KV secret. STATUS: confirmed.
DECISION: appwrite.io/install/compose as compose source BECAUSE the GitHub raw URL pointed to a dev compose (image: appwrite-dev, custom entrypoint: php -e app/http.php) that fails with the official image. STATUS: confirmed.
DECISION: Traefik pinned to 2.11.31 BECAUSE traefik:2.11 (floating tag) is incompatible with Docker Engine >= 29. STATUS: confirmed.
DECISION: systemd Type=oneshot RemainAfterExit=yes BECAUSE docker compose up -d exits after starting containers; oneshot keeps the unit in "active" state. STATUS: confirmed.
DECISION: node exporter uses security_opts label=disable BECAUSE on RHEL 9 with SELinux enforcing, :z on a / bind-mount would recursively relabel the entire filesystem. label=disable avoids this for a read-only mount. STATUS: confirmed.
DECISION: ANSIBLE_VAULT_IDENTITY_LIST moved to navigator set env vars BECAUSE ansible.config.path does not auto-mount the file into the EE — the path is set via ANSIBLE_CONFIG env var but the file is never present at that path inside the container. STATUS: confirmed.
DECISION: profile_tasks via ANSIBLE_CALLBACKS_ENABLED, not ANSIBLE_STDOUT_CALLBACK BECAUSE profile_tasks is an aggregate callback, not a stdout callback. Setting it as STDOUT_CALLBACK caused 'sort_order' error. STATUS: confirmed.

Key Numbers Generated or Discovered This Session

appwrite_version: "1.8.1" — current pinned version in install_appwrite.yml
appwrite_traefik_version: "2.11.31" — minimum Traefik version for Docker >29
appwrite_web_port: 8080 — host port mapping for Traefik HTTP
appwrite_websecure_port: 8443 — host port mapping for Traefik HTTPS
appwrite_traefik_trusted_ips: "192.168.0.0/22" — HAProxy subnet, used for both forwardedHeaders AND proxyProtocol trustedIPs
node_exporter_version: "1.9.0", node_exporter_port: 9100
HAProxy backend config: send-proxy-v2 check-send-proxy on both appwrite and babdevapi backends → Traefik MUST have proxyProtocol enabled
Context at handoff: 128.2k / 200k tokens (64%)

Conditional Logic Established

IF compose source is GitHub raw URL THEN it may be the dev build compose (image: appwrite-dev) BECAUSE Appwrite's main branch docker-compose.yml is for local development
IF Traefik proxyProtocol.trustedIPs is not set THEN HAProxy send-proxy-v2 causes 503 BECAUSE Traefik reads the PROXY protocol header as malformed HTTP/TLS data
IF ansible.config.path is set in navigator config WITHOUT a volume mount THEN the ansible.cfg settings are silently ignored inside the EE BECAUSE the file is not present at that path in the container

Files Created or Modified

File Path	Action	Description
`playbooks/templates/appwrite.env.j2`	Created	Full Appwrite .env template; secrets use `vault_appwrite_*` vars
`playbooks/templates/appwrite.service.j2`	Created	systemd unit, Type=oneshot RemainAfterExit=yes
`playbooks/install_appwrite.yml`	Modified	Added .env deploy, systemd, tags, handler, production compose URL
`playbooks/tasks/patch_appwrite_compose.yml`	Modified	Added Traefik pin, image fix, forwardedHeaders, proxyProtocol, handler notifications
`playbooks/upgrade_appwrite.yml`	Modified	Added docker prune after upgrade
`playbooks/install_node_exporter.yml`	Created	Prometheus node exporter; pid_mode=host, label=disable, SYS_TIME cap
`requirements.yml`	Modified	Added community.hashi_vault
`~/.ansible-navigator.yml`	Modified	Replaced file-mount approach with environment-variables.set; added SSH mux, fact caching
`~/Dev/inventories/bab-inventory/host_vars/bab1.mgmt.toal.ca/appwrite.yml`	Created	All non-secret Appwrite vars
`~/Dev/inventories/bab-inventory/host_vars/bab1.mgmt.toal.ca/monitoring.yml`	Created	node_exporter_version, node_exporter_port
`~/Dev/inventories/bab-inventory/host_vars/bab1.mgmt.toal.ca/secrets.yml`	Modified	HashiCorp Vault lookups for vault_appwrite_* vars
`secrets.yml.example`	Deleted	Contained plaintext secrets — security risk

What the NEXT Session Should Do

First: Populate HashiCorp Vault secret at kv/oys/bab-appwrite with fields: openssl_key, db_pass, db_root_pass, smtp_password, executor_secret, github_client_secret, github_webhook_secret, github_private_key
Then: Run ansible-navigator run playbooks/install_appwrite.yml --mode stdout --skip-tags deps,image to apply proxyProtocol patch and restart the Appwrite stack
Then: Verify curl -v https://appwrite.toal.ca no longer returns 503
Then: Install community.hashi_vault in the ee-demo execution environment (currently missing from the EE image)
Then: Run ansible-navigator run playbooks/install_node_exporter.yml --mode stdout to deploy node exporter

Open Questions Requiring User Input

Vault secret population: RESOLVED 2026-03-14 — populated by hand at kv/oys/bab-appwrite.
_APP_DOMAIN_TARGET: RESOLVED 2026-03-14 — added to appwrite.env.j2 defaulting to appwrite_domain. Fixes Domain::__construct() null in console.php:49.
community.hashi_vault in EE: RESOLVED 2026-03-14 — added to ee-demo EE image.
SSH_AUTH_SOCK not passed to EE: RESOLVED 2026-03-14 — confirmed working.

Assumptions That Need Validation

ASSUMED: appwrite.io/install/compose returns the production compose for 1.8.x — validate by inspecting the downloaded file on next run
ASSUMED: Traefik entrypoint names in production compose are appwrite_web and appwrite_websecure — these were confirmed in the dev compose; verify they match in production compose
ASSUMED: community.hashi_vault.hashi_vault lookup returns data.data fields directly for KV v2 — validate by running a test lookup

What NOT to Re-Read

The HAProxy config (provided inline by user) — key facts preserved above
The original Appwrite .env (provided inline by user) — fields captured in appwrite.env.j2

Files to Load Next Session

playbooks/install_appwrite.yml — if continuing install/configure work
playbooks/tasks/patch_appwrite_compose.yml — if debugging compose patches
~/Dev/inventories/bab-inventory/host_vars/bab1.mgmt.toal.ca/secrets.yml — if working on vault integration
docs/summaries/handoff-2026-03-14-appwrite-setup.md — this file (load at session start)

8.9 KiB Raw Blame History