# Oki Studios — Operations Runbook **Audience:** Oki Studios on-call, GLI / iTech compliance auditors, aggregator integration teams, jurisdictional regulators. **Companion docs:** `change_mgmt_policy.md` (what triggers re-cert), `integration_spec.md` §8.1 (build integrity), `intake_package_index.md` (safe-send/withhold status), `docs/NGINX_ROUTE_GROUPS_CURRENT.md` (current route/CSP evidence). **Last updated:** 2026-07-01 --- ## 1. Service topology ``` ┌──────────────────┐ HTTPS/WSS public ┌──────────────────┐ │ Player browser │ ◀────────────────────────▶ │ Cloudflare edge │ └──────────────────┘ │ (WAF + DDoS + │ │ bot mgmt + CDN) │ └────────┬──────────┘ │ TLS reterminate │ Cloudflare → origin │ Origin certs (full-strict) │ Authenticated Origin Pulls ┌──────────────▼─────────┐ │ nginx 443 → 8000 │ │ real_ip_from CF only │ │ firewall allows only │ │ Cloudflare CIDRs │ └──────────────┬─────────┘ │ HTTP/1.1 ┌────────────────────────────▼────────────────────┐ │ uvicorn :8000 (FastAPI app) │ │ • app/api/* — public + meta routes │ │ • app/services/* — game_service, progressive │ │ • app/core/* — CERT-SCOPED MATH CANDIDATE │ │ • app/policy/* — jurisdiction toggles │ └──────────┬──────────────────────────┬────────────┘ │ │ ┌─────────────────▼─────┐ ┌──────────▼────────────┐ │ Postgres :5432 │ │ Redis :6379 │ │ • game_rounds (audit) │ │ • session cache │ │ • progressive_jackpots │ │ • progressive bcast │ │ • player_limits │ └────────────────────────┘ └──────┬──────────────────┘ │ archive_command ┌───────────▼──────────────────┐ │ S3 (object-locked, 7-yr) │ │ • WAL segments │ │ • daily base backups │ └──────────────────────────────┘ ``` systemd unit: `refund-rush.service` (uvicorn supervises). Drop-in: `/etc/systemd/system/refund-rush.service.d/override.conf` (env vars, restart policy). nginx: `/etc/nginx/sites-enabled/refund-rush`, mirrored in `deploy/nginx/okistudios.conf` (TLS at edge/origin policy, WSS upgrade for `/api/v1/game/ws/*`, public `/games/rr`, `/games/tt`, `/games/dd`, `/games/dc`, `/games/sw` wrappers, and `/_game/` proxy to the FastAPI game shell). Current public route evidence and the no-public-CDN review-surface invariant are kept in `docs/NGINX_ROUTE_GROUPS_CURRENT.md` and enforced by `tools/validate_review_surface_self_contained.py`. ### 1.1 Cloudflare configuration The production deployment fronts the origin with **Cloudflare** for WAF, DDoS mitigation, bot management, CDN of static assets, and global edge presence. Three things to get right or the whole layer is bypassable: **(a) Origin firewall — accept inbound :443 only from Cloudflare IPs.** Cloudflare publishes their authoritative IP ranges at `https://www.cloudflare.com/ips-v4` and `/ips-v6`. Allow only these on the origin's port 443; drop everything else. Two-line policy on a typical cloud firewall: ```sh # AWS Security Group example (apply to the origin instance) # Pull current CF ranges: curl -s https://www.cloudflare.com/ips-v4 | while read cidr; do aws ec2 authorize-security-group-ingress \ --group-id sg-XXXXX --protocol tcp --port 443 --cidr "$cidr" done # (Run the same loop for /ips-v6 against an IPv6-capable SG rule.) ``` Re-run this monthly via cron — Cloudflare expands the range occasionally and you don't want to lock out new edge nodes. Better: use AWS-managed prefix lists or Cloudflare's Argo Tunnel (next bullet) to skip the maintenance entirely. **(b) Use either Authenticated Origin Pulls OR Argo Tunnel.** - **Authenticated Origin Pulls** (free on CF Pro and up): CF presents a CF-signed client certificate when reaching your origin. nginx is configured to require it via `ssl_client_certificate /etc/cloudflare/origin-pull-ca.pem; ssl_verify_client on;`. Even an attacker who finds your origin IP can't connect — they don't have the cert. This is the standard option. - **Cloudflare Tunnel (Argo Tunnel)**: even better, no inbound port at all. The origin establishes an outbound tunnel to CF's edge, all traffic flows through that. No firewall rules to maintain. Free on every plan since 2022. Worth migrating to when convenient. **(c) Trust `CF-Connecting-IP`, not `X-Forwarded-For`.** Already wired in `app/middleware/network_security.py`. Two compounding pieces of work: 1. nginx config: `set_real_ip_from ; real_ip_header CF-Connecting-IP;` for every CF range. By the time the request hits uvicorn, `request.client.host` is already the real client IP. 2. The middleware reads `CF-Connecting-IP` directly as a belt-and-suspenders. **(d) Cloudflare WAF rules to enable.** | Rule | Purpose | |---|---| | OWASP Core Rule Set | Default protection (SQLi, XSS, RCE patterns) | | Cloudflare Managed Rules — high sensitivity | Their curated rule pack | | Bot Fight Mode (or Super Bot Fight on Pro) | Block known scrapers/credential-stuffers | | Rate limit: 100 req/s per IP for `/api/v1/game/spin` | Caps a single client from hammering /spin | | Rate limit: 10 req/s per IP for `/api/v1/game/init` | Caps session creation abuse | | Geo-block: drop traffic from any country we are not licensed in | Compliance + reduces attack surface | | HTTP/3 + 0-RTT disabled on `/api/v1/game/spin` | 0-RTT permits replay; spin is not idempotent | | Cache: bypass on every `/api/v1/*` route | Game responses are never cacheable | | Cache: aggressive on `/assets/*`, `/css/*`, `/js/*`, `/audio/*` | The frontend ships from CF edge for free | **(e) Aggregator-callback channel.** Cloudflare's standard HTTP proxy is fine for player traffic but **less ideal for the aggregator wallet callback** (`/api/v1/wallet/*`) because: - mTLS termination at CF requires Cloudflare for SaaS / Cloudflare for Platforms (paid tier). - Aggregators typically expect to connect *directly* to your origin with their own client cert. Two clean options: 1. **Bypass CF for the wallet hostname.** Use a separate hostname (`wallet.refundrush.example`) that points directly to the origin, skipping CF. Your origin firewall then needs to allowlist the aggregator's IP block too. The `/api/v1/wallet/*` routes only serve from this hostname (use FastAPI host-based routing or nginx server_name). 2. **Use Cloudflare Spectrum** for raw TCP passthrough with mTLS terminated at the origin. CF does DDoS protection but doesn't see inside the TLS. For a first deal, option 1 is simpler. The IP-allowlist YAML in `ops/config/aggregator_ip_ranges.yaml` already supports this — when traffic comes in over the wallet hostname, the middleware enforces the per-aggregator CIDR check. **(f) Cloudflare Workers (optional, future).** CF Workers can host the launch-URL signing logic (verifies aggregator JWT, sets the session cookie, redirects to the iframe origin). Keeps the validation off the origin and at the edge. Nice-to-have, not required for first launch. --- ## 2. Database roles & privileges Two roles. App connects as one, migrations connect as the other. They have different passwords stored in different secret stores. | Role | Used by | Privileges | |---|---|---| | `refund_rush_app` | uvicorn process (per-spin reads/writes) | SELECT + INSERT on every table; UPDATE only on `game_sessions`, `progressive_jackpots`, `player_limits`, `game_config`; **never UPDATE/DELETE on `game_rounds`, `game_round_transactions`, `player_spend_log`**; never DELETE; never DDL | | `refund_rush_migrate` | alembic from CI / break-glass | Full DDL + DML on the database | Provisioning script: [`game_server/ops/sql/001_append_only_role.sql`](../game_server/ops/sql/001_append_only_role.sql). Idempotent; safe to re-run. **Why this matters for certification prep.** The app role *cannot* rewrite a past round even if the application code is compromised or the operator has shell access to the running container. Combined with the per-round audit hash chain (alembic 004), tampering with a historical row is detectable on the next chain-verify pull. This is the audit-trail integrity posture expected for a regulated slot deployment. The only path to rewriting `game_rounds` is the migration role, which is: 1. Stored in a separate secret manager (AWS Secrets Manager / HashiCorp Vault). 2. Accessed only by alembic from the deploy host (or by a human break-glass operator under §6). 3. Logged on every connection (Postgres `log_connections = on` + `log_disconnections = on` + `log_statement = ddl`, with logs shipped to immutable storage). --- ## 3. Audit chain & verifier Every game round is hashed into a per-session chain. See [`game_server/app/services/audit_chain.py`](../game_server/app/services/audit_chain.py) for the canonical-bytes spec and the verifier. **For a regulator pulling a session's chain:** ```sh curl -H "X-Meta-Token: $RR_META_TOKEN" \ https://game.refundrush.example/api/v1/_meta/audit-chain/ ``` Returns `{ok, round_count, first_broken_round_number, head_hash, scheme}`. `ok=false` means the chain has been tampered with. `first_broken_round_number` points at the earliest detectable break. Token provisioning: `RR_META_TOKEN` env var, set in the systemd drop-in. Rotated quarterly. Empty value puts the endpoint in fail-safe (refuses to serve). For self-testing: run `pytest tests/test_audit_chain.py` (no DB required) — covers the math; live integration is exercised via `tests/test_spin.py` end-to-end. --- ## 4. WAL archiving & disaster recovery Postgres write-ahead log is the canonical record of every committed transaction. Shipping WAL segments to immutable storage gives: - **Point-in-time recovery** to any second in the retention window. - **Independent verifiability** — regulator can replay the WAL and recompute every round, comparing to the live DB. - **Tamper-evidence at the storage layer** — even if app + migrate roles are simultaneously compromised, the WAL stream is append-only and shipped offsite. ### Postgres config (production) ```conf # /etc/postgresql/16/main/postgresql.conf wal_level = replica archive_mode = on archive_command = '/usr/local/bin/wal-archive %p %f' # see script below archive_timeout = 60 # force flush at least every 60s for low-volume periods max_wal_senders = 3 wal_keep_size = 1GB ``` ### Archive script (`/usr/local/bin/wal-archive`) Production target: S3 bucket `s3://refundrush-wal-prod//`, configured with: - **S3 Object Lock in COMPLIANCE mode**, 7-year retention. Cannot be deleted by any IAM principal, including root, until expiry. - **Server-side encryption** (SSE-KMS with customer-managed key). - **Bucket policy** denying any `s3:DeleteObject*` / `s3:PutObjectRetention` change. - **Cross-region replication** to a second bucket in a different region for DR. Pseudocode (the actual production script lives outside this repo for least-privilege reasons; written in Go for stability and shipped as a static binary): ```sh #!/bin/sh # Usage: wal-archive WAL_PATH="$1" WAL_NAME="$2" SHA=$(sha256sum "$WAL_PATH" | cut -d' ' -f1) aws s3api put-object \ --bucket refundrush-wal-prod \ --key "refund_rush/$(date -u +%Y/%m/%d)/$WAL_NAME" \ --body "$WAL_PATH" \ --metadata sha256="$SHA",hostname="$(hostname)" \ --object-lock-mode COMPLIANCE \ --object-lock-retain-until-date "$(date -u -d '+7 years' +%Y-%m-%dT%H:%M:%SZ)" \ --server-side-encryption aws:kms \ --ssekms-key-id alias/refundrush-wal ``` ### Base backups Daily base backup via `pg_basebackup` to `s3://refundrush-base-prod/`, 90-day rolling retention. Combined with WAL, supports recovery to any second in the past 90 days; for older incidents, the WAL alone supports replay since archive start. ### Recovery RTO / RPO - **RPO**: ≤ 60 seconds (`archive_timeout`). - **RTO**: ≤ 4 hours from cold (provision new instance, restore latest base backup, replay WAL forward). On a hot-standby setup (recommended for production), RTO drops to seconds. ### Verification A weekly cron runs a "phantom restore" — pulls the latest base + WAL chain into a sandboxed instance, runs `tests/test_audit_chain.py` against the restored DB, and reports pass/fail to PagerDuty. Untested backups are not backups. --- ## 5. Secrets management Production secrets are stored in **AWS Secrets Manager** (or equivalent), pulled into the running container via instance role + a bootstrap script at startup. They are never committed to source, never in env-var files in git, never in the systemd drop-in directly. | Secret | Purpose | Rotation | |---|---|---| | `DATABASE_URL` (app role) | uvicorn ↔ Postgres (refund_rush_app) | Quarterly | | `DATABASE_URL` (migrate role) | alembic from CI | Quarterly | | `RR_SESSION_SECRET` | HMAC for session tokens | Quarterly | | `RR_META_TOKEN` | guards `/_meta/audit-chain/*` | Quarterly | | `RR_EMERGENCY_DISABLE_TOKEN` | shared secret for `/_meta/emergency-disable` and `/_meta/emergency-enable` | Quarterly | | Aggregator-callback mTLS cert | server cert presented to aggregator wallet API | Per cert lifetime (typically annual) | | Aggregator-callback CA bundle | verifies aggregator client cert | Per CA lifetime | | S3 bucket access keys | WAL + base-backup writers | Quarterly via IAM role | | KMS key alias | S3 SSE-KMS | Never (customer-managed key) | | GPG signing key | git tags + lab submissions | 2-year lifetime | **Rotation procedure (90-day reminder triggers it):** 1. Generate new value in Secrets Manager (versioned). 2. Update the live secret pointer. 3. Roll the running pods (zero-downtime if multiple replicas; brief outage if single-instance). 4. Verify with `/_meta/build-info` ping. 5. Mark old version `pendingDeletion` after 24h of stable run. 6. Log rotation event in operational register. --- ## 6. Break-glass production access There are four break-glass scenarios. All require justification logged in the operational register *before* the action and a counter-sign within 7 days. ### 6.1 Direct DB query (read-only) Use case: regulator requests a specific player's round history in a 24h disclosure window, beyond what the audit-chain endpoint surfaces. ```sh # From the deploy host, using the read-only app role: PGPASSWORD=$(aws secretsmanager get-secret-value --secret-id refund-rush/db-app | jq -r .password) \ psql -h db.refundrush.internal -U refund_rush_app -d refund_rush \ -c "SELECT round_number, total_win, encode(audit_hash, 'hex') FROM game_rounds WHERE session_id = '' ORDER BY round_number;" ``` Logged automatically (Postgres `log_statement = ddl,mod` + `log_connections`). ### 6.2 Direct DB write (migrate role) Use case: emergency data correction (e.g. duplicate-row deduplication after a bug). **Must be paired with a re-cert ticket** because writes via the migrate role are math-affecting if they touch `game_rounds`. Procedure: 1. Open re-cert ticket with the lab. Get ticket reference. 2. Connect with migrate role. 3. Run the correction inside a transaction. 4. Recompute the audit chain forward from the corrected round with a reviewed incident-specific migration script; never hand-edit hash-chain rows. 5. Verify chain is intact via `/_meta/audit-chain`. 6. Commit. 7. Log + counter-sign. ### 6.3 SSH to the application host Use case: emergency debugging when log shipping is broken. - Bastion host with hardware-key MFA (YubiKey). - 1-hour TTL session. - All shell sessions recorded (asciinema → S3). - No production secrets accessible via the bastion shell — pull from Secrets Manager only. ### 6.4 Emergency game disable Use case: math hash mismatch detected, critical bug surfaced, regulator demand. ```sh curl -X POST -H "X-Emergency-Token: $RR_EMERGENCY_DISABLE_TOKEN" \ https://game.refundrush.example/api/v1/_meta/emergency-disable ``` Game returns 503 to all `/spin` and `/feature/*` calls until re-enabled. `/health` and `/_meta/*` continue to serve so the operator/regulator can verify the disable + read the audit chain. Re-enable after the incident is cleared: ```sh curl -X POST -H "X-Emergency-Token: $RR_EMERGENCY_DISABLE_TOKEN" \ https://game.refundrush.example/api/v1/_meta/emergency-enable ``` --- ## 7. On-call runbook (the actual runbook page) ### 7.1 Math hash mismatch **Symptom:** `/_meta/build-info` returns a `math_hash` that does not match the certificate. **Severity:** P0. Game must not serve real-money spins. **Action:** 1. Hit `/_meta/emergency-disable` immediately. 2. Notify aggregators within 24h (per §change_mgmt_policy.md §5). 3. Investigate: which file changed under `app/core/`? Was a deploy mis-pinned? Did someone push to main without going through CI? 4. Roll back to the last known-good build. 5. Verify hash match on rollback. 6. Re-enable game. 7. Post-mortem within 7 days; submit to lab if any spins were served at the bad hash (will trigger re-cert review). ### 7.2 Audit chain break detected **Symptom:** `/_meta/audit-chain/` returns `ok=false`. **Severity:** P0 if the affected session has any real-money rounds. P1 if demo only. **Action:** 1. Capture the response (record `first_broken_round_number`). 2. Pull the surrounding rows directly from the DB: who wrote them, when? 3. Cross-check against WAL archive — does the WAL agree with the live DB? If WAL agrees with DB but the chain is broken, the chain itself was corrupted at write time (likely a code bug in `game_service.py`'s chain-write block). If WAL disagrees, someone rewrote the live DB after the fact — that's a security incident. 4. Disable game (§6.4). 5. Notify lab + jurisdictional regulator (within their disclosure windows). 6. Investigate. Restore from WAL replay if needed. ### 7.3 Database write failures **Symptom:** `permission denied for relation game_rounds` (`UPDATE` attempted) in app logs. **Severity:** P1. **Action:** This is the append-only role doing its job — the app code tried to UPDATE a row it shouldn't. Find the code path, determine whether the intended behaviour is legitimate (rare, requires a new mutable-state table + grant) or a bug (someone added an `UPDATE game_rounds` statement). Fix code. The denied UPDATE never happened, so no data is corrupt. ### 7.4 Progressive pool drift **Symptom:** progressive jackpot pool value diverges from `(seed + sum_of_contributions - sum_of_awards)`. **Severity:** P1. **Action:** Pull the per-spin contribution log + award log. Reconcile. If the divergence is small (<1%), likely a rounding accumulation bug — log + monitor. If large, a row was lost or written incorrectly — investigate and potentially restore from WAL. ### 7.5 Aggregator wallet timeout / drift **Symptom:** spin requests timing out at the aggregator wallet boundary, or balance reconciliation diverges between our `game_rounds` and the aggregator's wallet ledger. **Severity:** P1. **Action:** Check aggregator status page first (often their issue, not ours). If ours, capture the failing round_id and bet/win, escalate to aggregator integration contact with the round-level evidence. --- ## 8. Compliance contact registry | Body | Function | Channel | |---|---|---| | GLI / iTech (lab) | Cert maintenance, math defense, change-mgmt approval | Account manager email + ticket portal | | MGA | Compliance attestation, complaint disclosure | MGA Operator Portal | | UKGC | RTS Annex A self-assessment, RG complaint disclosure | UKGC eServices | | AGCO Ontario | Standards 1–2 self-assessment, RG complaints | AGCO iAGCO portal | | Aggregator(s) | Operational incidents, integration changes | Aggregator integration Slack / shared on-call | Disclosure windows (see `change_mgmt_policy.md` §5 for full table): UKGC 24h, MGA 48h, AGCO 24h. **A 24h disclosure window starts at detection, not at root-cause-identification.** Always disclose; under-disclosure has cost more licenses than over-disclosure. --- ## 9. Annual re-validation calendar | Body | Cadence | Next deadline | Evidence required | |---|---|---|---| | MGA | Annual | _to set after cert_ | Compliance attestation, sample audit log review | | UKGC | Annual | _to set after cert_ | RTS Annex A self-assessment, technical sample | | AGCO | Annual | _to set after cert_ | Standards 1–2 self-assessment, RG check | | GLI | Biennial | _to set after cert_ | Full re-test of theoretical RTP per variant | A calendar reminder fires 60 days before each deadline. Evidence packages live in `aggregator/recert_packages///`. --- *This runbook is the working document for production operations. Every incident, every break-glass action, every quarterly secret rotation gets a register entry referencing the relevant section. Reviewed annually + after every material incident.*