Istio Kubernetes deployment runbook

The hosted topology runs the dashboard/API and optional grounding proxy on Kubernetes, uses managed PostgreSQL, and delegates AWS ingress, workload identity, mTLS, authorization, and network telemetry to Istio. The static landing site is independently hosted on Cloudflare and is outside this cluster.

Platform boundary

The platform team owns Kubernetes, Gateway API CRDs, Istio ambient components, DNS, cert-manager, the external secret controller, load balancers, and cluster observability. The Gettier release owns charts/gettier-hosted, application images, database migrations, routes, health endpoints, and application-level OIDC/tenant/billing authorization.

Istio authorization is defense in depth; it does not replace Gettier ingestion keys, OIDC sessions, tenant RLS, Stripe webhook verification, or application audit logs.

Rollout order

1. Install or upgrade Istio with the ambient profile and verify istiod, CNI, ztunnel, and the istio GatewayClass.
2. Create a dedicated namespace without a sidecar-injection label.
3. Reconcile the production environment Secret and wait until it exists.
4. Issue a TLS Secret covering app, api, and optional llm hostnames.
5. Build and push immutable web and migration images, plus proxy when enabled.
6. Back up PostgreSQL and verify point-in-time recovery.
7. Run helm upgrade --install --wait; Helm blocks on the pre-upgrade migration Job.
8. Wait for Gateway Programmed=True and HTTPRoute Accepted=True/ResolvedRefs=True.
9. Point proxied app, api, and optional llm DNS records at the Gateway address. Cloudflare owns the apex and www as landing custom domains; never point those records at AWS.
10. Execute login, ingestion, dashboard, invitation, Stripe, and quota smoke tests from the release checklist in docs/deployment.md.

Security validation

Verify workloads appear in ztunnel with HBONE, then inspect policy and TLS:

istioctl ztunnel-config workloads -n istio-system | grep gettier
kubectl -n gettier get peerauthentication,authorizationpolicy -o yaml
istioctl proxy-status

The application Pods carry istio.io/dataplane-mode=ambient; the migration Job carries istio.io/dataplane-mode=none and disables sidecar injection. Strict mTLS and an ALLOW-only policy restrict service ports to the generated Gateway identity plus explicitly configured principals. Application containers are non-root, read-only, tokenless, seccomp-confined, and capability-free.

Rollback

Use helm rollback <release> <revision> --wait --timeout 10m with the prior immutable image tags. Migrations are forward-only and must remain compatible with the previous application revision. Use database point-in-time recovery only for a genuine data incident. If an Istio policy change blocks traffic, first revert the Helm release; do not disable application authentication or expose ClusterIP Services externally as a workaround.

Observability

Alert on Gateway 5xx rates, route rejection, unhealthy replicas, migration Job failures, OIDC callback errors, Stripe webhook failures, ingestion rejects, and database saturation. Istio supplies transport metrics and identities; Gettier's dashboard remains the product source of truth for tenant usage, runs, facts, gates, and billing state.