Skip to main content
Control Plane

Production Checklist

Checklist for putting a self-hosted control plane into production.

We recommend passing this page to your coding agent to verify your configuration before deploying.

Security

  • Configure an admin token. Generate a strong random value and set RIVET__AUTH__ADMIN_TOKEN. Without one the API is unauthenticated. See Configuration.
  • Keep the admin token out of client reach. It must never appear in a public endpoint or anywhere a browser can read it.
  • Terminate TLS. Every connection to the control plane should be encrypted at a reverse proxy or load balancer. See TLS.
  • Harden the outbound destination policy. The control plane dials the serverless URLs in your runner configs from inside its own network, so anyone who can write a runner config can aim it at services only reachable from there. By default outbound denies every non-globally-routable destination except loopback. Set outbound.allow_loopback to false unless a worker genuinely runs on the control plane host, since loopback reaches the control plane’s own API, guard, and any sidecar. If your workers live on your network, prefer outbound.allow_cidrs or outbound.allow_hosts for the specific destinations over outbound.allow_private_networks, which opens every private range. See Configuration.
  • Disable crash reporting in locked-down environments. The control plane sends crash reports to Rivet by default. Set RIVET__TELEMETRY__ENABLED=false for air-gapped or regulated deployments. See Configuration.

Resources

  • Set container resource requests and limits. The shipped Kubernetes manifests request 2 CPU and 2 GB per control plane instance, with limits of 3 CPU and 4 GB. Size other platforms to match; 1 CPU is enough only for a quiet single-node deployment.
  • Configure health checks. Liveness and readiness probes against /health on port 6421, with a five second timeout. See Ports.
  • Raise load balancer idle timeouts to 3600 seconds. Long-lived WebSockets are dropped by the 30 to 60 second defaults, which causes reconnect storms. This is the most common self-hosting mistake. See Ports.

Scaling

  • Run two or more nodes. A single control plane node is a single point of failure. Deploy at least two behind a load balancer.
  • Do not run multiple file system nodes. RocksDB is single-node. Multi-node deployments need PostgreSQL or FoundationDB. See Storage.
  • Configure autoscaling, on a shared backend only. Target 60% CPU and 80% memory to leave headroom for traffic spikes. In Kubernetes this is a Horizontal Pod Autoscaler, and the shipped manifests include one at 2 to 10 replicas with a 15 minute scale-down stabilization window. Memory is a backstop rather than a scaling signal: it does not fall as fast as CPU, so scaling in needs to wait it out. Never autoscale a node on the file system backend: adding replicas to a single-node RocksDB deployment corrupts it.
  • Check the rate limit on your serverless worker platform. Actor start requests all originate from your control plane nodes, so they come from a small set of IPs. Per-IP rate limits throttle the control plane long before they would throttle end-user traffic. Size the limit to your peak actor create and wake rate.

Storage

  • Choose the backend that matches your node count. File system for a single node, PostgreSQL for multi-node up to roughly 1,000 concurrent actors, FoundationDB beyond that. See Storage.
  • Configure automated backups, and restore one into a scratch environment to prove the path works. See Backup & Restore.
  • Configure failover. A standby replica with automatic failover, for PostgreSQL.
  • Deploy two or more NATS replicas for multi-node pub/sub high availability.

Networking

  • Raise the WebSocket timeout on any L7 load balancer in front of the control plane. The connection at risk is the long-lived worker-to-control-plane WebSocket that every worker holds open, not just client-facing actor WebSockets. Most L7 HTTPS load balancers (GCP HTTPS LB / GKE Ingress, AWS ALB, Cloudflare, nginx) cap a single WebSocket’s lifetime, and ping/pong does not reset it. Defaults are typically 30 to 60 seconds, which kills that connection with WS code 1006 on a fixed cadence and tears down every actor WebSocket multiplexed over it. Set the backend timeout to at least 1 hour (timeoutSec: 86400 on a GCP BackendConfig, idle_timeout.timeout_seconds: 3600 on an AWS ALB, proxy_read_timeout on nginx), or front it with an L4 TCP load balancer that does not terminate the WebSocket.
  • Raise idle timeouts on every layer to at least 1 hour. The same applies to the load balancer fronting your own app, since workers connect to it over WebSocket.

Operations

  • Pin the image tag. Never run latest in production. See Upgrades.
  • Configure OpenTelemetry. Set RIVET_OTEL_ENABLED=1 and point RIVET_OTEL_GRPC_ENDPOINT at your collector. Tune RIVET_OTEL_SAMPLER_RATIO for trace volume.
  • Set up alerts. CPU, memory, request latency, and error rates, with thresholds that fire before an outage rather than during one.

Enterprise

Contact enterprise support for architecture review, scaling guidance, and FoundationDB.

FoundationDB is available on Enterprise deployments. If you run it:

  • Provision fast disks. Use NVMe SSDs with at least 500 MiB/s sustained throughput under a mixed small read/write workload. Keep volumes under 75% full and benchmark drives before production. FoundationDB fsyncs its transaction logs on every commit and its storage servers on a tight interval, so slow disks surface as rising storage-server durability lag and write-queue ratekeeper limiting under write-heavy load.
  • Do not rely on the cluster default disk class. Network-attached disks whose throughput scales with volume size (GCP pd-balanced, or the default StorageClass on many managed Kubernetes clusters) are often far below that target on small volumes and will bottleneck FoundationDB. Pin a fast disk class (GCP premium-rwo / pd-ssd, a provisioned Hyperdisk, or local NVMe) sized so its provisioned throughput meets the target.

Next steps