Operations

Reference for running AIControl in production: token lifecycle, human review workflow, monitoring, updates, and troubleshooting.

Token lifecycle

AIControl uses JWT tokens for all API authentication. Tokens do not expire but can be revoked immediately.

Token roles:

Issue a token:

# Via CLI
docker compose -f docker-compose.yml -f docker-compose.app.yml \
  exec api python scripts/issue_token.py \
  --role agent \
  --desc "claims-processing-agent-prod"

# Via API
curl -X POST http://localhost:8001/tokens \
  -H "Authorization: Bearer <admin-token>" \
  -H "Content-Type: application/json" \
  -d '{"role": "agent", "description": "claims-processing-agent-prod", "agent_id": "uuid"}'

The token value is returned once and not stored in plaintext. Save it to your secrets manager immediately.

Revoke a token:

# Via CLI
docker compose exec api python scripts/revoke_token.py --id TOKEN_UUID

# Via API — revoke all tokens for an agent
curl -X DELETE http://localhost:8001/agents/{agent_id}/token \
  -H "Authorization: Bearer <admin-token>"

Or use the Tokens tab in the dashboard.

Token rotation:

When you issue a new token with agent_id specified, any prior active token for that agent is automatically revoked. To rotate a token: issue a new one (the old one is revoked), update the environment variable in your agent's deployment, redeploy.

One active token per agent is enforced when agent_id is supplied. Tokens issued without an agent_id are not subject to this constraint (useful for admin scripts and automation).

User management

Dashboard users are distinct from machine API tokens. Dashboard users log in with email + password; machine API tokens are passed as Bearer headers by AI agents and scripts.

First run: The setup wizard creates the root administrator. All subsequent users are invited by the root admin via the dashboard (Settings → Users) or the API (POST /users).

Invite a user:

Via the dashboard: Settings → Users → Invite user → enter name, email, role → copy the magic link → share with the user. The magic link is valid for 24 hours and allows the user to set their own password.

Via the API:

curl -X POST http://localhost:8001/users \
  -H "Authorization: Bearer <human-admin-jwt>" \
  -H "Content-Type: application/json" \
  -d '{"email": "bob@acme.com", "name": "Bob Jones", "role": "viewer"}'

Deactivate a user:

In Settings → Users, click the deactivate icon next to a user row. Deactivated users cannot log in but their audit records are preserved.

Password reset:

For locked-out active users: Settings → Users → Reset password → copy the new magic link and share. The user's old password remains valid until they complete the new magic link flow.

For CLI (root user or pre-UI users):

docker compose exec api python scripts/reset_password.py --email admin@company.com

User roles:

Organisation settings

Set organisation name and timezone in the dashboard (Settings → Organisation). Timezone applies to all datetime displays across the dashboard.

Via the API:

# Get current settings
curl http://localhost:8001/org-settings \
  -H "Authorization: Bearer <human-jwt>"

# Update
curl -X PUT http://localhost:8001/org-settings \
  -H "Authorization: Bearer <human-admin-jwt>" \
  -H "Content-Type: application/json" \
  -d '{"org_name": "Acme Corp", "timezone": "America/Chicago"}'

timezone must be a valid IANA timezone string. Changes take effect immediately across all logged-in dashboard sessions (no page reload required).

Human review workflow (HITL)

This workflow requires a Business or Enterprise license. On Community, a policy with effect review is enforced as deny instead — there's no Slack notification and no dashboard review queue to resolve it against, so the call fails closed rather than blocking forever.

When a policy with effect review matches a call_tool request on a Business/Enterprise license, the gateway returns a JSON-RPC result with isError: true immediately. In the background, AIControl posts a Block Kit message to the configured Slack channel.

The Slack notification contains:

• Agent name and tool name
• The reason the review was triggered
• Full tool parameters (first 300 characters)
• A review ID (first 8 characters, for reference)
• Green Approve and red Deny buttons

The reviewer clicks a button. Slack sends the click to AIControl's webhook handler, which:

1. Verifies the Slack signing secret (5-minute replay window)
2. Row-locks and looks up the review record by ID
3. Sets status to approved or denied
4. Records the reviewer's Slack user ID
5. Posts a confirmation message to the channel

The same row lock is used by PATCH /reviews/{review_id} on the dashboard side, so a Slack resolution and a dashboard resolution racing on the same review can't both go through — the second one to acquire the lock sees the row already resolved and returns 409 instead of silently overwriting the first decision.

Important: the gateway returns to the agent immediately — it does not wait for the Slack review. Your agent receives an isError: true result naming the pending review, but not a review_id field to poll directly — check the dashboard's Reviews page for pending items. See Integration for how to detect a review result.

Retrying after approval. If your agent retries the identical call (same session_id, tool name, and parameters) within 15 minutes of an approval — via Slack or the dashboard — the retry is automatically allowed instead of being routed to review again. A denied review is never auto-retried; only approved unblocks the next identical call.

Configuring Slack:

Set these in your .env file before starting the stack:

SLACK_BOT_TOKEN=xoxb-your-token
SLACK_SIGNING_SECRET=your-signing-secret
SLACK_REVIEW_CHANNEL=#aicontrol-reviews

Your Slack app requires the chat:write scope and the Interactivity feature enabled, with the Request URL set to http://your-host:8001/slack/actions.

If Slack is not configured, review decisions are still created in hitl_reviews — they just won't have a notification path. The GET /reviews/{id} endpoint still works for polling.

Monitoring

Health check:

curl http://localhost:8001/health
# {"status": "ok"}

Use this endpoint for load balancer health checks and uptime monitoring. No authentication required.

Structured logs:

AIControl uses structlog for all application logging. Logs are emitted as JSON to stdout. To capture them:

docker compose logs -f api

Each log line includes: timestamp, log level, event name, and contextual fields (agent_id, session_id, decision, duration_ms for gateway calls).

Docker's default json-file log driver has no rotation configured by docker-compose.yml — logs accumulate on the host indefinitely unless you set log-driver/log-opts (or a max-size/max-file policy) yourself, or ship logs to an external collector. AIControl doesn't configure either; it's a host-level Docker setting, not something the app controls.

Container health checks:

Both bundled containers other than the app itself define a Docker healthcheck: postgres runs pg_isready every 5s, api polls its own /health every 10s, and frontend curls itself every 10s. Check status with:

docker compose ps
# STATUS column shows "healthy" / "unhealthy" / "starting" per container

docker inspect --format='{{json .State.Health}}' <container_name> | jq .

An unhealthy postgres container blocks api from starting (the depends_on: condition: service_healthy in docker-compose.app.yml) — check docker compose logs postgres first; a full pgdata volume (see disk usage below) is a common cause. An unhealthy frontend is almost always an nginx config or startup issue — check docker compose logs frontend.

Disk usage:

Three named volumes grow over time and none has a bundled alert:

There's no built-in metric or /health field for any of these — check volume size directly:

docker system df -v | grep -E "pgdata|wal_data|reports_data"

# or, for the underlying host path (volume name is prefixed with your
# Compose project name, e.g. "aicontrol_wal_data" if you deployed from a
# directory named "aicontrol" — check the exact name first):
docker volume ls | grep wal_data
du -sh $(docker volume inspect <volume_name> -f '{{.Mountpoint}}')

A wal_data file that keeps growing across repeated checks — rather than staying near zero between drains — means the shipper isn't keeping up; check docker compose logs api for wal_shipper errors.

Alerting:

AIControl doesn't ship an integration for PagerDuty, Datadog, or Grafana — there's no built-in webhook or exporter beyond the SIEM export (Business/Enterprise, audit events only; see SIEM Export). To get paged on an operational problem, wire your own monitoring around the signals above yourself:

• Poll GET /health on an interval and alert on a non-200 response or a connection failure.
• Point your log collector (e.g. a Datadog or Grafana Agent sidecar reading docker compose logs, or a json-file log driver with a shipping plugin) at the api container's stdout and alert on evaluation_error or wal_shipper error events.
• Alert on docker compose ps health status (most container platforms and orchestrators can poll this natively) for postgres, api, and frontend.
• Alert on volume disk usage thresholds for pgdata and wal_data using your host's or cloud provider's disk-usage monitoring.

Dashboard:

The React dashboard at http://localhost:3000 shows:

• Audit log with search and filter
• Decision breakdown (allow / deny / review counts)
• Metrics and gateway call trend
• Policy manager
• Agent registry with approved tools editor
• Token management
• HITL review queue

Updating AIControl

docker compose -f docker-compose.yml -f docker-compose.app.yml pull
docker compose -f docker-compose.yml -f docker-compose.app.yml up -d
docker compose exec api alembic upgrade head

Migrations are backward-compatible. Run alembic upgrade head after every update.

Downtime during the up -d step is typically under 5 seconds. For zero-downtime updates, place a load balancer in front of the API container and perform a rolling restart.

Rolling back a failed update:

Every migration in migrations/versions/ ships a real downgrade(), so a schema rollback is a normal alembic downgrade:

# Roll the schema back one migration
docker compose exec api alembic downgrade -1

# Or to a specific revision (see `alembic history` for the list)
docker compose exec api alembic downgrade <revision>

If the image pull itself is the problem (a bad tag, a broken build) rather than the migration, roll the image back and leave the schema alone — pin image: in docker-compose.app.yml to the previous known-good tag (or git checkout that file to the prior commit) and re-run up -d:

docker compose -f docker-compose.yml -f docker-compose.app.yml up -d

Only run alembic downgrade if the new migration is actually the thing that broke — rolling back the image without rolling back a schema change the old image doesn't understand will itself cause errors. Take a backup (see Backup below) before rolling back either one.

Backup

Database backup:

# Full database dump
docker compose exec postgres pg_dump \
  -U aicontrol aicontrol > aicontrol_$(date +%Y%m%d).sql

# Audit events only
docker compose exec postgres pg_dump \
  -U aicontrol -t audit_events aicontrol > audit_events_$(date +%Y%m%d).sql

For production, use a managed PostgreSQL service with automated backups (see Using a managed PostgreSQL service), or configure WAL archiving on the bundled container yourself. The audit_events table is append-only and will grow continuously — plan storage accordingly.

Scheduling backups:

AIControl doesn't run a backup job itself — the pg_dump commands above are the whole mechanism against the bundled container. Schedule them with cron on the host running Docker:

# /etc/cron.d/aicontrol-backup — daily at 02:00, keep 14 days
0 2 * * * root docker compose -f /path/to/docker-compose.yml exec -T postgres \
  pg_dump -U aicontrol aicontrol | gzip > /var/backups/aicontrol/aicontrol_$(date +\%Y\%m\%d).sql.gz && \
  find /var/backups/aicontrol -name '*.sql.gz' -mtime +14 -delete

The 14-day retention in that example is arbitrary — pick a window based on your own recovery requirements and storage budget; it is not tied to the audit-event retention window described in Retention, which governs what's still in the live database, not how long you keep backup files.

Restore procedure:

# Stop the api container so nothing writes during restore
docker compose stop api

# Restore into a running (but otherwise idle) postgres container
gunzip -c aicontrol_20260101.sql.gz | docker compose exec -T postgres \
  psql -U aicontrol -d aicontrol

# Restart the api container
docker compose start api

A pg_dump plain-SQL dump (as produced by the commands above) restores with psql, not pg_restore — pg_restore only applies to dumps taken with -Fc/-Fd/-Ft. Restoring into a database that already has data will error on conflicting rows (primary keys, unique constraints) rather than overwrite them; restore into a freshly created empty database, or drop and recreate the aicontrol database first, if you're restoring over an existing install.

Policy backup:

Policies live in the policies table. They are included in a full database dump. You can also export them via the API:

curl http://localhost:8001/policies \
  -H "Authorization: Bearer <admin-token>" | jq . > policies_backup.json

Disaster recovery

AIControl has no built-in failover — the bundled stack is a single Postgres instance and a single api container on one host. Recovering from a lost pgdata volume, a corrupted database, or a dead host is a restore, not an automatic failover.

RPO (recovery point objective): bounded by your own backup cadence, not anything AIControl schedules for you. With the cron example above (daily at 02:00), worst case you lose up to 24 hours of audit and policy data. Tighten the RPO by backing up more frequently, or by moving to a managed PostgreSQL service with continuous/point-in-time recovery (see Using a managed PostgreSQL service) — the bundled container has no PITR of its own.

RTO (recovery time objective): the time to provision a replacement host (or none, if recovering in place), pull images, restore the latest dump (see Restore procedure above), and bring the stack back up — this is operational time on your side, not a number AIControl publishes, since it depends on your restore automation and dump size.

Recovering from a lost Postgres volume:

docker compose down
docker volume rm aicontrol_pgdata   # confirm the actual volume name first: docker volume ls
docker compose up -d postgres
# wait for postgres healthcheck to pass, then restore the latest dump
gunzip -c aicontrol_<latest>.sql.gz | docker compose exec -T postgres \
  psql -U aicontrol -d aicontrol
docker compose -f docker-compose.yml -f docker-compose.app.yml up -d

If you're on a managed PostgreSQL instance instead of the bundled container, follow your provider's own DR procedure (snapshot restore, cross-region replica promotion, etc.) — AIControl has no opinion on that path beyond pointing DATABASE_URL at whatever instance comes back.

Deployment topology

AIControl runs as a single-node Docker Compose stack. This is the supported production topology today.

Three containers:

Horizontal scaling of the api container is possible with a shared PostgreSQL instance and a load balancer — there's no sidecar process tying an API instance to a specific host. Multi-node and Kubernetes support is on the roadmap.

For high-availability requirements today, deploy on a host with sufficient resources (4 cores, 4GB RAM recommended) and point the stack at a managed PostgreSQL instance instead of the bundled container — see Using a managed PostgreSQL service below.

Sizing beyond the baseline: the 4-core/4GB number above (and the 4GB/10GB minimums in Getting Started) is a single baseline for evaluation and small production installs, not a formula. AIControl hasn't published load-test numbers for how it scales with agent count, call throughput, or audit volume — qualitatively:

• The api container is I/O-bound per request (Cedar evaluation is in-process and cheap); CPU pressure comes from request concurrency, not audit volume.
• postgres disk usage scales with audit volume × retention window, not agent count — a high-throughput agent on a 7-day community retention window uses less disk than a low-throughput agent on a 90-day enterprise retention window. See Disk usage in Monitoring above.
• Agent count itself has little effect on either — each registered agent is a small row plus its share of sessions/audit_events.

If you're sizing for a specific high-throughput or high-agent-count deployment, load-test your own workload against a staging instance before committing to a production host size — there's no substitute for that until AIControl publishes its own benchmarks.

Using a managed PostgreSQL service

AIControl can run against a managed PostgreSQL instance (AWS RDS, Google Cloud SQL, Azure Database for PostgreSQL) instead of the bundled postgres container. This is a manual reconfiguration — there's no install-time flag for it.

Procedure:

1. Provision a PostgreSQL 15 instance (match the major version of the bundled postgres:15 image) and create a database and user for AIControl.
2. Point DATABASE_URL in .env at the managed instance:
   Copy

   DATABASE_URL=postgresql+asyncpg://user:password@your-instance-host:5432/aicontrol

3. Remove the postgres service from docker-compose.yml and the depends_on: postgres block from the api service in docker-compose.app.yml — the api container otherwise waits on the bundled container's healthcheck and never starts.
4. Keep the aicontrol external network and the reports_data/wal_data volumes; those aren't Postgres-related.
5. Before first start, run migrations against the managed instance directly: alembic upgrade head (with DATABASE_URL pointed at it in your shell environment).

Network access: open the managed instance's security group / firewall to the host or subnet running the api container only. Do not expose it publicly.

TLS: the driver AIControl uses (asyncpg, via SQLAlchemy's async engine) does not accept libpq-style sslmode= query parameters. If your provider requires TLS, use asyncpg's own query parameter instead: ?ssl=true. Certificate-based or IAM-token authentication (e.g. RDS IAM auth) needs additional connect_args wiring that isn't currently present in app/models/database.py — treat that as unsupported until it's built.

Connection pooling: the app uses SQLAlchemy's default async pool with pool_pre_ping=True — there's no pgbouncer or external pooler in front of it. If your managed instance caps concurrent connections, account for that when sizing api replicas.

Verification status: this procedure follows the standard SQLAlchemy async engine + asyncpg connection path and should work against any PostgreSQL 15 endpoint. It has not been independently tested by AIControl against a live RDS, Cloud SQL, or Azure Database instance — verify connectivity, TLS negotiation, and failover behavior against your specific provider in staging before relying on it in production.

Network requirements

AIControl's default Docker Compose stack exposes three ports and needs no outbound access for governance itself — Cedar evaluates in-process, so there is no policy-engine sidecar to reach over the network.

Outbound (egress) requirements:

No other outbound access is required. If your network team blocks egress by default, allow only the destinations above that apply to your configuration.

Troubleshooting

verify.sh fails:

bash diagnose.sh

Send the output to hello@aictl.io.

Migrations not applied:

docker compose exec api alembic current    # show current revision
docker compose exec api alembic upgrade head

Token rejected (401):

• Confirm the token was issued for the agent role with an agent_id for gateway calls
• Confirm the token has not been revoked (check the Tokens tab in the dashboard)
• Confirm Authorization: Bearer <token> header is present with no extra whitespace

Dashboard not loading:

docker compose logs frontend    # check for nginx errors
docker compose restart frontend

Support: hello@aictl.io

Environment Variables

React Dashboard

URL: http://localhost:3000

VITE_* environment variables are baked into the frontend image at build time. After changing any VITE_* variable in .env, rebuild and restart:

docker compose -f docker-compose.yml -f docker-compose.app.yml build frontend
docker compose -f docker-compose.yml -f docker-compose.app.yml up -d frontend

Local development (with hot reload):

cd frontend && npm install && npm run dev

Business / Enterprise License Setup

Licensing is not an environment variable or a restart — it's an activation code entered in the dashboard, validated and kept in sync against billing.aictl.io. See Getting Started → Business/Enterprise walkthrough for the full step-by-step; summary:

1. Contact hello@aictl.io from the pricing page for Business or Enterprise, or start the self-service $1, 90-day Enterprise trial directly on the pricing page — either way you get a one-time activation code by email.

2. Open Dashboard → Billing and submit the code. This calls PUT /settings/license-activation (admin-only), which validates the code against billing.aictl.io before storing it in the org_settings table — an invalid code is rejected and nothing is saved.

3. Licensed sections unlock immediately on a successful activation — no restart, no frontend rebuild. The dashboard reads the current plan from GET /license-info at runtime.

4. A daily background job re-syncs license_status/license_plan from billing.aictl.io using the stored activation code. If billing.aictl.io is unreachable, the last-known status is kept for up to 7 days before the license is forced to an unreachable/inactive state (fail-closed). A trial that reaches its 90-day limit without upgrading syncs back as expired (shown in the dashboard as "Trial expired"), reverting the org to Community — distinct from canceled, which shows a reactivate-via-portal banner that doesn't apply to a trial simply running out.

5. For AI-native features (compliance reports, Enterprise only), add your LLM key:

   bashCopy

   LLM_API_KEY=your-anthropic-or-openai-key

   This does require restarting the stack, since LLM_API_KEY is read from .env at process start, unlike the license itself.

What a Business license unlocks:

• Slack HITL notifications
• In-dashboard HITL review queue (Reviews page)
• Live pending review badge in sidebar
• Audit log — 30-day retention

What an Enterprise license unlocks additionally:

• Session view / drill-down
• Audit log CSV export
• Audit log — 90-day retention
• Compliance report export — EU AI Act, NIST AI RMF, SOC 2, ISO 42001 (Reports page)
• Natural-language policy authoring — draft, simulate, activate (Policies page)
• Notification bar: deny rate alerts

Billing and usage

GET /license-info (public, no auth) returns your current plan (community/business/enterprise), company name, and license expiry — it's what the frontend reads to decide what to show as locked versus available before you're even logged in.

GET /billing/usage (admin, human JWT required) returns your plan's flat annual_price_usd, retention_days, and cascaded feature list, plus a gateway call count (intercepts, every allowed or denied call counted from audit_events) for the current and previous calendar month.

This is a self-service usage view, not a live invoice — AIControl doesn't run automated metering or payment collection today. If an activation code is set, manage_subscription_url is populated (fetched live from billing.aictl.io's customer portal) so an admin can manage the subscription directly. upgrade_url is populated only for a trial-plan org — it points at a Stripe Checkout session (business plan) via billing.aictl.io's activation-code-keyed upgrade endpoint. Business and Enterprise customers manage or change their plan through manage_subscription_url (the Stripe customer portal) instead, or by contacting hello@aictl.io. On a community install with no activation code, both URLs are null.