Keystone 1.0

A model name in your code goes stale the day the next model ships. With ten services, you rewrite it ten times, and spread vendor keys across ten configs.

Keystone. The Model Switchboard. (KTMS) takes that out of your code. A service says how hard the task is; the gateway decides which model to call, with which key, and whether your own GPU should take it. It idles at 7 MiB of memory, and from 1.0 on its API doesn’t break within 1.x.

The web console ktms-admin and the usage meter light-tokenmeter reach 1.0 with it. All three are written in Go and licensed under Apache 2.0. The features below are as of the current 1.3.

What Keystone does

Keystone sits between your services and your model vendors. Your services call one endpoint, and KTMS:

  • keeps the vendor API keys, encrypted in Postgres with AES-256-GCM, so clients never see them;
  • picks the model from rules, not from a name hardcoded in your code;
  • falls back to the next model when a vendor fails or throttles;
  • enforces requests per minute, daily token budgets and hard spend caps per key, user and team;
  • sends one usage event per call to a sink such as light-tokenmeter.

KTMS architecture

Effort instead of model names

A service doesn’t ask for gpt-4o. It asks for an effort, LOW, MEDIUM or HIGH, and adds free-form attributes such as language or task. An ordered list of rules maps that to a chain of models:

{"rules":[
  {"name":"spanish-high", "priority":30, "effort":"EFFORT_HIGH",
   "attributes":{"language":"es"}, "use":["gemini-2.5-pro","gpt-4o"], "enabled":true},
  {"name":"low-default", "priority":20, "effort":"EFFORT_LOW",
   "use":["deepseek-chat","gpt-4o-mini"], "enabled":true},
  {"name":"anything-else", "priority":1, "use":["gpt-4o-mini"], "enabled":true}
]}

Rules change at runtime over the admin API or the console. When a new model comes out, you edit a rule, not forty services.

Two ways in

Typed gRPC (ktms.v1) for services, and an OpenAI-compatible HTTP API for everything that already speaks OpenAI:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="ktms_<your key>")
reply = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Say hi in three words."}],
)
print(reply.choices[0].message.content, reply.model)

Response headers (X-KTMS-Vendor, X-KTMS-Rule, X-KTMS-Placement) say who answered and why.

What it does (as of 1.3)

  • Routing: effort LOW/MEDIUM/HIGH plus attributes, prioritized rules changed at runtime, fallback chains, model: "auto" with a judge (250 ms, falls back to MEDIUM), canaries sticky per user, shadow traffic with evaluation, a dry run with ktms route explain.
  • Reliability: circuit breaker, a model paused on Retry-After, one attempt is one vendor call, defined behaviour when Postgres or Redis fail.
  • Money and limits: RPM/TPM and a model allowlist per key, daily and monthly spend caps per key and user, a monthly budget per team, prompt-cache pricing, vendor Batch APIs.
  • Your own GPUs: a node agent for Ollama, vLLM, TGI and llama.cpp, admission by real queue depth, “own nodes only” placement, capacity levels and load schedules, a local usage log with ktms-node stats.
  • Security: vendor keys in Postgres under AES-256-GCM with rotation, PII masking, prompt-injection detection, latent guardrails and a semantic cache (experimental).
  • API: typed gRPC ktms.v1 and an OpenAI-compatible HTTP API, streaming, tool calling, lossless JSON Schema, embeddings, speech, images, Judge.
  • Management: the ktms CLI with contexts and ktms init, configuration as YAML (plan/apply), an MCP server (ktms mcp) for AI assistants, a Claude Code plugin, an admin:read scope, and the ktms-admin console.
  • Operations: ktms_* metrics with suggested alerts, a Helm chart on OCI, signed images with SBOM and SLSA provenance, an upgrade test on every pull request.

Footprint

Keystone is one Go process with no interpreter. On our stand of eleven self-hosted gateways (Google Cloud c3-standard-4, 4 vCPU, mock upstream, two runs), it used about 7 MiB idle and 20 MiB at 200 RPS, added about 0.64 ms at p50, sustained 4,000 RPS, and answered its first request about 170 ms after docker start. agentgateway, APISIX and Kong add less latency; only agentgateway matches the memory. The full write-up is in Go vs Python for an LLM gateway.

What changed on the way to 1.0

  • Organizations and teams. Team keys share a model allowlist and a monthly budget that the gateway enforces across all of them.
  • Vendor calls you can reason about. Vendor SDK retries are off, so one router attempt is one vendor call. Retry-After pauses a model for that vendor credential, and the chain moves on.
  • Defined behaviour when a dependency fails. With Postgres down, keys verified in the last five minutes keep working and routing serves from the last config snapshot. With Redis down, rate limits fail open, and you choose what spend caps do. Usage events queue and are delivered once.
  • Lossless JSON Schema. enum, format, nullable and additionalProperties reach the vendor. Where a vendor can’t enforce a schema, Keystone checks the reply, retries once and falls back.
  • Local nodes by real queue depth. The node agent reports the engine’s own queue (vLLM, TGI, llama.cpp), and admission skips a node where the request would wait too long.
  • Safe key rotation. RotateEncryption re-encrypts vendor keys and node tokens in one locking transaction, and the full rotation runs against Postgres in CI.
  • Signed releases. Images and checksums.txt are signed with keyless cosign; images carry an SBOM and SLSA provenance; govulncheck runs on every pull request.
  • Apache 2.0 instead of MIT, which adds an explicit patent grant.

The neighbours

ktms-admin manages models, rules, keys and vendor credentials in the browser, and has a playground for trying a prompt through your rules. Sign-in is a password, Google or any OpenID Connect provider, GitHub, or the access proxy you already run, limited to an allowlist.

light-tokenmeter answers “who is spending our tokens, and are they worth it?”: one event per call in, summaries per project, user, action, vendor and model out. The console labels each group greedy or worthy from those summaries.

Routing rules in ktms-admin and spend in light-tokenmeter

Try it

git clone https://github.com/ai-process/ktms.git && cd ktms
OPENAI_API_KEY=sk-... docker compose up -d --build
curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer ktms_dev_default_key" -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Say hi in three words."}]}'

Upgrading from 0.14.x is one step. From older 0.x releases, upgrade to 0.14.x first.

What 1.0 promises

0.x made no compatibility promise. From 1.0 on, nothing below changes incompatibly within 1.x:

Surface What stays compatible
gRPC ktms.v1 No RPC, field, enum value or message is removed or renamed; field numbers and types stay.
HTTP /v1 and X-KTMS-* headers Paths, request and response fields, header names.
Environment variables Names, defaults and meaning.
Database Any 1.x release migrates the database of an earlier one by itself. No manual SQL.
Metrics, usage events, node agent ktms_* names and labels, usage.v1 fields, agent and gateway compatible both ways.
Error contract Status codes and reason strings keep their meaning.

Some features work and are supported but may still change in a minor release, and say so with an Experimental: comment in the proto: the judge’s effort choice for model: "auto", shadow traffic and its evaluation, the semantic cache and latent guardrails, JudgeStream and EvaluateRollouts, and the YAML configuration format and the ktms command until the format reaches v1.

How the promise is checked

A promise nobody tests is a wish. Every pull request and every release runs two gates:

  • buf breaking compares the proto with the previous release.
  • An upgrade test starts the previous release and the oldest release of the same major, lets them write a database, migrates it with the new build, checks that the new build serves it, and then runs the old release again on the migrated database. That last step is what makes rolling upgrades safe.

The release workflow publishes binaries only after both pass, and Docker images only after the binaries.

Next

Routing sensitive data to your own nodes only, hedged requests, a corrective retry for JSON, and a Grafana dashboard. None of it breaks the 1.x API.