Research-backed calculator guide

Routing policy as code: version, review and roll back provider decisions

Learn how to model LLM routing decisions as versioned policy, review provider constraints, estimate routed cost, and roll back safely when quality or spend changes.

What “policy as code” means for LLM routing

An LLM router decides where a request goes. A routing policy defines the boundaries within which that decision is allowed. For example, a request may be eligible for a low-cost model when it is a short classification task, but ineligible when the payload contains a regulated data tag or the caller requires a specific tool capability. The router then selects among the eligible options.

The distinction matters because routing code often mixes business rules, provider adapters, and transient operational behavior. A direct edit can silently change spend, data residency, response quality, or availability. A policy file makes those decisions reviewable before deployment. It also gives finance, security, and application owners a common object to inspect.

The exact representation can be YAML, JSON, Rego, or another typed format. The syntax is less important than the contract. Cloudflare’s Dynamic Routing documentation describes named, versioned flows with conditional nodes, percentage splits, rate limits, budget limits, fallbacks, and instant rollback of deployed versions.^[1] That is a useful model for the lifecycle even if your own router is built differently.

A policy should therefore separate stable intent from provider-specific syntax. “Do not send restricted data outside the approved region” is an intent. The provider adapter can translate that intent into an OpenRouter data_collection or zdr preference, an endpoint choice, or a local deployment constraint. When the provider API changes, the adapter can change without rewriting the approval logic.

A minimal policy contract

Start with a small contract that reviewers can understand. The following is illustrative configuration, not a vendor schema:

apiVersion: routing.agentcost/v1
metadata:
  name: support-default
  owner: ai-platform
  changeTicket: FIN-1842
spec:
  eligibility:
    allowedProviders: [openai, anthropic]
    dataClass: internal
    allowedRegions: [us]
    requiredCapabilities: [json]
  routes:
    - when: request.task == "classification" && request.tokens <= 1200
      choose: [fast-json]
    - when: request.risk == "high"
      choose: [reviewed-reasoning]
    - when: true
      choose: [balanced-default]
  controls:
    maxInputUsdPerMillion: 1.00
    maxOutputUsdPerMillion: 5.00
    timeoutMs: 8000
    fallback: balanced-default
    onFallbackFailure: fail_closed
  evidence:
    required: [policyVersion, decisionReason, provider, model, outcome]

The important fields are not the names. They are the review questions they answer:

• What can be used? allowedProviders, regions, data class, and capabilities.
• Why was it selected? An ordered rule or an explicit score and reason.
• What is the economic boundary? Maximum accepted rates, request budgets, or a monthly allocation.
• What happens when the preferred path fails? A named fallback and a terminal behavior.
• Can the decision be reconstructed? A policy version and normalized request metadata.

Do not put API keys, customer content, or unbounded prompt text in the policy repository. Reference secrets through the runtime secret manager. Hash or classify sensitive inputs when a decision needs evidence but the raw value is not appropriate to retain.

For schema-heavy systems, use machine-readable contracts rather than relying on comments. The vLLM Semantic Router documentation exposes a versioned configuration contract, named sections, algorithms, signals, and plugins, and supports progressively discovering only the section needed for an edit.^[3] That pattern reduces accidental changes to unrelated parts of a large router configuration.

Version, review, and deploy the policy

A practical change lifecycle has five gates.

1. Create a small, reviewable diff

Change one decision dimension at a time where possible. A provider order change, a new fallback, and a new data policy in one diff make it difficult to attribute an outcome. Include the policy version, owner, change reason, and expected impact in the commit or pull request.

2. Validate syntax and semantics

Schema validation catches missing fields and invalid types. Semantic validation catches unsafe combinations. Examples include a fallback that violates the request’s data class, a model that does not support the required response format, or a maximum price below every eligible provider. Fail the check rather than silently dropping an unsupported field.

3. Test decisions with a fixture set

Create fixtures that represent real traffic slices: short classification, long retrieval, tool use, restricted data, provider timeout, and an exhausted budget. Assert the selected route, the reason, and the permitted fallback. Keep expected outputs in version control. A policy test should not require a live paid provider call.

4. Replay or shadow before cutover

Run the proposed policy against sampled historical metadata or a shadow stream. Compare route mix, estimated cost, latency, error rates, and quality scores against the current policy. Shadow results are estimates until they are validated against production meters. Do not report a projected saving as an achieved saving.

5. Deploy an immutable version

Publish a version identifier with the runtime configuration. A request log should record that identifier, not just a mutable policy name. Cloudflare describes saving route versions, deploying one to make it live, and rolling back to earlier versions.^[1] Whether your platform has the same UI or uses Git plus a deployment controller, preserve that separation between draft, approved, and active versions.

Provider controls belong inside the policy boundary

Provider selection is more than “cheapest model wins.” A request can require a data policy, a parameter, a throughput floor, a latency ceiling, or a known tool behavior. OpenRouter’s provider-routing documentation illustrates this with request-level controls including ordered providers, fallback permission, parameter support, data-collection preference, zero-data-retention filtering, provider allow/deny lists, sorting, throughput and latency preferences, and maximum price.^[2]

Use those controls as inputs to a decision contract rather than sprinkling them through callers. For example, a privacy-sensitive route may set an approved provider list and a retention constraint. A latency-sensitive route may require a measured maximum latency and use a different fallback. A cost-capped route may reject providers above a stated maximum rather than falling through to an unbounded alternative.

Record the difference between policy eligibility and runtime availability. A provider can be allowed by policy but temporarily unavailable. Conversely, a provider can be healthy but prohibited for a particular data class. That distinction is essential when analyzing fallback rates: a policy rejection is not the same event as a network failure.

Also define whether retries are allowed to create a second billable attempt. If the provider may charge a request that times out locally, count the attempt in your cost ledger. Your AI API invoice reconciliation calculator is useful after the fact, but the router should emit an attempt identifier and policy version at request time.

Worked cost example

Assume a monthly workload of 100,000 requests. Each request averages 2,000 input tokens and 500 output tokens. The following are deliberately hypothetical planning rates, shown in USD per 1 million tokens; they are not vendor quotes and are not current provider pricing:

• Fast route: $0.20 input and $0.80 output.
• Balanced route: $1.00 input and $4.00 output.
• Reasoning route: $5.00 input and $15.00 output.
• Policy evaluation overhead: $0.0002 per request.
• Fallback exposure: 3% of requests that first use the balanced route make a second attempt on the reasoning route. Assume the first attempt is still billable.

The token cost per request is calculated as (input tokens / 1,000,000 × input rate) + (output tokens / 1,000,000 × output rate).

The same token mix sent entirely to the reasoning route would cost 100,000 × $0.0175 = $1,750, before policy overhead. The illustrative routed plan is therefore $347.25 per month, compared with $1,770.00 for the all-reasoning baseline after the same $20 policy overhead. The difference is $1,422.75, or about 80.4% in this constructed scenario.

That percentage is not a promise. It is sensitive to route mix, token lengths, provider rates, fallback frequency, and whether failed attempts are billed. If the policy sends more difficult work to the reasoning route, or if a classifier adds another model call, the result changes. Recalculate with your measured input/output meters; the AI token cost calculator can help normalize those units.

Rollback rules that prevent guesswork

Rollback should be a defined decision, not an emergency argument. Set thresholds before release, such as a statistically meaningful quality decline, a sustained increase in fallback rate, an error-budget breach, or a cost-per-successful-response limit. Compare like with like: p95 latency against the same traffic slice, quality against the same evaluation set, and cost per successful outcome rather than raw request cost alone.

Use a fast rollback when the new policy violates a hard constraint: restricted data reaches a disallowed provider, a required capability fails, or the route exceeds a safety budget. For softer regressions, pause the rollout, inspect route-level evidence, and decide whether to adjust weights, eligibility, or the provider adapter.

A rollback restores the previous policy version; it does not undo provider charges or erase evidence. Keep both the failed version and its logs. Post-incident review should ask whether the policy test was missing a fixture, whether the provider’s observed behavior changed, or whether the cost model omitted retries and overhead.

FAQ

Is an LLM routing policy the same as a model router?

No. The router is the runtime mechanism that selects a destination. The policy is the approved set of rules and constraints that the router must obey. A policy can permit several routing algorithms, while the router executes one of them for a request. Keeping the two separate makes approvals, testing, and rollback clearer.

Should routing rules live in application code or a gateway?

Put shared provider, privacy, budget, and fallback rules at a gateway or policy layer when several applications need the same behavior. Application code can still express task context, such as a risk label or required capability. Centralizing the shared rules avoids every service maintaining a different provider list. The trade-off is that the gateway becomes an important dependency, so its own availability and versioning need monitoring.

How do I review a provider routing configuration schema?

Review it in three passes: schema validity, policy semantics, and operational evidence. Confirm that fields have explicit types and defaults; then check that every route has an eligible provider, safe fallback, and bounded cost; finally verify that logs can identify the policy version, reason, provider, model, attempt, and outcome. A schema browser or progressive section API is helpful for large contracts, but it does not replace domain review.^[3]

When should a policy fail closed?

Fail closed when the request would violate a hard requirement and no approved alternative is known. Examples include missing data-consent metadata, no provider satisfying a residency rule, an unsupported required tool parameter, or an exhausted hard budget. For a non-critical latency preference, a controlled fallback may be preferable. Make the distinction explicit so an outage does not silently become a governance exception.

How often should routing policies be re-costed?

Re-cost whenever provider prices, cache treatment, tokenization, traffic mix, fallback behavior, or model capabilities change. A monthly review is a reasonable minimum for a material production route, but event-driven review is safer after a provider change or a large workload shift. Keep the assumptions with the policy version so a later invoice comparison can explain why the estimate differed from actual spend.