Routing & Reliability

What it is

Prism’s routing layer distributes requests across multiple providers and models to maximize reliability and performance. If one provider is down or slow, traffic automatically shifts to healthy alternatives. This ensures your application stays responsive even when individual providers experience outages or rate limiting.

Use cases

• High availability — Automatic failover to backup providers when primary is down or rate-limited
• Cost optimization — Route to the cheapest provider that supports the requested model
• Latency reduction — Route to the fastest provider based on recent response times
• Traffic distribution — Split traffic across providers by weight for capacity management

Key concepts

Configuration parameters

These parameters appear in the JSON configuration blocks throughout this page.

Failover:

Retries:

Circuit breaker:

Timeouts:

Routing strategies

Configuring a routing policy

from prism import Prism

client = Prism(
    api_key="sk-prism-your-key",
    base_url="https://gateway.futureagi.com",
    control_plane_url="https://api.futureagi.com",
)

# Create a weighted routing policy
policy = client.routing.create(
    name="Production routing",
    strategy="weighted",
    config={"weights": {"openai": 70, "anthropic": 30}},
    description="70/30 split between OpenAI and Anthropic",
)

# List all routing policies
policies = client.routing.list()

# Update an existing policy
client.routing.update(
    policy["id"],
    strategy="least-latency",
    config={"providers": ["openai", "anthropic", "gemini"], "failover_on": [429, 500, 502, 503, 504]},
)

import { Prism } from "@futureagi/prism";

const client = new Prism({
  apiKey: "sk-prism-your-key",
  baseUrl: "https://gateway.futureagi.com",
  controlPlaneUrl: "https://api.futureagi.com",
});

const policy = await client.routing.create({
  name: "Production routing",
  strategy: "weighted",
  config: { weights: { openai: 70, anthropic: 30 } },
  description: "70/30 split between OpenAI and Anthropic",
});

const policies = await client.routing.list();

await client.routing.update(policy.id, {
  strategy: "least-latency",
  config: { providers: ["openai", "anthropic", "gemini"], failoverOn: [429, 500, 502, 503, 504] },
});

Failover

Failover triggers on specific HTTP status codes and error conditions: 429 (rate limit), 5xx (server errors), timeouts, and connection errors. The providers array defines the failover order. When the primary provider fails, Prism automatically routes to the next provider in the list.

{
  "failover": {
    "enabled": true,
    "providers": ["openai", "anthropic", "gemini"],
    "failover_on": [429, 500, 502, 503, 504]
  }
}

Retries

Prism uses exponential backoff for retries. This means it waits progressively longer between each retry attempt — for example, 100ms, then 200ms, then 400ms. This gives struggling providers time to recover instead of flooding them with rapid retry requests.

{
  "retries": {
    "max_retries": 3,
    "initial_backoff_ms": 100,
    "max_backoff_ms": 10000,
    "backoff_multiplier": 2
  }
}

Circuit breaking

Think of a circuit breaker like a fuse box. When a provider starts failing repeatedly, the circuit “trips” — Prism stops sending requests to that provider entirely and routes to healthy alternatives instead. After a recovery window, Prism tests the provider with a few trial requests. If those succeed, the circuit “closes” and normal routing resumes. This prevents a single failing provider from degrading your entire application.

Circuit breaking prevents cascading failures by stopping requests to a provider that is experiencing issues. The circuit breaker has three states: Closed (normal operation), Open (rejecting requests), and Half-Open (testing recovery).

{
  "circuit_breaker": {
    "enabled": true,
    "error_threshold_percent": 50,
    "min_requests": 10,
    "open_duration_seconds": 60,
    "half_open_max_requests": 3
  }
}

Timeouts

Configure per-request and per-provider timeouts to prevent hanging requests.

{
  "timeouts": {
    "request_timeout_seconds": 30,
    "provider_timeout_seconds": 25
  }
}

Example: High-availability setup

This configuration combines weighted routing, failover, retries, and circuit breaking for a production-grade setup:

{
  "name": "Production HA",
  "strategy": "weighted",
  "config": {
    "weights": {
      "openai": 60,
      "anthropic": 30,
      "gemini": 10
    },
    "failover": {
      "enabled": true,
      "providers": ["openai", "anthropic", "gemini"],
      "failover_on": [429, 500, 502, 503, 504]
    },
    "retries": {
      "max_retries": 3,
      "initial_backoff_ms": 100,
      "max_backoff_ms": 10000,
      "backoff_multiplier": 2
    },
    "circuit_breaker": {
      "enabled": true,
      "error_threshold_percent": 50,
      "min_requests": 10,
      "open_duration_seconds": 60,
      "half_open_max_requests": 3
    },
    "timeouts": {
      "request_timeout_seconds": 30,
      "provider_timeout_seconds": 25
    }
  }
}

What you can do next