Manage Providers
Add, configure, and manage LLM providers in Prism.
What it is
Providers are the LLM services Prism routes requests to. Each has its own API format, authentication, and model catalog. Prism translates between its unified OpenAI-compatible API and each provider’s native format, allowing you to switch providers without changing client code.
Use cases
- Multi-provider redundancy — Configure backup providers for automatic failover
- Cost optimization — Route to cheaper providers for appropriate workloads
- Self-hosted models — Connect Ollama, vLLM, or LM Studio running on your infrastructure
- Provider evaluation — Test multiple providers with the same prompts to compare quality
Supported providers
| Provider | API Format | Supported Types | Models |
|---|---|---|---|
| OpenAI | Native OpenAI API | Chat, Completions, Embeddings | gpt-5.4, gpt-5.4-mini, gpt-4.1, gpt-4o, o3, o4-mini, and all other OpenAI models |
| Anthropic | Anthropic Messages API† | Chat | claude-opus-4-6, claude-sonnet-4-6, claude-haiku-4-5, and all other Claude models |
| Google Gemini | Google GenAI API† | Chat | gemini-3.1-pro, gemini-3-flash, gemini-2.5-pro, gemini-2.5-flash, and all other Gemini models |
| Mistral | OpenAI-compatible | Chat, Completions | mistral-large-latest, mistral-medium-latest, mistral-small-latest, and all other Mistral models |
| Cohere | Cohere API† | Chat, Completions | command-a-03-2025, command-r7b-12-2024, command-r-plus-08-2024, and all other Cohere models |
| Ollama | OpenAI-compatible | Chat, Completions | Any local model |
| vLLM | OpenAI-compatible | Chat, Completions | Any model supported by vLLM |
| LM Studio | OpenAI-compatible | Chat, Completions | Any model loaded in LM Studio |
Note
† Anthropic, Google Gemini, and Cohere each have their own native API formats. Prism automatically translates requests and responses between its unified OpenAI-format API and each provider’s native format. Your client code stays identical regardless of which provider handles the request.
Tip
Prism supports all models from each provider, including new releases. The models listed above are examples of the latest available models as of March 2026.
Adding a provider
- Open Prism dashboard at https://app.futureagi.com/dashboard/gateway/providers
- Click Add Provider
- Select provider from list
- Enter API key and optional configuration
- Click Save
from prism import Prism
client = Prism(
api_key="sk-prism-your-key",
base_url="https://gateway.futureagi.com",
control_plane_url="https://api.futureagi.com",
)
# Add a provider
provider = client.org_configs.create(
provider="openai",
api_key="sk-your-openai-key",
config={"organization": "your-org"}
)
# List all providers
providers = client.org_configs.list()
# Update provider configuration
client.org_configs.update(
provider["id"],
config={"organization": "new-org"}
)import { Prism } from "@futureagi/prism";
const client = new Prism({
apiKey: "sk-prism-your-key",
baseUrl: "https://gateway.futureagi.com",
controlPlaneUrl: "https://api.futureagi.com",
});
const provider = await client.orgConfigs.create({
provider: "openai",
apiKey: "sk-your-openai-key",
config: { organization: "your-org" }
});
const providers = await client.orgConfigs.list();
await client.orgConfigs.update(provider.id, {
config: { organization: "new-org" }
});Warning
Provider API keys are stored encrypted and never exposed in API responses.
Tip
Custom models can be configured via the Custom & Self-Hosted section or through org config.
Switching providers at request time
Use the same code with different model names to switch providers:
Python SDK:
# OpenAI
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Hello"}]
)
# Anthropic
response = client.chat.completions.create(
model="anthropic/claude-haiku-4-5",
messages=[{"role": "user", "content": "Hello"}]
)
# Google Gemini
response = client.chat.completions.create(
model="gemini/gemini-2.0-flash",
messages=[{"role": "user", "content": "Hello"}]
)TypeScript SDK:
// OpenAI
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Hello" }]
});
// Anthropic
const response = await client.chat.completions.create({
model: "anthropic/claude-haiku-4-5",
messages: [{ role: "user", content: "Hello" }]
});
// Google Gemini
const response = await client.chat.completions.create({
model: "gemini/gemini-2.0-flash",
messages: [{ role: "user", content: "Hello" }]
});Custom & Self-Hosted providers
Connect self-hosted models running on your infrastructure using Ollama, vLLM, or LM Studio.
- Open Prism dashboard at https://app.futureagi.com/dashboard/gateway/providers
- Click Add Provider
- Enter your model’s public endpoint URL
- Enter model name
- Click Save
from prism import Prism
client = Prism(
api_key="sk-prism-your-key",
base_url="https://gateway.futureagi.com",
control_plane_url="https://api.futureagi.com",
)
# Add Ollama provider
provider = client.org_configs.create(
provider="ollama",
config={
"base_url": "https://your-ollama-endpoint.example.com",
"model": "llama3.2"
}
)
# Add vLLM provider
provider = client.org_configs.create(
provider="vllm",
config={
"base_url": "https://your-vllm-endpoint.example.com",
"model": "meta-llama/Llama-3.1-8B-Instruct"
}
)import { Prism } from "@futureagi/prism";
const client = new Prism({
apiKey: "sk-prism-your-key",
baseUrl: "https://gateway.futureagi.com",
controlPlaneUrl: "https://api.futureagi.com",
});
// Add Ollama provider
const provider = await client.orgConfigs.create({
provider: "ollama",
config: {
baseUrl: "https://your-ollama-endpoint.example.com",
model: "llama3.2"
}
});
// Add vLLM provider
const provider = await client.orgConfigs.create({
provider: "vllm",
config: {
baseUrl: "https://your-vllm-endpoint.example.com",
model: "meta-llama/Llama-3.1-8B-Instruct"
}
});Note
Your self-hosted endpoint must be publicly accessible from Prism’s gateway. Use a tunnel service (ngrok, Cloudflare Tunnel), a cloud VM with a public IP, or deploy behind a reverse proxy with a public domain.
Provider health
Prism automatically monitors provider health and availability. Unhealthy providers are automatically excluded from routing until they recover. The system tracks:
- Response times and latency
- Error rates and failure patterns
- Rate limit status
- Availability and uptime
When a provider becomes unhealthy, Prism triggers failover to healthy alternatives and opens the circuit breaker to prevent cascading failures. Alerts notify you of provider issues so you can investigate and resolve problems.