Reduce LLM Costs With Semantic Caching

Point the OpenAI SDK at the gateway and send a request. The response headers tell you exactly what it cost and whether it came from cache.

import os
from openai import OpenAI

API_KEY = os.environ["AGENTCC_API_KEY"]

client = OpenAI(
    api_key=API_KEY,
    base_url="https://gateway.futureagi.com/v1",
)

r = client.chat.completions.with_raw_response.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What is your return policy?"}],
)
print(f"cache:   {r.headers.get('x-agentcc-cache')}")
print(f"cost:    ${r.headers.get('x-agentcc-cost')}")
print(f"latency: {r.headers.get('x-agentcc-latency-ms')}ms")

You should see x-agentcc-cache empty or miss on a fresh call. The cost and latency are what you’d pay every time without caching.

In the dashboard, go to Gateway → Providers → Cache and click Configure Cache. Toggle:

• Enable Response Cache: on
• Default TTL: 1h (or whatever fits your data freshness needs)

Save. Caching is now active for every request through the gateway. The dashboard shows Enabled: Yes with L1 Backend: memory, and Semantic Cache: Disabled confirms only exact matches are served right now. No client change required.

Run the same prompt twice:

prompt = [{"role": "user", "content": "What is your return policy?"}]

r1 = client.chat.completions.with_raw_response.create(model="gpt-4o-mini", messages=prompt)
print(f"call 1: {r1.headers.get('x-agentcc-cache')} | ${r1.headers.get('x-agentcc-cost')}")

r2 = client.chat.completions.with_raw_response.create(model="gpt-4o-mini", messages=prompt)
print(f"call 2: {r2.headers.get('x-agentcc-cache')} | ${r2.headers.get('x-agentcc-cost')}")

You should see call 1 come back miss. Call 2 comes back hit_exact, instant, with $0 provider cost. Exact caching is fast and free, but only helps when prompts are byte-identical.

Real customers don’t ask the same question the same way twice. Semantic caching matches prompts by meaning rather than exact text. It runs as an L2 fallback after the L1 exact-match check.

In the same Configure Cache dialog, enable:

• Enable Semantic Cache: on
• Similarity Threshold: 0.92 (similarity, 0 to 1, higher is stricter)

The same client code now matches paraphrases:

prompts = [
    "What is your return policy?",
    "Can I return a product I bought?",
    "How do refunds work at your store?",
    "Tell me about returning items.",
]
for p in prompts:
    r = client.chat.completions.with_raw_response.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": p}],
    )
    print(f"{(r.headers.get('x-agentcc-cache') or 'miss'):14} | ${r.headers.get('x-agentcc-cost')} | {p}")

You should see the first prompt come back miss. The rest are paraphrases above the 0.92 similarity threshold and come back as hit_semantic with near-zero cost.

Loop over a realistic mixed batch and tally cache hits, total cost, and latency.

import time
from collections import Counter

batch = [
    "What is your return policy?",
    "Can I return a product?",
    "How do I get a refund?",
    "What's the shipping cost?",
    "How long does shipping take?",
    "Do you ship internationally?",
] * 5  # 30 calls total

tally = Counter()
total_cost = 0.0
start = time.time()

for p in batch:
    r = client.chat.completions.with_raw_response.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": p}],
    )
    tally[r.headers.get("x-agentcc-cache") or "miss"] += 1
    total_cost += float(r.headers.get("x-agentcc-cost") or 0)

print(f"cache results: {tally}")
print(f"total cost:    ${total_cost:.5f}")
print(f"wall time:     {time.time() - start:.1f}s")

For this illustrative batch, expect most calls to land as hit_exact or hit_semantic after the first pass over each unique question, with only the six first-seen prompts costing full price. Compare the total cost against the same batch with caching disabled. That’s your savings, and it scales with how repetitive your real traffic is.

When you change a system prompt or want a fresh response for a specific call, send x-agentcc-cache-force-refresh: true on that request. The gateway skips the cache read but still writes the new response back, so subsequent calls hit the refreshed entry.

r = client.chat.completions.with_raw_response.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What is your return policy?"}],
    extra_headers={"x-agentcc-cache-force-refresh": "true"},
)
print(f"forced refresh: {r.headers.get('x-agentcc-cache')}")

You should see miss on the forced call.

For a global wipe after a prompt-template update, route your traffic to a fresh namespace by setting x-agentcc-cache-namespace: support-v2 instead of support. The old cache stays available to anything still pointed at support.