TypeSafe (Jev) decision tracing with Future AGI

Trace every TypeSafe System One decision your Python code makes, with each typed answer recorded on one EVALUATOR span in Future AGI.

TypeSafe serves Jev through the System One decision API: you send some content (state) and named questions, and get back a typed answer per question. It is not a chat API and returns no text. traceAI-typesafe instruments the official Python client, typesafe-sdk: every TypeSafeClient.system_one(...) and AsyncTypeSafeClient.system_one(...) call becomes one span, however many questions it asks.

A run of the example on this page is one trace:

triage_ticket              your own span (CHAIN)
├── typesafe.system_one    billing (noul), tone (choice), urgency (score)  (EVALUATOR)
└── typesafe.system_one    escalate (noul), from the async client           (EVALUATOR)

This package only traces calls your own code makes to TypeSafe. It does not add an evaluator model to Future AGI evals, does not call the Future AGI gateway, and never turns answers into Pass/Fail. System One is not OpenAI-compatible, so traceAI-openai cannot trace it.

Use Python 3.10 to 3.13, as required by traceAI-typesafe 0.1.0, with typesafe-sdk 0.7.0 or later, below 1.0. The instrumentor source is in traceAI python/frameworks/typesafe.

1. Install

pip install traceAI-typesafe typesafe-sdk

2. Set environment variables

export FI_API_KEY="your-futureagi-api-key"
export FI_SECRET_KEY="your-futureagi-secret-key"
export TYPESAFE_API_KEY="your-typesafe-api-key"

The TypeSafe key goes to the SDK only and never onto a span. FI_BASE_URL is optional (set it to your own collector when you self-host). The SDK also reads TYPESAFE_BASE_URL and TYPESAFE_DEFAULT_MODEL itself; this package reads neither and never sets a host or a model.

3. Register the tracer and instrument the client

import os

from fi_instrumentation import register
from fi_instrumentation.fi_types import ProjectType
from traceai_typesafe import TypeSafeInstrumentor
from typesafe_sdk import AsyncTypeSafeClient, Choice, Noul, Score, TypeSafeClient

trace_provider = register(
    project_type=ProjectType.OBSERVE,
    project_name=os.environ.get("FI_PROJECT_NAME", "support-triage"),
)
TypeSafeInstrumentor().instrument(tracer_provider=trace_provider)

Call instrument() once, before you make decisions. Clients created before or after are both traced.

4. Make decisions

Nothing changes in how you call TypeSafe. Define the content you want decided and your questions:

TICKET = (
    "Order 1042: I was charged twice for the same order and nobody answers my emails. "
    "Please refund the duplicate today."
)

QUESTIONS = {
    "billing": Noul(instructions="Is this message about billing?"),
    "tone": Choice(
        instructions="What is the tone of this message?",
        criteria={"calm": None, "angry": "Upset or hostile", "excited": None},
    ),
    "urgency": Score(
        instructions="How urgent is this message?",
        criteria=["Can wait", "Needs attention this week", "Needs attention today"],
    ),
}

Then make the call:

with TypeSafeClient() as client:
    result = client.system_one(state=TICKET, questions=QUESTIONS)
print("billing", result.nouls["billing"].noul)
print("tone", result.choices["tone"].choice, result.choices["tone"].confidence)
print("urgency", result.scores["urgency"].score, result.scores["urgency"].confidence)

The async client is traced the same way (inside an async function):

async with AsyncTypeSafeClient() as client:
    result = await client.system_one(
        state={"ticket": TICKET, "customer_tier": "gold"},
        questions={"escalate": Noul(instructions="Should a human agent take this ticket now?")},
    )
print("escalate", result.nouls["escalate"].noul)

In a short-lived script, call trace_provider.force_flush() before the process exits so the last spans are sent.

5. Open the trace

Open the support-triage project (or the name you set in FI_PROJECT_NAME) in Observe. Each decision is a typesafe.system_one span with type Evaluator, the requested model, token counts, and the answers as span attributes.

What a span contains

One span per system_one call, named typesafe.system_one, kind EVALUATOR. Questions are attributes, not child spans: TypeSafe answers them all in one response.

Never on the span: the probability maps (probabilities), score legends, image bytes, the API key, and any explanation (the API returns none, so none is invented).

Privacy

traceAI-typesafe follows the same TraceConfig rules as every traceAI instrumentor: content is captured by default, and the standard flags turn it off.

Set the flags in the environment before the process starts, or pass a TraceConfig:

from fi_instrumentation import TraceConfig

TypeSafeInstrumentor().instrument(
    tracer_provider=trace_provider,
    config=TraceConfig(hide_inputs=True, hide_outputs=True),
)

Recorded in every mode: question names and answer types, numeric answers (noul, score, confidence, box edges), model, token counts, request id and host. Question names are attribute keys, so do not put personal data or secrets in them. Never recorded in any mode: the API key, stack traces, and the text of errors raised inside the SDK before a request is sent (they can name a local file).

Errors and retries

A failed call ends its span with status ERROR and these attributes: error.type (the SDK exception class, for example TypeSafeAuthenticationError), http.response.status_code, typesafe.error_type (TypeSafe’s own code, for example authentication_error), typesafe.request_id, and typesafe.retry_after_ms on a 429. The status message is the class and the HTTP status. While content is captured, TypeSafe’s error message is added to it (an API can echo request values, so it is left out when inputs or outputs are hidden).

The instrumentor adds no retries. The SDK’s own RetryPolicy (by default up to two retries on 408, 429 and 5xx responses) runs inside the one span, so a 429 followed by a success is a single OK span. If you call system_one again yourself, that is a second span.

Errors raised by the SDK before a request is sent (no question, an image file it cannot read) also end the span as ERROR, with only the exception class.

Not supported

• client.models.list() (GET /v1/models) is not a decision and creates no span.
• The TypeSafe JavaScript SDK. Python only.
• Probability maps and legends. Only the chosen value and its confidence are recorded.
• A custom response_model. The span reads the answers field; a model that drops it still gets a span with model and usage, but no per-question attributes.
• system-one.dev. A separate platform with its own keys and credits; this page does not cover it. If you point the client there, you still get a span, but its credit headers are not read.
• Idempotency replay. An Idempotency-Key you send is recorded; nothing is cached or replayed.

Troubleshooting

• No spans. Call TypeSafeInstrumentor().instrument(...) before the first decision, and trace_provider.force_flush() before a short script exits. Check that FI_API_KEY and FI_SECRET_KEY are set in the same process.
• pip cannot find a matching traceAI-typesafe. Use Python 3.10 to 3.13.
• TypeSafeError: No API key was provided. The SDK needs TYPESAFE_API_KEY (or api_key=). This is raised when the client is created, before any span.
• An ERROR span with TypeSafeAuthenticationError (HTTP 401). The key is wrong or revoked.
• TypeSafeUnprocessableEntityError (HTTP 422). TypeSafe rejected the request body; with content captured, the message names the field (for example questions.tone.choice.criteria). A choice question needs a mapping of criteria, not a list.
• The Model column shows jev-latest, not a version. That is the model you asked for. The resolved version is in gen_ai.response.model and typesafe.response.model.
• A noul shows no confidence. Expected: the API returns none for noul.
• The choice label is missing. FI_HIDE_OUTPUTS or FI_HIDE_OUTPUT_TEXT is on.