TrueFoundry AI Gateway client tracing

Trace Python OpenAI-client calls through TrueFoundry AI Gateway in Future AGI, including streaming, token usage and controls for sensitive content.

Trace your application’s OpenAI-client requests through TrueFoundry AI Gateway in Future AGI.

This guide covers Python client tracing only. For gateway-wide telemetry, see TrueFoundry’s native OpenTelemetry export configuration. Client instrumentation does not capture the gateway’s internal routing, guardrails or caller identity, and this recipe does not join client and gateway traces.

Set up client tracing

Install

Use Python 3.10 to 3.13 and install these published versions:

pip install "traceai-openai==0.1.11" "openai==3.42.0" "fi-instrumentation-otel==1.2.2"

Configure credentials and the model

Set FI_API_KEY and FI_SECRET_KEY from your Future AGI account. These authenticate trace export, not gateway requests. For self-hosted Future AGI, also set FI_BASE_URL; the SDK sends traces to /tracer/v1/traces on that host.

In the TrueFoundry Playground, select a model and click Code to find your gateway settings:

Environment variable	Value
`TFY_API_KEY`	A TrueFoundry Personal Access Token for development or Virtual Account Token for applications.
`TFY_MODEL`	An authorized `provider_account/model_name`, such as `openai-main/gpt-4o-mini`. Use your own provider account name.
`TFY_BASE_URL`	Optional for SaaS, where it defaults to `https://gateway.truefoundry.ai`. For a self-hosted gateway, copy the base URL from the Playground.

The SaaS base URL has no /v1 suffix: the OpenAI client appends /chat/completions.

Warning

Prompts, responses, tool arguments and error details are captured by default. Set the privacy controls before running with sensitive data.

Trace a request

import os

from fi_instrumentation import register
from fi_instrumentation.fi_types import ProjectType
from openai import OpenAI
from traceai_openai import OpenAIInstrumentor

trace_provider = register(
    project_type=ProjectType.OBSERVE,
    project_name="my-chatbot",
    verbose=False,
)
OpenAIInstrumentor().instrument(tracer_provider=trace_provider)

try:
    with OpenAI(
        base_url=os.environ.get("TFY_BASE_URL", "https://gateway.truefoundry.ai"),
        api_key=os.environ["TFY_API_KEY"],
    ) as client:
        response = client.chat.completions.create(
            model=os.environ["TFY_MODEL"],
            messages=[{"role": "user", "content": "What is the capital of France?"}],
        )
        print(response.choices[0].message.content)
finally:
    trace_provider.force_flush()
    trace_provider.shutdown()

Open Observe, select my-chatbot, and open the trace. For an agent with a tool call or a streamed response, use client_example.py.

What the span shows

Field	Value
Type and name	`LLM`, `ChatCompletion`.
Model	A non-streamed response’s model when supplied; otherwise the requested gateway model id. Streamed and failed calls retain the requested model id. Request parameters also retain that id unless hidden.
Tokens	Input, output and total counts when the response supplies `usage`. For streaming, request `stream_options={"include_usage": True}` and consume the stream to completion.
Provider	`openai`, the instrumentor’s fallback for this OpenAI-compatible host. This is not a TrueFoundry label or evidence of which upstream provider handled the request.
Prompt, response, tool calls	Captured by default; controlled by the settings below.
Error	Status `ERROR`. Error details are captured unless hidden; the error type remains available when details are hidden.

A response model can differ from the gateway model id you requested. Gateway-side tfy.* fields, including routing and cost, are not added to these client spans. Instrument the client only once; adding another OpenAI instrumentor can create duplicate spans.

Privacy

Set these variables before initializing instrumentation to hide prompts, responses, invocation parameters and error details from traceAI client spans:

export FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true
export FI_HIDE_INPUT_MESSAGES=true FI_HIDE_OUTPUT_MESSAGES=true
export FI_HIDE_LLM_INVOCATION_PARAMETERS=true
export FI_HIDE_ERROR_DETAILS=true

With the pinned core version, FI_HIDE_ERROR_DETAILS=true removes error status descriptions, exception messages and exception stack traces while retaining ERROR status and exception type. Content-hiding flags also enable error-detail hiding by default unless explicitly overridden. See TraceConfig for code-based controls.

Model ids, token counts and tool names remain visible. These settings do not scrub arbitrary custom span attributes, application logs or TrueFoundry’s own logs and traces. Do not put credentials or personal data in names or custom attributes.

Troubleshooting

What you see	What to check
No client spans	Initialize `register()` and instrumentation before the call, set `FI_API_KEY` and `FI_SECRET_KEY`, and flush before process exit. For self-hosted Future AGI, set `FI_BASE_URL`.
Gateway rejects the model	Copy an authorized provider-account/model id from the Playground into `TFY_MODEL`. A token alone does not grant access to every model.
Missing stream tokens	Request usage and consume the stream to completion. The gateway and upstream model must return usage for counts to be available.
Sensitive error text in traces	Install both pinned tracing packages and set `FI_HIDE_ERROR_DETAILS=true` before instrumentation. Check that the text is not being added separately in your own spans or logs.

Next steps

Was this page helpful?

Questions & Discussion