CloudWeGo Eino tracing with Future AGI

About

traceaieino (module github.com/future-agi/traceAI/go/eino) traces Go services built with CloudWeGo Eino. It is an Eino callbacks.Handler, the framework’s own extension point: you register it once at start-up, and every component run Eino reports becomes one span. That covers graphs, chains and workflows and their nodes, chat models (model name, token usage, messages, tool calls), tools, retrievers, embedders, prompts, lambdas and ADK agents. Runs normally nest under the enclosing run or your own span; see callback parenting when multiple tracing handlers share a context.

A run of the example on this page is one trace:

trip_request                     your request span (CHAIN)
└── trip_assistant               the ADK agent (AGENT)
    └── trip_assistant           the agent's graph, as Eino reports it (CHAIN)
        └── ReAct                Eino's ReAct loop (CHAIN), with Init, CancelCheck, ... nodes
            ├── ChatModel        first model call: asks for the tool (LLM, model + tokens)
            ├── ToolNode         (CHAIN)
            │   └── get_weather  the tool call (TOOL)
            └── ChatModel        second model call: the answer (LLM, model + tokens)

This is not the same as:

• The OpenAI, Anthropic and Gemini Go modules (traceAI go/openai, go/anthropic, go/genai). Those wrap one provider SDK’s HTTP calls and see nothing of Eino. This module reads Eino’s callbacks, so it traces whatever model provider an Eino component uses, plus the graph, tool and agent structure around it.
• Eino’s own callback handlers for other vendors (eino-ext/callbacks/langfuse, APMPlus, CozeLoop). They export another product’s attributes and headers; pointing one at Future AGI does not work. This module does not use or replace them. They can stay registered next to it, but callback ordering can affect span parents.
• The Python framework integrations (LangGraph, Google ADK, …). Eino is a Go framework; this module writes the same traceAI attribute names, so its spans read the same in Future AGI.

Example versions	traceAI go/eino v0.1.0, Eino v0.9.21, eino-ext OpenAI chat model v0.1.13
Go	1.26 or later
Source	traceAI go/eino

1. Install

For a new example application, create a module and download the complete program from the traceAI release:

mkdir eino-go-example
cd eino-go-example
go mod init example.com/eino-go-example
curl --fail --location --output main.go \
  https://raw.githubusercontent.com/future-agi/traceAI/go/eino/v0.1.0/go/eino/example/main.go
go get github.com/future-agi/traceAI/go/eino@v0.1.0 github.com/cloudwego/eino@v0.9.21
go get github.com/cloudwego/eino-ext/components/model/openai@v0.1.13
go get go.opentelemetry.io/otel/sdk@v1.47.0 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@v1.47.0
go mod tidy

Stay in eino-go-example for the remaining commands. Steps 3–6 explain the tracing already included in main.go; you do not need to paste those snippets into it. To instrument an existing application instead, run the go get commands in your application’s module and add the tracing setup below.

2. Set environment variables

export FI_API_KEY="your-futureagi-api-key"
export FI_SECRET_KEY="your-futureagi-secret-key"
export FI_PROJECT_NAME="my-service"

FI_BASE_URL is optional (default https://api.futureagi.com; set it to your own collector when you self-host). These keys are for the exporter only: they never go to the model provider.

3. Create the tracer provider

The module does not build an exporter. Use the stock OpenTelemetry OTLP/HTTP exporter with the three things Future AGI needs: both auth headers, the project_name resource attribute, and an uncompressed body (the collector rejects gzip).

import (
	"context"
	"errors"
	"fmt"
	"os"
	"strings"
	"time"

	"github.com/cloudwego/eino-ext/components/model/openai"
	"github.com/cloudwego/eino/adk"
	"github.com/cloudwego/eino/callbacks"
	"github.com/cloudwego/eino/components/tool"
	"github.com/cloudwego/eino/components/tool/utils"
	"github.com/cloudwego/eino/compose"
	traceaieino "github.com/future-agi/traceAI/go/eino"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

func newTracerProvider(ctx context.Context) (*sdktrace.TracerProvider, error) {
	apiKey, secretKey := os.Getenv("FI_API_KEY"), os.Getenv("FI_SECRET_KEY")
	project := os.Getenv("FI_PROJECT_NAME")
	if apiKey == "" || secretKey == "" || project == "" {
		return nil, errors.New("set FI_API_KEY, FI_SECRET_KEY and FI_PROJECT_NAME")
	}
	base := strings.TrimRight(envOr("FI_BASE_URL", "https://api.futureagi.com"), "/")

	exp, err := otlptracehttp.New(ctx,
		otlptracehttp.WithEndpointURL(base+"/tracer/v1/traces"),
		otlptracehttp.WithHeaders(map[string]string{
			"X-Api-Key":    apiKey,
			"X-Secret-Key": secretKey,
		}),
		otlptracehttp.WithCompression(otlptracehttp.NoCompression),
	)
	if err != nil {
		return nil, err
	}
	res, err := resource.Merge(resource.Default(), resource.NewSchemaless(
		attribute.String("project_name", project),
		attribute.String("project_type", "observe"),
		attribute.String("service.name", project),
	))
	if err != nil {
		return nil, err
	}
	return sdktrace.NewTracerProvider(
		sdktrace.WithResource(res),
		sdktrace.WithBatcher(exp),
	), nil
}

resource.Default() adds SDK metadata and reads OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES. This setup overrides service.name with your project name and enables no host, process or OS detectors. However, attributes supplied through OTEL_RESOURCE_ATTRIBUTES are still exported, including any host names, user names or local paths you put there. Review those values separately: the handler’s privacy switches do not filter resource attributes.

4. Register the handler and flush on exit

	tp, err := newTracerProvider(ctx)
	if err != nil {
		return err
	}
	h := traceaieino.NewHandler(traceaieino.WithTracerProvider(tp))
	callbacks.AppendGlobalHandlers(h) // once, before any graph or agent runs
	defer shutdown(h, tp)             // flush before exit

func shutdown(h *traceaieino.Handler, tp *sdktrace.TracerProvider) {
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()
	if err := h.Shutdown(ctx); err != nil {
		fmt.Fprintln(os.Stderr, "trace handler:", err)
	}
	if err := tp.Shutdown(ctx); err != nil {
		fmt.Fprintln(os.Stderr, "trace export:", err)
	}
}

• Register once. Eino’s AppendGlobalHandlers is not thread-safe: call it at start-up, before any graph or agent runs. Registering the handler twice gives two spans for every component.
• To trace one run instead of all of them, pass the handler to that run: compose.WithCallbacks(h) for a graph or chain, adk.WithCallbacks(h) for an agent.
• h.Shutdown stops the handler (later callbacks start no span and never panic) and waits for spans that are still being finished in the background, such as a streamed reply the handler is still reading. tp.Shutdown then exports the last batch. Spans still queued when the process exits are lost, so a long-running service must call both on SIGTERM too; nothing is flushed on SIGKILL.
• Without WithTracerProvider the handler uses the OpenTelemetry global provider (otel.SetTracerProvider).

Build your graphs and agents as usual; nothing else changes. The example’s agent:

	chat, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{
		APIKey:  os.Getenv("OPENAI_API_KEY"),
		BaseURL: os.Getenv("OPENAI_BASE_URL"), // empty: api.openai.com
		Model:   modelName,
		Timeout: 2 * time.Minute,
	})
	if err != nil {
		return nil, err
	}
	weather, err := utils.InferTool("get_weather", "Get tomorrow's weather forecast for a city.", getWeather)
	if err != nil {
		return nil, err
	}
	return adk.NewChatModelAgent(ctx, &adk.ChatModelAgentConfig{
		Name:        "trip_assistant",
		Description: "Plans what to pack for a trip.",
		Instruction: "Call get_weather for the city first, then answer in two short sentences.",
		Model:       chat,
		ToolsConfig: adk.ToolsConfig{ToolsNodeConfig: compose.ToolsNodeConfig{
			Tools: []tool.BaseTool{weather},
		}},
	})

5. Sessions and your own spans

Put a session id on the context you pass to the run; every span the handler starts from it, for the whole graph or agent run, carries session.id, and Future AGI groups those traces into one session. ContextWithUserID does the same for user.id (use an opaque id, not an email address).

	ctx = traceaieino.ContextWithSessionID(ctx, sessionID)

For a top-level Eino run, the current OpenTelemetry span becomes its parent unless another callback replaces it first. The example uses one tracing handler and wraps each question in a request span, the way an HTTP server span would:

	ctx, span := tp.Tracer("eino-example").Start(ctx, "trip_request",
		trace.WithAttributes(append(traceaieino.ContextAttributes(ctx),
			attribute.String(traceaieino.GenAISpanKind, traceaieino.SpanKindChain))...))
	defer span.End()
	fmt.Println("trace_id=" + span.SpanContext().TraceID().String())
	tc := traceaieino.TraceConfigFromEnv()
	tc.SetAttributes(span, attribute.String(traceaieino.InputValue, q))

TraceConfig.SetAttributes applies the same key-based privacy rules to attributes you add yourself: the question in input.value and the answer in output.value become __REDACTED__ when their respective content switches are enabled. It does not scrub arbitrary custom attributes. The handler puts its own span on the context it returns; with no other handler replacing that span, a span you start inside a tool or a lambda nests under that node.

Callback parenting

A top-level run with no enclosing traceAI run inherits the current OpenTelemetry span, including one placed on the context by another callback handler. Eino v0.9.21 calls OnStart handlers in reverse registration order: with compose.WithCallbacks(h, other), other runs first and can make its span the parent of the traceAI top-level span.

For a nested run with an enclosing traceAI span, the handler keeps the current span only if it is valid and uses the same tracer provider as the enclosing span; otherwise it uses the enclosing traceAI span. This protects nested runs from cross-provider reparenting, not top-level runs or other handlers using the same provider. Use a single tracing handler per run if you do not want callback ordering to affect the parent.

6. Streams and agents

	runner := adk.NewRunner(ctx, adk.RunnerConfig{Agent: agent, EnableStreaming: *stream})
	events := runner.Query(ctx, q)

• A streamed run (Stream, Transform) is one span. Eino hands the handler its own copy of the stream; the span ends when the handler has read that copy to the end. Chunks are not spans. The reply text is joined, and token usage comes from the chunk that reports it (the eino-ext OpenAI model asks for usage in the last chunk). Chunks are never summed.
• An ADK agent’s span ends when the agent’s event stream ends, so it covers the whole run, and is ERROR when an event carries an error.
• An ADK ChatModelAgent also shows Eino’s internal ReAct graph (ReAct, Init, ChatModel, ToolNode, CancelCheck, AfterToolCalls, …) as CHAIN spans, because Eino reports each of them as a component run.

7. Run the example

Step 1 downloaded the complete program into eino-go-example/main.go. It is also available in examples/traceai/eino-go/ in this docs repo. With the Future AGI variables from step 2 set, run these commands from the same eino-go-example directory:

export OPENAI_API_KEY="your-openai-api-key"
export MODEL="gpt-4.1-mini"  # OPENAI_BASE_URL for an OpenAI-compatible endpoint
go run . -stream -fail

It prints a trace id per question. -fail asks a second agent whose model does not exist. Open Observe → your project → Tracing: the first trace shows the agent, both model calls with model name and token counts, and the tool call with its arguments and result; the second shows the model call and everything above it as ERROR with error.type *openai.APIError.

What is captured

Eino component (RunInfo.Component)	gen_ai.span.kind
ChatModel, AgenticModel	LLM (gen_ai.operation.name=chat)
Tool	TOOL (execute_tool)
Agent, AgenticAgent (ADK)	AGENT (invoke_agent)
Retriever	RETRIEVER
Embedding	EMBEDDING (embeddings)
Graph, Chain, Workflow, Lambda, Passthrough, ToolsNode, ChatTemplate, AgenticChatTemplate, Indexer, Loader, DocumentTransformer	CHAIN
any other (custom) component	UNKNOWN, kept with its name

The span name is the node, tool or agent name (compose.WithNodeName, the tool name, the agent name), otherwise Eino’s display name, for example OpenAIChatModel.

Attribute	When
gen_ai.span.kind	always
gen_ai.operation.name	only on LLM (chat), tool (execute_tool), embedding (embeddings) and agent (invoke_agent) spans
eino.component, eino.type, eino.name	only when RunInfo supplies the corresponding non-empty field
gen_ai.request.model, gen_ai.request.max_tokens, gen_ai.request.temperature, gen_ai.request.top_p, gen_ai.request.stop_sequences, gen_ai.request.parameters	when supplied by the chat model callback; numeric zero values are omitted. The parameters JSON can also contain tool_choice
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.total_tokens	chat model spans: all three are emitted when usage has any non-zero input, output or total count; individual input/output counts may be zero. A zero total is derived from input + output; never a sum of chunks
gen_ai.usage.cache_read_tokens, gen_ai.usage.output_tokens.reasoning	chat model spans: only positive counts, and only when ordinary token usage is present
gen_ai.response.finish_reasons	when the chat model response supplies a non-empty finish reason
gen_ai.tool.definitions.*	when the chat model input supplies encodable named tool definitions
gen_ai.input.messages.*, gen_ai.output.messages.* (role, text, image URLs, tool calls)	chat model spans when the corresponding message fields are supplied (content)
gen_ai.tool.name, gen_ai.tool.call.id; gen_ai.tool.call.arguments, gen_ai.tool.call.result (content)	tool spans: name from non-empty RunInfo.Name, call ID from context, arguments/result from the tool callback
gen_ai.agent.name	agent spans with a non-empty RunInfo.Name
gen_ai.retrieval.query (content), gen_ai.retrieval.top_k, retrieval.documents.{i}.document.id / .score / .content	retriever callbacks: query and document content are content attributes; top-k is positive-only, IDs must be non-empty, scores require _score metadata
gen_ai.embeddings.dimension.count	embedding callbacks with a non-empty first vector; this mapping omits the vectors themselves
input.value, output.value (content)	when the component mapping emits a value; retrieval, embedding and indexer outputs use dedicated attributes instead
session.id, user.id	from ContextWithSessionID / ContextWithUserID
error.type, error.message	on errors, see below

These attributes depend on the callback payload and privacy settings. Embedding callbacks emit gen_ai.usage.input_tokens and gen_ai.usage.total_tokens when usage has a non-zero count; either exported count may be zero, and this mapping does not derive a missing total. They can also supply gen_ai.request.model.

gen_ai.provider.name is not written: Eino reports the component (OpenAI), not the endpoint behind it. Cost is not computed by the module; Future AGI derives it from the model name and token counts when it knows the model.

Privacy

Content is captured by default, as in every traceAI instrumentor. The traceAI switches, with the same names and defaults as the Python package, turn it off. Set them in the environment before NewHandler runs (they are read once), or pass traceaieino.WithTraceConfig(traceaieino.TraceConfig{...}).

Variable	Effect
FI_HIDE_INPUTS=true	input.value becomes __REDACTED__; input messages, tool-call arguments, the retriever query and error.message are dropped
FI_HIDE_OUTPUTS=true	output.value becomes __REDACTED__; output messages, tool-call results, retrieved text and error.message are dropped
FI_HIDE_INPUT_MESSAGES, FI_HIDE_OUTPUT_MESSAGES	drop gen_ai.input.messages.* / gen_ai.output.messages.* only
FI_HIDE_INPUT_TEXT, FI_HIDE_OUTPUT_TEXT	redact text only in the structured gen_ai.input.messages.* / gen_ai.output.messages.* attributes; keep roles and tool calls. They do not redact input.value / output.value
FI_HIDE_INPUT_IMAGES, FI_BASE64_IMAGE_MAX_LENGTH	apply only to structured input-message image attributes: drop image data, or redact base64 data:image/ URLs longer than the limit (default 32 000 bytes). Output images are unaffected
FI_HIDE_LLM_INVOCATION_PARAMETERS	drop gen_ai.request.parameters

Boolean switches take effect only when set to true (any case; surrounding whitespace is ignored). FI_BASE64_IMAGE_MAX_LENGTH takes a positive integer byte limit. With FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true, the handler hides the mapped conversation, tool arguments and results; the span tree, model names, token counts, tool and agent names and error types can still be exported.

These switches are not general-purpose metadata sanitizers. Session/user IDs, span and component names, document IDs, tool definitions and stop sequences can still be exported: keep sensitive data out of those fields. FI_HIDE_LLM_INVOCATION_PARAMETERS drops only the gen_ai.request.parameters JSON attribute, not individual request attributes such as gen_ai.request.stop_sequences.

Message-specific switches do not redact input.value / output.value, which also contain conversation text for ordinary schema.Message inputs and ChatModelAgent runs, not just Agentic or generic JSON payloads. Set FI_HIDE_INPUTS=true and FI_HIDE_OUTPUTS=true to keep conversation text out of these values and the structured message attributes.

Structured schema.Message capture excludes audio, video and file data from the dedicated media fields. This exclusion does not inspect data embedded in message text or tool arguments.

AgenticModel, AgenticAgent and other generic JSON payloads can include inline media data (including base64 audio, video and files) and extra fields in input.value / output.value. Message-specific switches (FI_HIDE_INPUT_MESSAGES, FI_HIDE_OUTPUT_MESSAGES, FI_HIDE_INPUT_TEXT, FI_HIDE_OUTPUT_TEXT, FI_HIDE_INPUT_IMAGES and FI_BASE64_IMAGE_MAX_LENGTH) do not sanitize generic JSON. Set FI_HIDE_INPUTS=true and FI_HIDE_OUTPUTS=true when those payloads must not leave the process. The 32 000-byte limit on JSON values truncates content; it is not redaction.

The embedding callback mapping omits embedding vectors. The module adds no resource attributes.

Errors

• A failed run ends its span with status ERROR and error.type: canceled or timeout for the context, otherwise the Go type of the innermost error (for example *openai.APIError), or error for a plain Go error. The enclosing graph and agent spans are ERROR too, because Eino reports the error at each level.
• error.message uses the first line of the error with best-effort scrubbing of matching URLs, email addresses, file paths, IPv4 addresses, host:port pairs and known local host/user/home values. IPv6 endpoints and other unmatched details can remain. Set FI_HIDE_INPUTS=true or FI_HIDE_OUTPUTS=true to suppress error.message when network details must not be exported. The handler adds no stack trace or exception event.
• When the handler receives an error from a component’s output stream, it ends that component’s span ERROR. A later OnEnd for a run that already failed changes nothing.
• The handler never changes an input, an output or an error, never returns an error to Eino, and recovers from its own panics.

Scope limits

• The handler creates spans from Eino callbacks. A component that fires no callbacks and runs outside a graph produces no span.
• This guide covers ChatModelAgent runs through adk.Runner. It does not cover ADK interrupt/resume, checkpoints, DeepAgent, multi-agent transfers or workflow agents.
• AgenticModel and AgenticAgent content is recorded as JSON in input.value / output.value, not as structured message attributes. See Privacy to keep media and extra fields out of exported JSON.
• gen_ai.usage.cache_write_tokens is not recorded.
• The Eino DevOps UI, evaluators and prompt tools are separate projects and are not traced.

Troubleshooting

• No traces. Check that the process calls h.Shutdown and tp.Shutdown before it exits, that the handler was registered before the first run, and that FI_API_KEY, FI_SECRET_KEY and FI_PROJECT_NAME are set. The exporter URL must be the full traces URL, {FI_BASE_URL}/tracer/v1/traces.
• Export errors about the body. Keep otlptracehttp.WithCompression(otlptracehttp.NoCompression); the collector rejects gzip.
• Spans in the wrong project or none at all. project_name must be a resource attribute on the tracer provider, not a span attribute.
• Two spans for every node. The handler is registered twice: AppendGlobalHandlers ran twice, or the handler is global and also passed with compose.WithCallbacks / adk.WithCallbacks.
• No model name or token counts on a model span. The chat model fires no callbacks itself, so the graph reports only its messages. The eino-ext models report both; for your own model, fire callbacks.OnStart / OnEnd with *model.CallbackInput / *model.CallbackOutput.
• A streamed model span ends late or never. Read the stream to the end or close it; the span ends when the handler’s copy is read to the end.
• A traceAI root appears under another tracing handler’s span. Top-level runs inherit the current OpenTelemetry span. Another handler can replace it before traceAI runs; review callback ordering and the narrower nested-run protection.