Embabel tracing with Future AGI

Trace an Embabel agent’s runs, actions, model calls and tools in Future AGI using Embabel’s observability starter and Spring Boot’s OTLP exporter.

This guide uses Embabel 1.5.3, Spring Boot 4.1.1, Java 21 or later, the embabel-agent-starter-openai-custom model starter, and a self-hosted Future AGI deployment. Set FI_BASE_URL to your self-hosted Future AGI origin before running the application. Get your Future AGI API and secret keys and choose an Observe project before you begin.

1. Add the dependencies

Embabel 1.5.3 is built on Spring Boot 4.1.1, so use that parent. Add the observability starter and the OTLP exporter next to your Embabel and model starters:

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>4.1.1</version>
    <relativePath/>
</parent>

<properties>
    <java.version>21</java.version>
    <embabel-agent.version>1.5.3</embabel-agent.version>
</properties>

<dependencies>
    <dependency>
        <groupId>com.embabel.agent</groupId>
        <artifactId>embabel-agent-starter</artifactId>
        <version>${embabel-agent.version}</version>
    </dependency>
    <!-- your model starter, for example any OpenAI-compatible endpoint: -->
    <dependency>
        <groupId>com.embabel.agent</groupId>
        <artifactId>embabel-agent-starter-openai-custom</artifactId>
        <version>${embabel-agent.version}</version>
    </dependency>

    <!-- tracing -->
    <dependency>
        <groupId>com.embabel.agent</groupId>
        <artifactId>embabel-agent-starter-observability</artifactId>
        <version>${embabel-agent.version}</version>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-otlp</artifactId>
    </dependency>
</dependencies>

All of these resolve from Maven Central. No snapshot repository is needed, and the Langfuse exporter (opentelemetry-exporter-embabel) is not used.

2. Configure the exporter

Add this to application.yml:

spring:
  application:
    name: my-agent-app            # becomes service.name

embabel:
  agent:
    platform:
      observability:
        enabled: true
        # Content off by default. Set EMBABEL_CAPTURE_CONTENT=true to opt in.
        capture-message-content: ${EMBABEL_CAPTURE_CONTENT:false}
        trace-http-details: false

management:
  tracing:
    sampling:
      probability: 1.0
  opentelemetry:
    resource-attributes:
      project_name: ${FI_PROJECT_NAME}
      project_type: observe
    tracing:
      export:
        otlp:
          endpoint: ${FI_BASE_URL}/tracer/v1/traces
          transport: http
          compression: none
          headers:
            X-Api-Key: ${FI_API_KEY}
            X-Secret-Key: ${FI_SECRET_KEY}

Piece	Value
Endpoint	${FI_BASE_URL}/tracer/v1/traces, where FI_BASE_URL is your self-hosted origin. Spring Boot uses the URL as written, so include the path. Use HTTP, not gRPC; this setup exports traces only.
Headers	Both X-Api-Key and X-Secret-Key. A Bearer token does not authenticate.
Resource	project_name and project_type=observe. Future AGI files spans by the resource attribute project_name, and a batch without it is dropped. A span attribute with that key does not count, and service.name is not read as a project name. On Embabel this property needs the bean in step 3.
Compression	none, which is also Spring Boot’s default and works on every Future AGI release. Future AGI cloud and self-hosted v1.56.0 and later also accept gzip.
Sampling	1.0. Embabel already defaults to 1.0, overriding Spring Boot’s 0.1. Below 1.0, whole runs are dropped at random.
Content	Off. See Privacy. Embabel’s own default is on.

3. Add the configuration class

Two beans are needed on top of application.yml:

• The resource. Embabel’s observability starter registers its own OpenTelemetry Resource bean, which carries only service.name. It replaces Spring Boot’s resource, so management.opentelemetry.resource-attributes and OTEL_RESOURCE_ATTRIBUTES are ignored and project_name never reaches Future AGI. Declaring a Resource bean yourself restores Spring Boot’s behaviour: both Embabel’s and Spring Boot’s beans step aside when one exists.
• The exporter wrapper. capture-message-content: false removes prompts, answers and tool payloads, but Embabel still exports error text: in the span status, in the exception event (message and stack trace), and in embabel.llm.error.message and embabel.tool.error.message. Error text often repeats the input that caused it. The wrapper drops it when content capture is off, and drops stack traces always. SpanExporters is the Spring Boot 4 bean that the batch span processor exports to, so declaring it wraps the OTLP exporter without exporting anything twice.

import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.sdk.resources.ResourceBuilder;
import io.opentelemetry.sdk.trace.export.SpanExporter;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.micrometer.tracing.opentelemetry.autoconfigure.SpanExporters;
import org.springframework.boot.opentelemetry.autoconfigure.OpenTelemetryProperties;
import org.springframework.boot.opentelemetry.autoconfigure.OpenTelemetryResourceAttributes;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.env.Environment;

@Configuration(proxyBeanMethods = false)
public class FutureAgiTracingConfiguration {

    /** Builds the resource the way Spring Boot does, so project_name is sent. */
    @Bean
    Resource futureAgiResource(Environment environment, OpenTelemetryProperties properties) {
        ResourceBuilder builder = Resource.builder();
        new OpenTelemetryResourceAttributes(environment, properties.getResourceAttributes()).applyTo(builder::put);
        return Resource.getDefault().merge(builder.build());
    }

    /** Error text follows the content switch; stack traces are never exported. */
    @Bean
    SpanExporters spanExporters(
            ObjectProvider<SpanExporter> exporters,
            @Value("${embabel.agent.platform.observability.capture-message-content:true}") boolean captureContent) {
        return SpanExporters.of(exporters.orderedStream()
                .map(exporter -> new ErrorTextRedactingSpanExporter(exporter, captureContent))
                .toList());
    }
}

import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.common.AttributesBuilder;
import io.opentelemetry.sdk.common.CompletableResultCode;
import io.opentelemetry.sdk.trace.data.DelegatingSpanData;
import io.opentelemetry.sdk.trace.data.EventData;
import io.opentelemetry.sdk.trace.data.SpanData;
import io.opentelemetry.sdk.trace.data.StatusData;
import io.opentelemetry.sdk.trace.export.SpanExporter;

import java.util.Collection;
import java.util.List;
import java.util.Set;

/** Drops error text (unless keepErrorText) and stack traces; keeps the ERROR status and the error type. */
public final class ErrorTextRedactingSpanExporter implements SpanExporter {

    static final Set<String> ERROR_TEXT_KEYS = Set.of(
            "exception.message", "error.message", "embabel.llm.error.message", "embabel.tool.error.message");
    static final String STACKTRACE_KEY = "exception.stacktrace";

    private final SpanExporter delegate;
    private final boolean keepErrorText;

    public ErrorTextRedactingSpanExporter(SpanExporter delegate, boolean keepErrorText) {
        this.delegate = delegate;
        this.keepErrorText = keepErrorText;
    }

    @Override
    public CompletableResultCode export(Collection<SpanData> spans) {
        return delegate.export(spans.stream().map(this::redact).toList());
    }

    @Override
    public CompletableResultCode flush() {
        return delegate.flush();
    }

    @Override
    public CompletableResultCode shutdown() {
        return delegate.shutdown();
    }

    SpanData redact(SpanData span) {
        StatusData status = span.getStatus();
        boolean statusHasText = !keepErrorText && !status.getDescription().isEmpty();
        boolean attributesHaveText = hasDroppedKey(span.getAttributes());
        boolean eventsHaveText = span.getEvents().stream().anyMatch(e -> hasDroppedKey(e.getAttributes()));
        if (!statusHasText && !attributesHaveText && !eventsHaveText) {
            return span;
        }
        StatusData cleanStatus = statusHasText ? StatusData.create(status.getStatusCode(), "") : status;
        Attributes cleanAttributes = withoutDroppedKeys(span.getAttributes());
        List<EventData> cleanEvents = span.getEvents().stream()
                .map(e -> hasDroppedKey(e.getAttributes())
                        ? EventData.create(e.getEpochNanos(), e.getName(), withoutDroppedKeys(e.getAttributes()))
                        : e)
                .toList();
        return new DelegatingSpanData(span) {
            @Override
            public StatusData getStatus() {
                return cleanStatus;
            }

            @Override
            public Attributes getAttributes() {
                return cleanAttributes;
            }

            @Override
            public int getTotalAttributeCount() {
                return cleanAttributes.size();
            }

            @Override
            public List<EventData> getEvents() {
                return cleanEvents;
            }
        };
    }

    private boolean dropped(String key) {
        return STACKTRACE_KEY.equals(key) || (!keepErrorText && ERROR_TEXT_KEYS.contains(key));
    }

    private boolean hasDroppedKey(Attributes attributes) {
        return attributes.asMap().keySet().stream().anyMatch(k -> dropped(k.getKey()));
    }

    private Attributes withoutDroppedKeys(Attributes attributes) {
        AttributesBuilder builder = Attributes.builder();
        attributes.forEach((key, value) -> {
            if (!dropped(key.getKey())) {
                @SuppressWarnings("unchecked")
                AttributeKey<Object> k = (AttributeKey<Object>) key;
                builder.put(k, value);
            }
        });
        return builder.build();
    }
}

Put both classes in a package your @SpringBootApplication scans. Your agents, actions and tools need no changes.

4. Set environment variables and run

export FI_API_KEY="your-futureagi-api-key"
export FI_SECRET_KEY="your-futureagi-secret-key"
export FI_PROJECT_NAME="my-agent-app"            # the Observe project the traces land in
# Self-hosted Future AGI only. Traces go to $FI_BASE_URL/tracer/v1/traces.
export FI_BASE_URL="http://localhost:4318"       # replace with your self-hosted origin
# export EMBABEL_CAPTURE_CONTENT=true            # opt in to prompts, answers, tool payloads and error text

# the model, here through embabel-agent-starter-openai-custom
export OPENAI_CUSTOM_BASE_URL="https://api.openai.com/v1"
export OPENAI_CUSTOM_API_KEY="your-model-key"
export OPENAI_CUSTOM_MODELS="gpt-4.1-mini"
export EMBABEL_MODELS_DEFAULT_LLM="gpt-4.1-mini"

Set FI_BASE_URL to the origin only. application.yml adds /tracer/v1/traces.

Run your application’s entry point. For a complete runnable agent, use the Embabel example application, which runs one weather question and closes the Spring context when it finishes.

Spans are exported in batches every few seconds. When the Spring context closes (a normal shutdown, or SpringApplication.exit in a command-line app), the remaining spans are flushed. A process that is killed loses the spans of its last few seconds.

5. Look at the trace

Open Observe → Tracing in your project. One agent run is one trace:

agent WeatherAgent                        agent
├── planning ready
├── planning report
├── action report
│   ├── llm <model>
│   │   └── tool-loop
│   │       ├── chat <model>                  llm   (model, input and output tokens)
│   │       ├── llm.invocation <model>              (the same token counts again)
│   │       ├── execute_tool currentWeather   tool
│   │       ├── chat <model>                  llm
│   │       └── llm.invocation <model>
│   └── tool-loop-completed
├── planning ready
├── goal report
└── COMPLETED

This example shows an agent with one action and one tool. The right-hand column is the span type shown in Future AGI. chat spans come from Spring AI’s model call, the others from Embabel. Spans without a type in the column are shown as plain spans in the trace tree.

What is captured

With content off (the default on this page):

Span	What it carries
agent <name>	Agent name, planner type, status, goal, run id (embabel.run.id, also sent as gen_ai.conversation.id), gen_ai.operation.name=agent. Status ERROR when the run fails.
action <name>	Action name, agent name, run id.
llm <model>, tool-loop	Model, action, tool names, iteration limits and counts.
chat <model>	gen_ai.request.model, gen_ai.response.model, gen_ai.provider.name, finish reasons, response id, gen_ai.usage.input_tokens and output_tokens. Future AGI shows the model and token counts on this span.
llm.invocation <model>	Model and token usage again.
execute_tool <tool>	Tool name, tool status, duration, correlation id. Status ERROR when the tool throws.
planning, goal, COMPLETED / lifecycle	Plan goal and action count, goal name, final state.

With content on, the agent, action and tool-loop spans also carry input.value and output.value (shown as input and output in Future AGI), chat spans carry the prompt and answer messages, tool spans carry the arguments and result as attributes, and error spans carry the error text.

Errors

• Tool throws. The execute_tool span is ERROR with the error type (ToolReturnedError). Embabel hands the error to the model, which usually answers anyway, so the run can still end COMPLETED. On Embabel 1.5.2 the same tool failure was recorded with status OK; 1.5.3 marks it ERROR.
• Model call fails. Every span on the failing path is ERROR, plus llm.retry and llm.error spans. Embabel retries the call (10 attempts by default) and then the action (up to 5 attempts).
• With content off, the wrapper clears status messages and removes its listed error-message attributes. With content on, it retains those messages. It removes exception.stacktrace attributes in both modes.
• If Future AGI cannot be reached, the agent run is not affected. Check the project for missing traces.

Privacy

Content capture is off on this page: capture-message-content: false, plus the exporter wrapper for error text. Embabel’s own default is on, so set it explicitly as shown.

For Embabel’s built-in instrumentation, the content switch suppresses prompts, answers, agent and action inputs and outputs, tool payloads and planning world state. The wrapper additionally clears status descriptions and removes the error-message keys listed in ERROR_TEXT_KEYS and exception.stacktrace attributes. It is not a general-purpose scrubber for custom spans, event names or arbitrary attributes: do not put sensitive data there.

Span names, agent, action, goal and tool names, model names, token counts, durations, statuses, error types and generated ids remain. Class and method names can appear in goal names, for example com.example.WeatherAgent.report.

To opt in, set EMBABEL_CAPTURE_CONTENT=true (or capture-message-content: true). Embabel then records prompts, answers, inputs and outputs, tool payloads and error text, subject to its max-attribute-length setting (4000 characters by default). The wrapper still removes exception.stacktrace attributes.

Keep trace-http-details off: it records HTTP bodies and headers.

How this differs from related pages

• Spring Boot and Spring AI uses the traceAI starter and wraps a Spring AI ChatModel you choose, so it traces model calls only. This page uses Embabel’s own instrumentation and traces the whole agent run: planning, actions, tool loops, tools and model calls. Use one or the other: with both on the same model, each call would be recorded twice.
• Embabel’s README also describes a Langfuse and LangSmith exporter (opentelemetry-exporter-embabel). It is not used here: Future AGI takes the standard OTLP exporter.
• There is no traceAI Java package for Embabel. The starter already emits the spans; this page only routes them.

Known gaps

With Embabel 1.5.3:

• Only chat (llm), execute_tool (tool) and agent spans get a span type. Action, LLM wrapper, tool-loop, planning, goal and lifecycle spans show as plain spans.
• Token counts appear twice per model call, on chat and on llm.invocation. A sum over every span in a trace counts them twice; count llm spans only.
• Token counts are sent as strings (Micrometer tags are strings). Future AGI still reads them.
• Embabel sends a session id as gen_ai.conversation.id (the run id when there is no conversation), but Future AGI does not read it as the session, so runs are not grouped into sessions.
• With content on, tool arguments and results are span attributes, not the tool span’s input and output.
• On a failed run, the agent span’s embabel.agent.status attribute can still read RUNNING while its span status is ERROR.

Troubleshooting

• Nothing arrives, and the log says No SpanExporter beans found. OpenTelemetry tracing will be disabled. The endpoint is set with the Spring Boot 3 name management.otlp.tracing.endpoint, or opentelemetry-exporter-otlp is missing. Use management.opentelemetry.tracing.export.otlp.endpoint.
• The project never appears. The batch had no project_name resource attribute. Check that FutureAgiTracingConfiguration is scanned: without its Resource bean, Embabel’s resource is used and management.opentelemetry.resource-attributes is ignored.
• Only one key header. Both X-Api-Key and X-Secret-Key are required.
• gzip. Releases before v1.56.0 answered HTTP 400; current releases accept gzip. compression: none works on every release.
• The last spans of a short-lived app are missing. Close the Spring context before the JVM exits (SpringApplication.exit); a killed process does not flush.
• Every model call fails with 404 on embabel-agent-starter-openai-custom. Include the /v1 suffix in the base URL (https://host/v1).

Example application

See the Embabel example application for a complete agent and configuration.

Next Steps

• Spring Boot and Spring AI to trace Spring AI model calls with the traceAI starter.
• Java integrations for other Java frameworks and providers.