Observing a LangGraph agent and obtaining insights

Prove your keys and ingestion work before writing any agent code. Register against an Observe project, emit one span, and flush.

from fi_instrumentation import register, FITracer
from fi_instrumentation.fi_types import ProjectType

# Reuse this trace_provider and tracer for the rest of the cookbook.
trace_provider = register(
    project_type=ProjectType.OBSERVE,
    project_name="support-agent",
    set_global_tracer_provider=True,
)
tracer = FITracer(trace_provider.get_tracer(__name__))

with tracer.start_as_current_span("preflight") as span:
    span.set_attribute("raw.input", "ping")
    span.set_attribute("raw.output", "pong")

trace_provider.force_flush()
print("sent preflight span")

You should see a preflight trace in Observe → Traces → support-agent within a few seconds. If it never lands, fix that here first — see Troubleshooting.

The agent answers from a small inline knowledge base, so the recipe runs with no external data. The search_help_center tool stashes what it retrieved so you can score grounding later.

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_core.tools import tool
from langgraph.prebuilt import create_react_agent

POLICIES = [
    "Refunds: Orders can be refunded within 30 days of delivery. Items marked 'final sale' "
    "are not refundable. Refunds return to the original payment method in 5-7 business days.",
    "Shipping: Standard shipping takes 3-5 business days. Express takes 1-2. We ship to the US and Canada only.",
    "Account: Customers can reset a password from the login page. Support cannot see or change a password.",
    "Returns: To return an item, start a return from the Orders page to get a prepaid label. "
    "Returns must be shipped within 14 days of approval.",
]

store = InMemoryVectorStore.from_texts(POLICIES, OpenAIEmbeddings())
retriever = store.as_retriever(search_kwargs={"k": 2})

last_context = {"text": ""}  # holds the policy retrieved this turn, for scoring

@tool
def search_help_center(query: str) -> str:
    """Search the help center for relevant policy text."""
    docs = retriever.invoke(query)
    last_context["text"] = "\n\n".join(d.page_content for d in docs)
    return last_context["text"]

@tool
def lookup_order(order_id: str) -> str:
    """Look up the status of an order by its ID."""
    return f"Order {order_id}: delivered on 2026-06-20, standard shipping."

SYSTEM_PROMPT = "You are a helpful customer-support agent. Use the tools to answer."

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
agent = create_react_agent(llm, tools=[search_help_center, lookup_order], prompt=SYSTEM_PROMPT)

Turn on auto-instrumentation, then run the conversations. The same LangChainInstrumentor captures LangGraph. Wrap each turn in using_session (the conversation) and using_user (the customer), and record the three fields an eval needs: question, answer, retrieved policy.

from fi_instrumentation import using_session, using_user
from traceai_langchain import LangChainInstrumentor

# trace_provider and tracer come from step 1 — do not register again.
LangChainInstrumentor().instrument(tracer_provider=trace_provider)

def support_turn(session_id: str, user_id: str, question: str) -> str:
    with using_session(session_id), using_user(user_id):
        with tracer.start_as_current_span("support_turn") as span:
            result = agent.invoke({"messages": [{"role": "user", "content": question}]})
            answer = result["messages"][-1].content
            span.set_attribute("raw.input", question)
            span.set_attribute("raw.output", answer)
            span.set_attribute("raw.context", last_context["text"])  # retrieved policy
            return answer

# two customers, five turns between them
CONVERSATIONS = [
    ("chat_1001", "cust_42", [
        "How long do refunds take?",
        "My order was a final sale, can I still get a refund?",
        "Can you look up order A-3391?",
    ]),
    ("chat_1002", "cust_77", [
        "What shipping options do you have?",
        "Can you change my password for me?",
    ]),
]

for sid, uid, questions in CONVERSATIONS:
    for q in questions:
        print(support_turn(sid, uid, q))

trace_provider.force_flush()  # flush before the script exits

You should see two sessions in Observe → Traces → support-agent: chat_1001 with three traces for cust_42, and chat_1002 with two for cust_77.

Before any eval runs, the traces already answer questions you’d otherwise guess at. Open the support-agent project: every row is one turn, with its status, model, latency, tokens, and cost. Click the final-sale turn to see the ReAct loop as a span tree — the LLM call that decides to search, the search_help_center call with the exact policy it returned, and the LLM call that writes the answer.

Three things are readable before any score exists:

• Cost — two LLM calls per answered question, and the token split between them
• Latency — the retriever and tool spans are milliseconds; the LLM spans are the wait
• Grounding — the tool span carries the policy text, so you can eyeball whether the answer used it

On the final-sale turn, the retrieved policy says final-sale items are not refundable, and the answer hedges toward a refund anyway. An eval can now confirm at scale what this one trace suggests.

Configure evals in the platform as an Eval Task, not from the SDK, so they run on your traces and the whole team sees the same scores. The turn span already carries raw.input, raw.output, and raw.context to map to.

Create an Eval Task on the support-agent project and pick three evals, each chosen so a low score points at one component:

• Context Relevance — did retrieval fetch the right policy? · input → raw.input, context → raw.context
• Context Adherence — did the answer stay grounded in it? · output → raw.output, context → raw.context
• Completeness — did the answer fully address the question? · input → raw.input, output → raw.output

Run it as Historical over the five turns.

You should see three scores per trace (illustrative shape — your numbers will differ):

Filter the trace explorer to the low-scoring slice:

scores.context_adherence < 0.8 AND fi.span.kind = LLM

The final-sale turn surfaces, and its two scores split the layers: retrieval is fine (Context Relevance 0.87), grounding is not (Context Adherence 0.41). To replay the whole conversation around it, scope with user.id = 'cust_42' and open the session.

Each eval’s failure indicts one layer:

What you now know, from one run:

• The prompt is the broken layer, not retrieval — the right policy was retrieved and ignored
• It fails on policy exceptions, not routine turns — the plain refund question scored fine; the final-sale one didn’t
• It’s one turn, not an outage — every other turn for cust_42 is healthy

That’s a ticket your team can act on, not “the bot sometimes gives wrong answers”.

Scores drift as prompts, models, and traffic change. Turn this check into standing monitors so the next regression pages you, not a customer:

• Alert — an Evaluation-metric alert on context_adherence, operator Less than, static 0.8
• Saved view — save the scores.context_adherence < 0.8 filter as an “Ungrounded answers” view, so tomorrow’s check is one click
• Regression set — the low-scoring turns are your best future test cases; harvest them so any fix has to clear them