Evaluate a LangChain RAG Pipeline with Future AGI

Load three Wikipedia pages on transformer architectures, split them into fixed-size chunks, and index them in Chroma.

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import WebBaseLoader
from langchain_community.vectorstores import Chroma

llm = ChatOpenAI(model_name="gpt-4o-mini")
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")

urls = [
    "https://en.wikipedia.org/wiki/Attention_Is_All_You_Need",
    "https://en.wikipedia.org/wiki/BERT_(language_model)",
    "https://en.wikipedia.org/wiki/Generative_pre-trained_transformer",
]

docs = []
for url in urls:
    docs.extend(WebBaseLoader(url).load())

# Fixed-size chunking: the baseline every other strategy is measured against
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
recursive_splits = text_splitter.split_documents(docs)

vectorstore = Chroma.from_documents(
    documents=recursive_splits,
    embedding=embeddings,
    persist_directory="chroma_recursive",
)
recursive_retriever = vectorstore.as_retriever()

print(f"Indexed {len(recursive_splits)} chunks from {len(docs)} pages")

You should see:

Indexed 187 chunks from 3 pages

Register a trace provider with traceai-langchain so every LLM call, retrieval, and chain step is captured, and set up an Evaluator to score answers after the fact.

from getpass import getpass
import os

from fi.evals import Evaluator
from fi_instrumentation import register
from fi_instrumentation.fi_types import (
    ProjectType,
    EvalName,
    EvalSpanKind,
    EvalTag,
    EvalTagType,
    ModelChoices,
)
from traceai_langchain import LangChainInstrumentor

os.environ["FI_API_KEY"] = os.environ.get("FI_API_KEY") or getpass("Enter your FI API key: ")
os.environ["FI_SECRET_KEY"] = os.environ.get("FI_SECRET_KEY") or getpass("Enter your FI API secret: ")

evaluator = Evaluator(fi_base_url="https://api.futureagi.com")

# Tag every LLM span for context adherence and groundedness as it's traced
eval_tags = [
    EvalTag(
        type=EvalTagType.OBSERVATION_SPAN,
        value=EvalSpanKind.LLM,
        eval_name=EvalName.CONTEXT_ADHERENCE,
        model=ModelChoices.TURING_FLASH,
    ),
    EvalTag(
        type=EvalTagType.OBSERVATION_SPAN,
        value=EvalSpanKind.LLM,
        eval_name=EvalName.GROUNDEDNESS,
        model=ModelChoices.TURING_FLASH,
    ),
]

trace_provider = register(
    project_type=ProjectType.EXPERIMENT,
    project_name="RAG-Cookbook",
    project_version_name="v1",
    eval_tags=eval_tags,
)

LangChainInstrumentor().instrument(tracer_provider=trace_provider)

Confirm the wiring works end to end before building the full comparison: retrieve for one question, answer it, and score the answer with fi.evals.

from fi.evals.templates import ContextRelevance

sample_docs = recursive_retriever.invoke("What is a transformer?")
sample_context = "\n\n".join(doc.page_content for doc in sample_docs)
sample_answer = llm.invoke(
    [{"role": "user", "content": f"Question: What is a transformer?\n\nContext: {sample_context}"}]
).content

check = evaluator.evaluate(
    eval_templates=[ContextRelevance(config={"check_internet": False})],
    inputs=[{"input": "What is a transformer?", "context": sample_context}],
    model_name="turing_flash",
)
print(check.eval_results[0].metrics[0].value)

You should see a single score between 0 and 1, for example 0.9. That confirms the chain, the tracer, and the evaluator are all wired correctly before the three-strategy comparison in the next steps.

Define a small, fixed test set and a retrieve-then-answer function, then score every answer on context relevance, chunk utilization, and groundedness.

from fi.evals.templates import ContextRelevance, ChunkUtilization, Groundedness

test_questions = [
    "What are the key differences between the transformer architecture in "
    "'Attention Is All You Need' and the bidirectional approach used in BERT?",
    "Explain the positional encoding mechanism in the original transformer "
    "paper and why it was necessary.",
    "How does GPT differ from BERT in terms of pretraining objective?",
]


def answer_question(question, retriever):
    retrieved_docs = retriever.invoke(question)
    context = "\n\n".join(doc.page_content for doc in retrieved_docs)
    messages = [{"role": "user", "content": f"Question: {question}\n\nContext: {context}"}]
    response = llm.invoke(messages)
    return context, response.content


def run_pipeline(retriever, questions):
    rows = []
    for question in questions:
        context, answer = answer_question(question, retriever)
        rows.append({"query": question, "context": context, "answer": answer})
    return rows


def score_pipeline(rows, model="turing_flash"):
    relevance_template = ContextRelevance(config={"check_internet": False})
    utilization_template = ChunkUtilization(config={"check_internet": False})
    groundedness_template = Groundedness(config={"check_internet": False})

    scored = []
    for row in rows:
        inputs = {"input": row["query"], "context": row["context"], "output": row["answer"]}
        relevance = evaluator.evaluate(eval_templates=[relevance_template], inputs=[inputs], model_name=model)
        utilization = evaluator.evaluate(eval_templates=[utilization_template], inputs=[inputs], model_name=model)
        grounded = evaluator.evaluate(eval_templates=[groundedness_template], inputs=[inputs], model_name=model)
        scored.append({
            **row,
            "context_relevance": relevance.eval_results[0].metrics[0].value,
            "chunk_utilization": utilization.eval_results[0].metrics[0].value,
            "groundedness": grounded.eval_results[0].metrics[0].value,
        })
    return scored


def average_scores(scored):
    keys = ["context_relevance", "chunk_utilization", "groundedness"]
    return {key: sum(row[key] for row in scored) / len(scored) for key in keys}


recursive_results = run_pipeline(recursive_retriever, test_questions)
recursive_scored = score_pipeline(recursive_results)
recursive_avg = average_scores(recursive_scored)
print(recursive_avg)

You should see three scores between 0 and 1 (illustrative, your run will vary):

{'context_relevance': 0.44, 'chunk_utilization': 0.80, 'groundedness': 0.33}

A low groundedness score here means the answer states things the retrieved context doesn’t support, usually because the chunk boundaries split a fact away from the sentence that needed it.

Fixed-size chunks cut mid-idea. SemanticChunker splits on embedding-distance breakpoints instead, keeping related sentences in the same chunk.

from langchain_experimental.text_splitter import SemanticChunker

semantic_chunker = SemanticChunker(embeddings, breakpoint_threshold_type="percentile")
semantic_splits = semantic_chunker.create_documents([doc.page_content for doc in docs])

semantic_vectorstore = Chroma.from_documents(
    documents=semantic_splits,
    embedding=embeddings,
    persist_directory="chroma_semantic",
)
semantic_retriever = semantic_vectorstore.as_retriever()

semantic_results = run_pipeline(semantic_retriever, test_questions)
semantic_scored = score_pipeline(semantic_results)
semantic_avg = average_scores(semantic_scored)
print(semantic_avg)

You should see an improvement on at least one metric over the baseline (illustrative):

{'context_relevance': 0.48, 'chunk_utilization': 0.86, 'groundedness': 0.67}

Some questions need more than one retrieval pass. Break the question into sub-questions first, retrieve for each, then answer from the combined context.

from langchain_core.runnables import RunnablePassthrough, RunnableLambda
from langchain_core.prompts import PromptTemplate

subq_prompt = PromptTemplate.from_template(
    "Break this question into 2-3 sub-questions needed to answer it fully.\n"
    "Question: {input}\n"
    "Format: one sub-question per line, prefixed with 'SUBQ:'"
)


def parse_subquestions(message):
    return [line.split("SUBQ:")[1].strip() for line in message.content.split("\n") if "SUBQ:" in line]


subquestion_chain = subq_prompt | llm | RunnableLambda(parse_subquestions)

answer_prompt = PromptTemplate.from_template(
    "Answer using all context below, connecting information across sub-questions.\n"
    "CONTEXTS:\n{contexts}\n\nQuestion: {input}\nFinal answer:"
)

subq_chain = (
    RunnablePassthrough.assign(subqs=lambda x: subquestion_chain.invoke(x["input"]))
    .assign(contexts=lambda x: "\n\n".join(
        doc.page_content for q in x["subqs"] for doc in semantic_retriever.invoke(q)
    ))
    .assign(answer=answer_prompt | llm)
)

subq_rows = []
for question in test_questions:
    result = subq_chain.invoke({"input": question})
    subq_rows.append({
        "query": question,
        "context": result["contexts"],
        "answer": result["answer"].content,
    })

subq_scored = score_pipeline(subq_rows)
subq_avg = average_scores(subq_scored)
print(subq_avg)

You should see the sub-question variant lead on chunk utilization and groundedness, at some cost to relevance (illustrative):

{'context_relevance': 0.46, 'chunk_utilization': 0.92, 'groundedness': 1.0}

Sub-question decomposition retrieves more targeted context per question, which raises chunk_utilization and groundedness. It costs an extra LLM call per question, so it’s slower and more expensive than the other two strategies.

Plot the three averages side by side to see the actual tradeoff, not just the printed numbers.

import matplotlib.pyplot as plt
import pandas as pd

summary = pd.DataFrame({
    "Recursive": recursive_avg,
    "Semantic": semantic_avg,
    "SubQ": subq_avg,
})
print(summary)

summary.plot(kind="bar", figsize=(10, 5))
plt.title("Context relevance, chunk utilization, and groundedness by chunking strategy")
plt.ylabel("Score (0-1)")
plt.xticks(rotation=0)
plt.tight_layout()
plt.show()

You should see a bar chart with three groups (illustrative values, from the runs above):

Context relevance stays nearly flat across all three strategies; chunk utilization and groundedness are what actually move

Every llm.invoke() and retriever.invoke() call from the three runs was captured by LangChainInstrumentor. Open the Future AGI dashboard to inspect them.

Open the Prototype tab and find the RAG-Cookbook project → open any trace to see the retrieval span, the LLM call, and the context_adherence and groundedness eval scores attached to it. This span-attached groundedness score is computed independently of the printed averages from Step 3-5, so the two don’t need to match.

Each row is one trace: expand it to see the exact chunks retrieved and the eval score attached to the LLM span