EvalTags

Attach evaluations to traces with EvalTags.

EvalTags let you configure automatic evaluations that run server-side on your traced spans. Attach them during register() and the platform scores spans as they arrive.

from fi_instrumentation import register
from fi_instrumentation.fi_types import (
    ProjectType, EvalTag, EvalTagType,
    EvalSpanKind, EvalName, ModelChoices
)

trace_provider = register(
    project_name="my-project",
    project_type=ProjectType.EXPERIMENT,
    project_version_name="v1.0",
    eval_tags=[
        EvalTag(
            type=EvalTagType.OBSERVATION_SPAN,
            value=EvalSpanKind.LLM,
            eval_name=EvalName.GROUNDEDNESS,
            model=ModelChoices.TURING_FLASH,
        ),
        EvalTag(
            type=EvalTagType.OBSERVATION_SPAN,
            value=EvalSpanKind.LLM,
            eval_name=EvalName.TOXICITY,
            model=ModelChoices.TURING_FLASH,
        ),
    ],
)

import {
  register, ProjectType, EvalTag,
  EvalTagType, EvalSpanKind, EvalName, ModelChoices
} from "@traceai/fi-core";

const tracerProvider = register({
  projectName: "my-project",
  projectType: ProjectType.EXPERIMENT,
  projectVersionName: "v1.0",
  evalTags: [
    await EvalTag.create({
      type: EvalTagType.OBSERVATION_SPAN,
      value: EvalSpanKind.LLM,
      eval_name: EvalName.GROUNDEDNESS,
      model: ModelChoices.TURING_FLASH,
    }),
    await EvalTag.create({
      type: EvalTagType.OBSERVATION_SPAN,
      value: EvalSpanKind.LLM,
      eval_name: EvalName.TOXICITY,
      model: ModelChoices.TURING_FLASH,
    }),
  ],
});

Note

EvalTag.create() is async in TypeScript because it validates the eval configuration with the server.

using FIInstrumentation;
using FIInstrumentation.Types;

var tracer = TraceAI.Register(opts =>
{
    opts.ProjectName = "my-project";
    opts.ProjectType = ProjectType.Experiment;
    opts.ProjectVersionName = "v1.0";
    opts.EvalTags = new List<EvalTag>
    {
        new EvalTag(EvalSpanKind.Llm, EvalName.Groundedness)
        {
            Model = ModelChoices.TuringFlash,
        },
        new EvalTag(EvalSpanKind.Llm, EvalName.Toxicity)
        {
            Model = ModelChoices.TuringFlash,
        },
    };
});

EvalSpanKind

Which span types to evaluate:

Value	Description
`LLM`	Language model calls
`RETRIEVER`	Document retrieval spans
`TOOL`	Tool/function calls
`AGENT`	Agent spans
`EMBEDDING`	Embedding generation
`RERANKER`	Re-ranking operations

ModelChoices

Which evaluation model to use:

Value	Description
`TURING_FLASH`	Fast evaluation model
`TURING_SMALL`	Small evaluation model
`TURING_LARGE`	High-accuracy evaluation model
`PROTECT`	Safety-focused model
`PROTECT_FLASH`	Fast safety model

Note

EvalTags only work with ProjectType.EXPERIMENT. For production monitoring without evals, use ProjectType.OBSERVE.

Was this page helpful?

Questions & Discussion