Python

What is traceAI?

Future AGI

Future AGI is an AI lifecycle platform designed to support enterprises throughout their AI journey. It combines rapid prototyping, rigorous evaluation, continuous observability, and reliable deployment to help build, monitor, optimize, and secure generative AI applications.

What is Future AGI?

Overview

Quickstart

Future AGI's proprietary models trained on a vast variety of datasets to perform evaluations

Future AGI Models

The Knowledge Base (KB) is the foundation for grounded, context-aware synthetic data generation and accurate evaluations. It ensures that every output whether it's data generation or evaluation is informed by your uploaded content, which is semantically processed and abstracted to reflect your organization’s unique domain.

This section provides a comprehensive framework for building and managing datasets to create, structure, and enhance data efficiently.

Annotations are essential for refining datasets, evaluating model outputs, and improving the quality of AI-generated responses.

Add Annotations

Column types define the kind of data that can be stored within a dataset column.

Change Column Type

Static columns store fixed values directly within a dataset. They do not require computation, external processing, or updates unless manually modified.

Create Static Column

Synthetic data generation allows you to create realistic, structured datasets without using real-world data. This powerful feature helps you

Create Synthetic Data

Add evaluations to a dataset in your account using SDK.

Add Evaluations through SDK

Evals for Prototype

Choose Winner

Future AGI's Observability platform delivers enterprise-grade monitoring and evaluation for large language models (LLMs) in production. Our solution provides deep visibility into LLM application performance through advanced telemetry data tracing and sophisticated evaluation metrics.

Future AGI's Eval tasks allows you to create and run automated tasks on your data. These tasks enable **automated workflows** to manage model **evaluation** at scale. They provide ways to operationalize evaluations and track ongoing results without requiring manual intervention. Users can create and run automated tasks on their data.

How to run evals?

Sessions in Future AGI are used to group traces, such as those from chatbot conversations. This feature allows users to view and analyze interactions between a human and AI, making it easier to build or debug chatbot applications.

Sessions

Alerts and Monitors in Future AGI are designed to detect anomalies and issues in your data. This feature helps you stay informed about critical metrics such as latency, cost, token usage, and evaluation metrics like toxicity, bias detection, and more.

Alerts and Monitors

Understanding how your LLM application performs is essential for optimization. Future AGI's observability platform helps you monitor critical metrics like cost, latency, and evaluation results through comprehensive tracing capabilities.

Concept

Future AGI’s Protect module brings real-time safety and policy enforcement directly into your GenAI application flow. Unlike traditional offline checks, Protect enables live monitoring and screening of every model input and output blocking or flagging harmful content before it reaches end users.

Future AGI’s Protect acts as a vital guardrail for AI applications, ensuring security, reliability, and ethical compliance during real-time interactions **across both text and audio**. By combining custom screening logic with Future AGI’s proprietary safety metrics, Protect enables teams to instantly detect, flag, and mitigate risks enhancing the integrity of AI applications without compromising performance.

How to Use Future AGI Protect

MCP Server

Centralized dashboard for monitoring usage, managing access, configuring integrations, tracking billing, and maintaining personal settings

Administration Panel

Find answers to common questions about Future AGI products.

Frequently Asked Questions (FAQ)

Power Of Combining Future AGI And Portkey

How to Integrate Portkey and FutureAGI for Automated LLM Evaluation

Release Notes

Installation of the Future AGI Python SDK and Tracing Libraries

Installation

Using the Future AGI Python SDK for running evaluations, listing available evaluators, and configuring the Evaluator client.

Evaluations

Reference for the Dataset class in the Future AGI Python SDK.

Datasets

Reference for the Protect class in the Future AGI Python SDK.

Protect

Reference for the KnowledgeBase class in the Future AGI Python SDK.

KnowledgeBase

Reference for tracing and telemetry in the Trace AI Python SDK.

Tracing

Reference for the Test Case classes in the Future AGI Python SDK.

Test Case

Build, evaluate, observe, and optimize your LLMs applications 10x faster without Human-in-the-Loop or Ground Truth

Hallucination

Multimodal AI

LLM as a Judge

Agent as a Judge

Evaluate Conversation

Evaluate Images

Ensure Safe and Inclusive AI

Identify Hallucination

Evaluate RAG Applications

Custom Evaluation Using Deterministic Eval

Validate Text Structures and Patterns

Validate JSON

Evaluate API Calls

Evaluate LLM Function Calling

Detect Prompt Injection

Evaluate Using LLM as a Judge

Evaluate Using Agent as a Judge

Ensure Data Compliance

Validate Emails

Ensure Valid Links

Evaluate Effective Use of Context Chunks

Detect Ambiguous Prompts Using Prompt Perplexity Eval

Evaluate Translation

Evaluate Summary

Verify If a Query Has Enough Context Using Context Sufficiency Eval

Conversation Coherence

Conversation Resolution

Deterministic Eval

Content Moderation

Context Adherence

Context Relevance

Completeness

Context Similarity

PII Detection

Toxicity

Tone

Sexist

Prompt Injection

Not Gibberish

Safe for Work Text Eval

Prompt/Instruction Adherence

Data Privacy Compliance

Is JSON

Regex

API Call

Custom Code

JSON Schema Validation

Context Sufficiency

Grading Criteria

Groundedness

Summarization Accuracy

Answer Similarity

Eval Output

Eval Context Retrieval

Eval Ranking

Eval Image Instruction

Score Eval

Summary Quality

Factual Accuracy

Translation Accuracy

Cultural Sensitivity

Bias Detection

LLM Function Calling

Length Evals

Contain

Prompt Perplexity

Chunk Attribution

Chunk Utilization

Valid Links

Is Email

Static Columns

Dynamic Columns

Learn how to set up and run experiments in Future AGI platform

How to Run Experiments

Understanding Observability

Components of Observability

What are Spans ?

What are Traces ?

What is OpenTelemetry?

Evals for Observability

Using Platform

Using Python SDK

How to Build and Incrementally Improve RAG Applications in Langchain

Creating Trustworthy RAGs for Chatbots

Master AI observability with FutureAGI. Track LLM performance, monitor metrics, and optimize Python apps. Step-by-step guide with examples.

Package	Description	Version
`traceAI-openai`	traceAI Instrumentation for OpenAI.
`traceAI-anthropic`	traceAI Instrumentation for Anthropic.
`traceAI-llamaindex`	traceAI Instrumentation for LlamaIndex.
`traceAI-langchain`	traceAI Instrumentation for LangChain.
`traceAI-mcp`	traceAI Instrumentation for MCP.
`traceAI-mistralai`	traceAI Instrumentation for MistralAI.
`traceAI-vertexai`	traceAI Instrumentation for VertexAI.
`traceAI-crewai`	traceAI Instrumentation for CrewAI.
`traceAI-haystack`	traceAI Instrumentation for Haystack.
`traceAI-litellm`	traceAI Instrumentation for liteLLM.
`traceAI-groq`	traceAI Instrumentation for Groq.
`traceAI-autogen`	traceAI Instrumentation for Autogen.
`traceAI-guardrails`	traceAI Instrumentation for Guardrails.
`traceAI-openai-agents`	traceAI Instrumentation for OpenAI Agents.
`traceAI-smolagents`	traceAI Instrumentation for SmolAgents.
`traceAI-dspy`	traceAI Instrumentation for DSPy.
`traceAI-bedrock`	traceAI Instrumentation for AWS Bedrock.
`traceAI-instructor`	traceAI Instrumentation for Instructor.

Introduction

Evaluation

Knowledge Base

Dataset

Prototype

Observe

Tracing

Optimization

Prompt Workbench

Protect

MCP

Admin & Settings

FAQs

What is traceAI?

Python

Introduction

Evaluation

Knowledge Base

Dataset

Prototype

Observe

Tracing

Optimization

Prompt Workbench

Protect

MCP

Admin & Settings

FAQs

​Python

Python