​

Test Case Classes

The Test Case classes define the structure for test cases used in evaluations, including support for text, conversational, LLM, multimodal (image/audio), and more.

​

TestCase Class

Represents a general test case for evaluation.

class TestCase(BaseModel):
    text: Optional[str] = None
    document: Optional[str] = None
    input: Optional[str] = None
    output: Optional[str] = None
    prompt: Optional[str] = None
    criteria: Optional[str] = None
    actual_json: Optional[dict] = None
    expected_json: Optional[dict] = None
    expected_text: Optional[str] = None
    query: Optional[str] = None
    response: Optional[str] = None
    context: Union[List[str], str] = None

Attributes:

• text (Optional[str]): Text for the test case.
• document (Optional[str]): Document content.
• input (Optional[str]): Input string.
• output (Optional[str]): Output string.
• prompt (Optional[str]): Prompt used.
• criteria (Optional[str]): Evaluation criteria.
• actual_json (Optional[dict]): Actual JSON object.
• expected_json (Optional[dict]): Expected JSON object.
• expected_text (Optional[str]): Expected text output.
• query (Optional[str]): Query string.
• response (Optional[str]): Response string.
• context (Union[List[str], str]): Context for the test case.

​

ConversationalTestCase Class

Represents a conversational test case, consisting of a list of LLM test cases (messages).

class ConversationalTestCase(BaseModel):
    messages: List[LLMTestCase]

Attributes:

• messages (List[LLMTestCase]): List of LLM test case messages.

​

LLMTestCase Class

Represents a test case for LLM (Language Model) evaluation.

class LLMTestCase(BaseModel):
    query: str
    response: str
    context: Optional[Union[str, List[str]]] = None
    expected_response: Optional[str] = None

Attributes:

• query (str): The input query.
• response (str): The model’s response.
• context (Optional[Union[str, List[str]]]): Context for the test case.
• expected_response (Optional[str]): The expected response.

​

MLLMImage Class

Represents an image input for multimodal LLM test cases.

class MLLMImage(BaseModel):
    url: str
    local: Optional[bool] = None

Attributes:

• url (str): URL or local path to the image.
• local (Optional[bool]): Whether the image is local.

​

MLLMAudio Class

Represents an audio input for multimodal LLM test cases.

class MLLMAudio(BaseModel):
    url: str
    local: Optional[bool] = None
    is_plain_text: bool = False

Attributes:

• url (str): URL or local path to the audio file.
• local (Optional[bool]): Whether the audio is local.
• is_plain_text (bool): Whether the input is plain text (not audio).

​

MLLMTestCase Class

Represents a multimodal LLM test case, supporting image and audio inputs.

class MLLMTestCase(TestCase):
    image_url: Optional[Union[str, MLLMImage]] = None
    input_image_url: Optional[Union[str, MLLMImage]] = None
    output_image_url: Optional[Union[str, MLLMImage]] = None
    input_audio: Optional[Union[str, MLLMAudio]] = None
    call_type: Optional[str] = None

Attributes:

• image_url (Optional[Union[str, MLLMImage]]): Image input.
• input_image_url (Optional[Union[str, MLLMImage]]): Input image.
• output_image_url (Optional[Union[str, MLLMImage]]): Output image.
• input_audio (Optional[Union[str, MLLMAudio]]): Input audio.
• call_type (Optional[str]): Type of call (if applicable).

​

Example Usage

from fi.testcase import TestCase, LLMTestCase, ConversationalTestCase, MLLMTestCase

# Simple test case
tc = TestCase(input="What is the capital of France?", output="Paris")

# LLM test case
llm_tc = LLMTestCase(query="Who wrote 1984?", response="George Orwell")

# Conversational test case
conv_tc = ConversationalTestCase(messages=[llm_tc])

# Multimodal test case
mllm_tc = MLLMTestCase(
    input="Describe this image.",
    image_url="path/to/image.jpg"
)