Skip to main content

​
Test Case Classes

The Test Case classes define the structure for test cases used in evaluations, including support for text, conversational, LLM, multimodal (image/audio), and more.

​
TestCase Class

Represents a general test case for evaluation. 
class TestCase(BaseModel):
    text: Optional[str] = None
    document: Optional[str] = None
    input: Optional[str] = None
    output: Optional[str] = None
    prompt: Optional[str] = None
    criteria: Optional[str] = None
    actual_json: Optional[dict] = None
    expected_json: Optional[dict] = None
    expected_text: Optional[str] = None
    query: Optional[str] = None
    response: Optional[str] = None
    context: Union[List[str], str] = None
Attributes:
  • text (Optional[str]): Text for the test case.
  • document (Optional[str]): Document content.
  • input (Optional[str]): Input string.
  • output (Optional[str]): Output string.
  • prompt (Optional[str]): Prompt used.
  • criteria (Optional[str]): Evaluation criteria.
  • actual_json (Optional[dict]): Actual JSON object.
  • expected_json (Optional[dict]): Expected JSON object.
  • expected_text (Optional[str]): Expected text output.
  • query (Optional[str]): Query string.
  • response (Optional[str]): Response string.
  • context (Union[List[str], str]): Context for the test case.

​
ConversationalTestCase Class

Represents a conversational test case, consisting of a list of LLM test cases (messages). 
class ConversationalTestCase(BaseModel):
    messages: List[LLMTestCase]
Attributes:
  • messages (List[LLMTestCase]): List of LLM test case messages.

​
LLMTestCase Class

Represents a test case for LLM (Language Model) evaluation. 
class LLMTestCase(BaseModel):
    query: str
    response: str
    context: Optional[Union[str, List[str]]] = None
    expected_response: Optional[str] = None
Attributes:
  • query (str): The input query.
  • response (str): The model’s response.
  • context (Optional[Union[str, List[str]]]): Context for the test case.
  • expected_response (Optional[str]): The expected response.

​
MLLMImage Class

Represents an image input for multimodal LLM test cases. 
class MLLMImage(BaseModel):
    url: str
    local: Optional[bool] = None
Attributes:
  • url (str): URL or local path to the image.
  • local (Optional[bool]): Whether the image is local.

​
MLLMAudio Class

Represents an audio input for multimodal LLM test cases. 
class MLLMAudio(BaseModel):
    url: str
    local: Optional[bool] = None
    is_plain_text: bool = False
Attributes:
  • url (str): URL or local path to the audio file.
  • local (Optional[bool]): Whether the audio is local.
  • is_plain_text (bool): Whether the input is plain text (not audio).

​
MLLMTestCase Class

Represents a multimodal LLM test case, supporting image and audio inputs. 
class MLLMTestCase(TestCase):
    image_url: Optional[Union[str, MLLMImage]] = None
    input_image_url: Optional[Union[str, MLLMImage]] = None
    output_image_url: Optional[Union[str, MLLMImage]] = None
    input_audio: Optional[Union[str, MLLMAudio]] = None
    call_type: Optional[str] = None
Attributes:
  • image_url (Optional[Union[str, MLLMImage]]): Image input.
  • input_image_url (Optional[Union[str, MLLMImage]]): Input image.
  • output_image_url (Optional[Union[str, MLLMImage]]): Output image.
  • input_audio (Optional[Union[str, MLLMAudio]]): Input audio.
  • call_type (Optional[str]): Type of call (if applicable).

​
Example Usage

from fi.testcase import TestCase, LLMTestCase, ConversationalTestCase, MLLMTestCase

# Simple test case
tc = TestCase(input="What is the capital of France?", output="Paris")

# LLM test case
llm_tc = LLMTestCase(query="Who wrote 1984?", response="George Orwell")

# Conversational test case
conv_tc = ConversationalTestCase(messages=[llm_tc])

# Multimodal test case
mllm_tc = MLLMTestCase(
    input="Describe this image.",
    image_url="path/to/image.jpg"
)