Conversation resolution refers to the ability of an LLM model to effectively and accurately resolve a conversation. It ensures that the exchange of information or ideas between participants reaches a satisfactory conclusion. In the context of natural language processing and conversational AI, resolution is a critical metric to evaluate how well an AI or chatbot can bring conversations to appropriate and meaningful conclusions.

​
Required Parameters

ParameterDescriptionRequired
messagesArray of conversation messagesYes

​
Configuration

The evaluation accepts the following configuration:

ParameterTypeDescriptionRequired
modelstringThe LLM model to use for evaluationYes
from fi.evals import ConversationResolution

resolution = ConversationResolution(config = {"model": "gpt-4o-mini"})

​
Test Case Setup

The evaluation requires a list of messages between the user and assistant:

from fi.test_cases import ConversationalTestCase

test_case = ConversationalTestCase(
    messages=[
        {"role": "user", "content": "Hello, how are you?"},
        {"role": "assistant", "content": "I am fine, thank you."}
    ]
)

​
Client Setup

Initialize the evaluation client with your API credentials:

from fi.evals import EvalClient

evaluator = EvalClient(
    fi_api_key="your_api_key", 
    fi_secret_key="your_secret_key"
)

​
Complete Example

from fi.evals import ConversationResolution, EvalClient
from fi.test_cases import ConversationalTestCase

resolution = ConversationResolution(
    config = {"model": "gpt-4o-mini"}
)

test_case = ConversationalTestCase(
    messages=[
        LLMTestCase(
            query="I need help with my math homework",
            response="I'd be happy to help! What specific math problem are you working on?"
        ),
        LLMTestCase(
            query="I'm trying to solve this equation: 2x + 5 = 13",
            response="Let's solve this step by step:\n1. Subtract 5 from both sides: 2x = 8\n2. Divide both sides by 2: x = 4\nThe solution is x = 4. Would you like me to verify this answer with you?"
        ),
        LLMTestCase(
            query="Yes, please check if it's correct",
            response="Let's verify:\nIf x = 4, then:\n2(4) + 5 = 8 + 5 = 13\nThe equation balances, so x = 4 is correct! Is there anything else you'd like help with?"
        ),
        LLMTestCase(
            query="No, that's all. Thank you!",
            response="You're welcome! Good luck with your math studies. Feel free to ask if you need help with any other problems."
        ),
    ]
)

evaluator = EvalClient(fi_api_key="your_api_key", fi_secret_key="your_secret_key")
result = evaluator.evaluate(resolution, test_case)
print(result)
Conversation CoherenceCustom