Conversation coherence refers to the logical flow, relevance, and consistency of dialogue in a conversation. It ensures that the exchange of information or ideas between participants aligns with the context and builds meaningfully upon prior interactions. In the context of natural language processing and conversational AI, coherence is a critical metric to evaluate how well an AI or chatbot can maintain a contextually relevant and natural conversation.

​
Required Parameters

ParameterDescriptionRequired
messagesArray of conversation messagesYes

​
Configuration

The evaluation accepts the following configuration:

ParameterTypeDescriptionRequired
modelstringThe LLM model to use for evaluationYes
from fi.evals import ConversationCoherence

coherence = ConversationCoherence(config = {"model": "gpt-4o-mini"})

​
Test Case Setup

The evaluation requires a list of messages between the user and assistant:

from fi.test_cases import ConversationalTestCase

test_case = ConversationalTestCase(
    messages=[
        {"role": "user", "content": "Hello, how are you?"},
        {"role": "assistant", "content": "I am fine, thank you."}
    ]
)

​
Client Setup

Initialize the evaluation client with your API credentials:

from fi.evals import EvalClient

evaluator = EvalClient(
    fi_api_key="your_api_key", 
    fi_secret_key="your_secret_key"
)

​
Complete Example

from fi.evals import ConversationCoherence, EvalClient
from fi.test_cases import ConversationalTestCase

# Initialize the coherence evaluator
coherence = ConversationCoherence(
    config = {"model": "gpt-4o-mini"}
)

# Create a test case
test_case = ConversationalTestCase(
        messages=[
            LLMTestCase(
                query="I need help with my math homework",
                response="I'd be happy to help! What specific math problem are you working on?"
            ),
            LLMTestCase(
                query="I'm trying to solve this equation: 2x + 5 = 13",
                response="Let's solve this step by step:\n1. Subtract 5 from both sides: 2x = 8\n2. Divide both sides by 2: x = 4\nThe solution is x = 4. Would you like me to verify this answer with you?"
            ),
            LLMTestCase(
                query="Yes, please check if it's correct",
                response="Let's verify:\nIf x = 4, then:\n2(4) + 5 = 8 + 5 = 13\nThe equation balances, so x = 4 is correct! Is there anything else you'd like help with?"
            ),
            LLMTestCase(
                query="No, that's all. Thank you!",
                response="You're welcome! Good luck with your math studies. Feel free to ask if you need help with any other problems."
            ),
        ]
    )

# Run the evaluation
evaluator = EvalClient(fi_api_key="your_api_key", fi_secret_key="your_secret_key")
result = evaluator.evaluate(coherence, test_case)
print(result)
QuickstartConversation Resolution