​
Evaluation Using Interface

Input:

  • Required Inputs:
    • input: The original text column.
    • output: The translated text column.
  • Configuration Parameters:
    • Check Internet: Boolean - Whether to verify information using external sources for contextual validation.

Output:

  • Score: Percentage score between 0 and 100

Interpretation:

  • Higher scores: Indicate superior translation quality, reflecting better semantic accuracy, cultural appropriateness, and preservation of original meaning.
  • Lower scores: Suggest lower translation quality, potentially due to inaccuracies, cultural insensitivity, or unnatural phrasing.

​
Evaluation Using Python SDK

Click here to learn how to setup evaluation using the Python SDK.

Input TypeParameterTypeDescription
Required InputsinputstringThe original text.
outputstringThe translated text.
Configuration Parameterscheck_internetboolWhether to verify information using external sources for contextual validation.
OutputTypeDescription
ScorefloatReturns a score between 0 and 1, where higher values indicate superior translation quality.
from fi.testcases import TestCase
from fi.evals.templates import TranslationAccuracy

test_case = TestCase(
    input="Hello, how are you?",
    output="¡Hola, cómo estás?",
)

template = TranslationAccuracy(config={
    "check_internet": False
})

response = evaluator.evaluate(eval_templates=[template], inputs=[test_case])

print(f"Score: {response.eval_results[0].metrics[0].value}")
print(f"Reason: {response.eval_results[0].reason}")

​
What to Do When Translation Accuracy Evaluation Gives a Low Score

Reassess the evaluation criteria to ensure they are well-defined and aligned with the evaluation’s objectives, making adjustments if necessary to enhance their comprehensiveness and relevance. Analyse the translation for semantic accuracy, cultural appropriateness, and natural linguistic expression, identifying any discrepancies that may affect meaning. If inconsistencies are found, refine the translation to ensure it accurately conveys the original intent while maintaining contextual and cultural integrity.

​
Differentiating Translation Accuracy with Factual Accuracy

Translation Accuracy focuses on ensuring that the meaning and context of the input are accurately conveyed in the output, while Factual Accuracy verifies whether the output contains factually correct information based on the provided context.

Translation Accuracy assesses semantic accuracy, cultural appropriateness, and preservation of meaning, Factual Accuracy evaluates the correctness of factual statements relative to the given context.

Was this page helpful?