​
Evaluation Using Interface

Input:
  • Required Inputs:
    • response: The text content column to evaluate for coherence.
Output:
  • Score: Float score between 0 and 1
Interpretation:
  • Higher scores: Indicate that the response content is more coherent, logical, and flows naturally.
  • Lower scores: Suggest that the response content is disjointed, illogical, or difficult to understand.

​
Evaluation Using Python SDK

Click here to learn how to setup evaluation using the Python SDK.
Input TypeParameterTypeDescription
Required InputsresponsestringThe text content to evaluate for coherence.
OutputTypeDescription
ScoreboolReturns a score between 0 and 1, where higher values indicate more coherent and meaningful content.
from fi.evals import Evaluator
from fi.testcases import TestCase
from fi.evals.templates import NotGibberishText

evaluator = Evaluator(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key",
    fi_base_url="<https://api.futureagi.com>"
)

gibberish_eval = NotGibberishText()

test_case = TestCase(
    response="This is a sample text to check for gibberish text"
)

result = evaluator.evaluate(eval_templates=[gibberish_eval], inputs=[test_case], model_name="turing_flash")
gibberish_result = result.eval_results[0].metrics[0].id
What to do when Gibberish is Detected If gibberish content is detected, it should be removed or flagged for further review. Requesting a regeneration of the problematic content can help ensure clarity and coherence. Implementing quality checks before content delivery helps prevent incoherent outputs from reaching users. Enhancing gibberish detection mechanisms and updating language models can reduce the likelihood of generating meaningless text.
Differentiating Not Gibberish Text Eval with Safe for Work Text Eval Not Gibberish Text eval ensures that the generated content is coherent, structured, and meaningful, preventing outputs that are random or incomprehensible. In contrast, Safe for Work Text evaluates whether the content is appropriate for professional or public environments, ensuring it does not contain explicit, offensive, or overly casual language. While Not Gibberish focuses on linguistic integrity, Safe for Work prioritises content suitability for workplace and general audiences.