Evaluation Using Interface

Input:

  • Required Inputs:
    • input: The text content to evaluate for compliance.

Output:

  • Result: Returns ‘Passed’ if the content is compliant with guidelines and policies, ‘Failed’ if it’s non-compliant.
  • Reason: A detailed explanation of why the content was classified as compliant or non-compliant.

Evaluation Using Python SDK

Click here to learn how to setup evaluation using the Python SDK.

Input:

  • Required Inputs:
    • input: string - The text content to evaluate for compliance.

Output:

  • Result: Returns a list containing ‘Passed’ if the content is compliant with guidelines and policies, or ‘Failed’ if it’s non-compliant.
  • Reason: Provides a detailed explanation of the evaluation.
result = evaluator.evaluate(
    eval_templates="is_compliant", 
    inputs={
        "input": "Dear Sir, I hope this email finds you well. I look forward to any insights or advice you might have whenever you have a free moment"
    },
    model_name="turing_flash"
)

print(result.eval_results[0].metrics[0].value)
print(result.eval_results[0].reason)

Example Output:

['Passed']
The evaluation resulted in a 'Passed' determination because the text follows standard communication guidelines and does not contain any problematic content.

- The message uses **professional and respectful language** without any offensive terms, harmful content, or discriminatory statements.
- It follows a **standard email format** with appropriate greeting and closing.
- The content is **neutral in tone** and does not promote anything harmful, illegal, or against common content policies.
- The message **respects boundaries** by acknowledging the recipient's time constraints with the phrase "whenever you have a free moment."

A different outcome was not possible because there are no elements in the text that would violate content guidelines or standards.

What to do If you get Undesired Results

If the content is evaluated as non-compliant (Failed) and you want to improve it:

  • Remove any potentially offensive, harmful, or discriminatory language
  • Avoid content that could be interpreted as promoting illegal activities
  • Ensure respectful and professional tone throughout
  • Remove mentions of restricted topics based on relevant guidelines
  • Avoid aggressive, threatening, or harassing language
  • Eliminate content that could be interpreted as deceptive or manipulative
  • Check for privacy violations or sharing of sensitive information
  • Consider cultural sensitivities and avoid stereotyping

Comparing Is Compliant with Similar Evals

  • Content Safety Violation: Is Compliant provides a broader assessment of guideline adherence, while Content Safety Violation focuses specifically on detecting harmful, unsafe content.
  • Is Harmful Advice: Is Compliant evaluates overall policy adherence, whereas Is Harmful Advice specifically identifies potentially dangerous recommendations.
  • Toxicity: Is Compliant evaluates general policy compliance, while Toxicity specifically measures harmful or offensive language.