Toxicity

Toxicity checks whether generated content contains harmful or offensive language. Run it to keep hate speech, threats, and abusive text out of what your model produces.

What it does

Toxicity is an LLM-as-Judge eval. It reads the output and flags whether it contains toxic or harmful language.

Input

Output

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

from fi.evals import evaluate

result = evaluate(
    "toxicity",
    output="Dear Sir, I hope this email finds you well. I look forward to any insights or advice you might have whenever you have a free moment",
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "toxicity",
  {
    output: "Dear Sir, I hope this email finds you well. I look forward to any insights or advice you might have whenever you have a free moment"
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Run Toxicity wherever a model’s output reaches a user or gets stored for later use.

• Text, audio, image, and chat outputs, to catch hate speech, threats, or abusive language
• Moderation pipelines that need to block harmful content before it’s published
• Safety checks on any surface where users can prompt the model into hostile territory

What to do when Toxicity is detected

If toxicity is detected in your response, the first step is to remove or rephrase harmful language to ensure the text remains safe and appropriate. Implementing content moderation policies can help prevent the dissemination of toxic language by enforcing guidelines for acceptable communication.

Additionally, enhancing toxicity detection mechanisms can improve accuracy, reducing false positives while ensuring that genuinely harmful content is effectively identified and addressed.