Synthetic Image Evaluator

Evaluates whether an image was generated by AI or captured by a camera/created by humans. 

result = evaluator.evaluate(
    eval_templates="synthetic_image_evaluator",
    inputs={
        "image": "https://www.esparklearning.com/app/uploads/2024/04/Albert-Einstein-generated-by-AI-1024x683.webp"
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)
import { Evaluator, Templates } from "@future-agi/ai-evaluation";

const evaluator = new Evaluator();

const result = await evaluator.evaluate(
  "synthetic_image_evaluator",
  {
    image: "https://www.esparklearning.com/app/uploads/2024/04/Albert-Einstein-generated-by-AI-1024x683.webp"
  },
  {
    modelName: "turing_flash",
  }
);

console.log(result);
Input
Required InputTypeDescription
imagestringURL or file path to the image to be evaluated.
Output
FieldDescription
ResultReturns Score representing the synthetic image evaluator, where higher values indicate confidence in the image being AI-generated.
ReasonProvides a detailed explanation of why the image was classified as AI-generated or not.

What to do If you get Undesired Results

If you’re evaluating images and the results don’t match your expectations:

  • For actual photographs mistakenly identified as synthetic:

    • Ensure the image has not been heavily processed or filtered
    • Check that the image doesn’t have unusual artifacts from compression or editing
    • Consider providing higher resolution versions if available

  • For synthetic images not being detected:

    • Be aware that newer AI generation models are becoming increasingly photorealistic
    • Some AI-generated images that were post-processed or combined with real photographs may be harder to detect
    • The evaluation works best with full images rather than small crops or heavily modified versions

Comparing Synthetic Image Evaluator with Similar Evals

  • Caption Hallucination: While Synthetic Image Evaluator determines if an image was artificially created, Caption Hallucination evaluates whether descriptions of images contain fabricated elements not visible in the image.
  • Toxicity: Synthetic Image Evaluator focuses on the creation method of images, whereas Toxicity evaluates whether content contains harmful elements.
Was this page helpful?

Questions & Discussion