OCR Evaluation
Evaluates the accuracy of optical character recognition output by checking extracted JSON content against the source PDF.
OCR Evaluation checks whether text extracted from a PDF into JSON faithfully represents the source document. Run it wherever document extraction feeds a downstream pipeline.
What it does
OCR Evaluation is an LLM-as-Judge eval. It reads the source PDF and the extracted JSON content, then scores how accurately the extraction represents the document.
Input
| Required Input | Type | Description |
|---|---|---|
input_pdf | string | The PDF document to verify against |
json_content | string | The JSON content extracted from OCR to evaluate |
Output
| Field | Type | Description |
|---|---|---|
| Result | score | Higher values indicate more accurate OCR extraction |
| Reason | string | A plain-language explanation of the OCR quality assessment |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"ocr_evaluation",
input_pdf="path/to/document.pdf",
json_content='{"name": "John Doe", "date": "2024-01-01", "amount": "$100.00"}',
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"ocr_evaluation",
{
input_pdf: "path/to/document.pdf",
json_content: '{"name": "John Doe", "date": "2024-01-01", "amount": "$100.00"}'
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run OCR Evaluation wherever a document goes through OCR and the extracted structure needs to be verified against the source.
- Document extraction pipelines, to confirm scanned PDFs are converted to accurate structured data
- Text and PDF/document workflows that feed downstream systems on extracted fields
- Quality checks after OCR tooling changes or model upgrades
What to do when OCR Evaluation fails
If the OCR evaluation score is lower than expected, check for poor scan quality or low-resolution images in the PDF, and verify that the OCR tool supports the fonts and languages present in the document.
Review the JSON structure to ensure it maps correctly to the document fields, and look for misinterpreted characters (e.g. 0 vs O, 1 vs l). Ensure tables and multi-column layouts are being parsed correctly, and consider pre-processing the PDF to improve contrast and clarity before OCR.
Questions & Discussion