​
Evaluation Using Interface

Input:
  • Required Inputs:
    • actual_json: The JSON content column to validate.
    • expected_json: The reference JSON schema column to validate against.
  • Configuration Parameters:
    • validations: List of strings - Specific validation criteria to apply.
Output:
  • Result: Passed / Failed
Interpretation:
  • Passed: Indicates that the actual_json content successfully conforms to the structure and rules defined in the expected_json schema, according to the specified validations.
  • Failed: Signifies that the actual_json content does not match the expected_json schema based on the specified validations (e.g., missing required fields, incorrect data types, invalid structure)

​
Evaluation Using Python SDK

Click here to learn how to setup evaluation using the Python SDK.
Input TypeParameterTypeDescription
Required Inputsactual_jsonobject or stringThe JSON content to validate.
expected_jsonobject or stringThe reference JSON schema to validate against.
Configuration Parametersvalidationslist[string]List of specific validation criteria (e.g., ["type_check"]).
OutputTypeDescription
ResultfloatReturns 1.0 if the JSON matches the schema (Pass), 0.0 otherwise (Fail).
from fi.evals import Evaluator
from fi.testcases import TestCase
from fi.evals.templates import JsonSchemeValidation

test_case = TestCase(
    actual_json={
        "name": "John Doe",
        "age": 30,
        "email": "john@example.com"
    },
    expected_json={
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "age": {"type": "integer"},
            "email": {"type": "string"}
        },
        "required": ["name", "age", "email"]
    }
)

template = JsonSchemeValidation(
    config={
        "validations": [
            "type_check",
            "required_fields"
        ]
    }
)

evaluator = Evaluator(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key",
    fi_base_url="<https://api.futureagi.com>"
)

response = evaluator.evaluate(eval_templates=[template], inputs=[test_case], model_name="turing_flash")

score = response.eval_results[0].metrics[0].value
What to do when JSON Scheme Validation Fails Start with a schema review by checking the actual JSON structure against the expected schema and ensuring alignment with the defined validation criteria. Then, review the validation criteria to confirm that they are appropriate for the use case, complete in covering all necessary constraints, and free from conflicting rules that might cause unnecessary validation failures.
Differentiating JSON Scheme Validation from Is JSON Eval While both evaluations handle JSON, their scope and complexity differ. JSON Schema Validation ensures that a JSON structure adheres to specific rules and validation criteria, while Is JSON simply checks whether the content is a valid JSON format. JSON Schema Validation involves a more complex process, verifying structure, data types, and constraints against predefined schemas, whereas Is JSON performs a basic syntax check for correct JSON formatting.