Evaluate API Calls

AI applications often rely on APIs to fetch real-time data (e.g., stock prices, news, or user-specific information) to enrich their responses. APIs allow AI systems to integrate with third-party services or internal systems, making them more versatile. Without properly functioning API calls, the AI model might provide incomplete, incorrect, or outdated information, leading to poor user experiences and loss of trust. Evaluating API calls ensures that the AI application understands the user’s intent, executes correctly and receives valid response.

The Eval provided by the Future AGI ensures the API calls generated by the AI system work as intended by validating the response against predefined success criteria. This evaluation is essential for maintaining the reliability of AI workflows and ensuring the system meets user expectations.

Click here to read the eval definition of API Call

​

a. Using Interface

Required Inputs

• Response: Column that has the API’s response to the call. This is the data the AI system will process to construct its output.
• URL: The endpoint for the API call. This determines the type of request being made (e.g., fetching data or executing an action).
• Headers: Any additional information required for the API call, such as authentication tokens or content type specifications.
• Payload: The data sent with the request. This can include parameters, user inputs, or filters relevant to the operation.

Evaluation Output

• Pass: Indicates that the API call worked as expected, providing a valid response that aligns with the requirements.
• Fail: Indicates an issue, such as invalid response data, incorrect structure, or an error in the API call process.

​

b. Using SDK

from fi.evals import EvalClient
from fi.evals.templates import ApiCall
from fi.testcases import TestCase

test_case = TestCase(
    response='{"temperature": 75, "conditions": "sunny"}'
)

template = ApiCall(
    config={
        "url": "https://api.weather.com/v1/current?apiKey=YOUR_WEATHER_API_KEY",  # Add API key in URL
        "headers": {
            "apiKey": "YOUR_WEATHER_API_KEY",
            "Content-Type": "application/json"
        },
        "payload": {
            "city": "London",
            "units": "fahrenheit"
        }
    }
)

evaluator = EvalClient(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key",
    fi_base_url="https://api.futureagi.com"
)

response = evaluator.evaluate(eval_templates=[template], inputs=[test_case])