Semantic List Contains
Checks if a response semantically contains one or more reference phrases or keywords, using meaning-based matching rather than exact text.
Semantic List Contains checks whether a response covers a set of expected phrases or keywords, even when the exact wording differs. Run it when you’re checking for the presence of specific concepts rather than overall similarity.
What it does
Semantic List Contains is a statistical metric. It encodes the output and the expected phrase or phrases into vectors, compares them by cosine similarity against a configurable threshold, and returns whether the response semantically includes them.
Input
| Required Input | Type | Description |
|---|---|---|
output | string | The content to be evaluated for semantic list contains against the reference |
expected | string or List[string] | A single phrase or list of phrases that the response is expected to semantically include |
Output
| Field | Type | Description |
|---|---|---|
| Result | score | Higher values indicate stronger semantic coverage of the reference phrases |
| Reason | string | A plain-language explanation of the semantic list contains evaluation |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"semantic_list_contains",
expected="The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
output="The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high.",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"semantic_list_contains",
{
expected: "The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
output: "The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high."
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Semantic List Contains wherever a response needs to hit a checklist of concepts rather than match a reference text word for word.
- Text outputs graded against a set of expected keywords or key phrases
- Checks where the reference is a list of required points and any valid phrasing of each should count
- Cases where you need to know whether specific concepts are present, not how similar the whole response is overall
What to do when Semantic List Contains is low
Lower the similarity_threshold value if your use case allows relaxed semantic matches. Use match_all=False if partial coverage of the reference phrases is acceptable, rather than requiring every phrase to match.
Questions & Discussion