Is Helpful

Evaluates whether the response is helpful in solving the user problem or answering their question 

result = evaluator.evaluate(
    eval_templates="is_helpful",
    inputs={
        "input": "Why doesn't honey go bad?",
        "output": "Honey doesn't spoil because its low moisture and high acidity prevent the growth of bacteria and other microbes."
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)
import { Evaluator, Templates } from "@future-agi/ai-evaluation";

const evaluator = new Evaluator();

const result = await evaluator.evaluate(
  "is_helpful",
  {
    input: "Why doesn't honey go bad?",
    output: "Honey doesn't spoil because its low moisture and high acidity prevent the growth of bacteria and other microbes."
  },
  {
    modelName: "turing_flash",
  }
);

console.log(result);
Input
Required InputTypeDescription
inputstringUser query to the model
outputstringModel’s response to the user query
Output
FieldDescription
ResultReturns Passed if the response is helpful, or Failed if it’s not
ReasonProvides a detailed explanation of the evaluation

What to Do When Is Helpful Score is Low

  • Ensure that both the input (user query) and output (AI response) parameters are provided
  • The helpfulness evaluation works best when the context of the request is clear
  • If evaluating complex responses, make sure the entire response is included
  • Consider combining with other evaluations like completeness for more comprehensive assessment

Comparing Is Helpful with Similar Evals

  • Completeness: Is Helpful evaluates whether the response is useful overall, while Completeness checks if all aspects of the query are addressed.
  • Task Completion: Is Helpful assesses general usefulness, while Task Completion checks if a specific requested task was accomplished.
  • Instruction Adherence: Is Helpful evaluates usefulness, while Instruction Adherence evaluates if the response follows specific instructions.
  • Is Concise: Is Helpful focuses on effectiveness, while Is Concise assesses whether the response avoids unnecessary verbosity.
Was this page helpful?

Questions & Discussion