​
is-code

This evaluation template checks whether a given text contains valid programming code. It can identify code snippets across multiple programming languages and distinguish code from natural language.

​
Interface Usage

result = evaluator.evaluate(
    eval_templates="is_code", 
    inputs={
        "input": """  
                  def fibonacci(n):  
                      a, b = 0, 1  
                      for _ in range(n):  
                          print(a)  
                          a, b = b, a + b  
                  """
    },
    model_name="turing_flash"
)

print(result.eval_results[0].metrics[0].value)
print(result.eval_results[0].reason)

​
Python SDK Usage

from futureagi import Evaluator

# Initialize the evaluator
evaluator = Evaluator(api_key="your_api_key")

# Evaluate whether text contains valid code
result = evaluator.evaluate(
    eval_templates="is_code", 
    inputs={
        "input": """  
                  def fibonacci(n):  
                      a, b = 0, 1  
                      for _ in range(n):  
                          print(a)  
                          a, b = b, a + b  
                  """
    },
    model_name="turing_flash"
)

# Access the result
contains_code = result.eval_results[0].metrics[0].value
reason = result.eval_results[0].reason

print(f"Contains code: {contains_code}")
print(f"Reason: {reason}")

​
Example Output

True
The input is clearly Python code that defines a function called 'fibonacci'. It has proper Python syntax including function definition with 'def', parameter declaration, variable assignments, a for loop with range(), indentation to denote code blocks, and function logic. This is a valid implementation of a function to print Fibonacci numbers.

​
Troubleshooting

If you encounter issues with this evaluation:

  • Ensure the code is properly formatted with appropriate indentation and syntax for its language
  • This evaluation can identify code across common programming languages like Python, JavaScript, Java, etc.
  • Mixed content (code with extensive natural language explanations) might yield uncertain results
  • Code snippets with syntax errors might still be identified as code, as the evaluation focuses on structural patterns
  • is-json: Specifically checks if the text is a valid JSON structure
  • regex: Validates if a string matches a specified pattern, useful for validating specific code patterns
  • json-schema: Evaluates if a JSON object conforms to a specified schema