Protect Class

The Protect class provides a client for evaluating and protecting against unwanted content (such as toxicity, prompt injection, and more) using various metrics and rules. It leverages the Evaluator class and a set of built-in evaluation templates.

Initialization

def __init__(
    self,
    fi_api_key: Optional[str] = None,
    fi_secret_key: Optional[str] = None,
    fi_base_url: Optional[str] = None,
    evaluator: Optional[Evaluator] = None,
)

Arguments:

• fi_api_key (Optional[str]): API key for authentication. If not provided, will be read from environment variables.
• fi_secret_key (Optional[str]): Secret key for authentication. If not provided, will be read from environment variables.
• fi_base_url (Optional[str]): Base URL for the API. If not provided, will be read from environment variables.
• evaluator (Optional[Evaluator]): An instance of the Evaluator class to use for evaluations. If not provided, a new one will be created.

Raises:

• InvalidAuthError: If API key or secret key is missing.

Instance Methods

protect

Evaluates input strings against a set of protection rules and returns messages for any failed checks.

def protect(
    self,
    inputs: str,
    protect_rules: List[Dict],
    action: str = "Response cannot be generated as the input fails the checks",
    reason: bool = False,
    timeout: int = 300,
) -> List[str]

Arguments:

• inputs (str): The input string to evaluate.
• protect_rules (List[Dict]): List of protection rule dictionaries. Each rule must contain:
  • metric (str): Name of the metric to evaluate (e.g., "Toxicity", "Tone", "Sexism").
  • contains (List[str]): Values to check for in the evaluation results.
  • type (str): Either "any" or "all", specifying the matching logic.
  • action (str): Message to return when the rule is triggered.
  • reason (bool, optional): Whether to include the evaluation reason in the message.
• action (str, optional): Default message to return when a rule is triggered. Defaults to "Response cannot be generated as the input fails the checks".
• reason (bool, optional): Whether to include the evaluation reason in the message. Defaults to False.
• timeout (int, optional): Timeout for evaluations in seconds. Defaults to 300.

Returns:

• List[str]: List of protection messages for failed rules, or ["All checks passed"] if no rules are triggered.

Raises:

• ValueError: If inputs or protect_rules do not match the required structure.
• TypeError: If inputs contains non-string objects.

Example Usage

from fi.evals import Protect

protect_client = Protect(fi_api_key="your_api_key", fi_secret_key="your_secret_key")

rules = [
    {
        "metric": "Toxicity",
    },
    {
        "metric": "Sexism",
    },
]

result = protect_client.protect(
    inputs="Some user input to check.",
    protect_rules=rules,
    timeout=60,
)

print(result)