Alerts and Monitors

​

Key Features

• Anomaly Detection: Monitors continuously analyze data to detect anomalies in various metrics, ensuring you are alerted to potential issues promptly.

• Customizable Alerts: Define specific thresholds for metrics such as latency, cost, and evaluation metrics. Alerts can be set to trigger when these thresholds are exceeded.

• Email Notifications: Receive notifications directly to your email. You can configure alerts to send notifications to up to five email addresses, ensuring the right people are informed.

• Metric Flexibility: Choose from a wide range of metrics to monitor, including:

  • Latency
  • Cost
  • Token Usage
  • Evaluation Metrics (e.g., toxicity)

​

How to Set Up Alerts

​

1. Choose the Metric/Evaluation

Select the metric you want to monitor from the dropdown menu. This includes:

• System metrics (latency, cost, token usage)
• All types of evaluations:
  • Pass/fail evaluations
  • Numeric evaluations
  • Deterministic evaluations

​

2. Define the Alert

Choose between two types of threshold settings:

• Auto Thresholding:

  • This option will detect anomalies that are [greater than, less than, greater or equal to, less than or equal to, equal to] certain user-defined standard deviations.

• Manual Thresholding:

  • For pass/fail evaluations:
    • Trigger an alert when the fail rate is [greater than, less than, greater or equal to, less than or equal to, equal to] a certain user-defined percentage.
  • For numeric evaluations:
    • Trigger an alert when the evaluation value is [greater than, less than, greater or equal to, less than or equal to, equal to] a certain user-defined value.
  • For deterministic metrics:
    • Trigger an alert when certain values of the deterministic metric percentage is [greater than, less than, greater or equal to, less than or equal to, equal to] a certain user-defined percentage.

​

3. Configure Notifications

Enter the email addresses to receive notifications. You can add up to five email addresses to receive emails when alerts are triggered.

By using Alerts and Monitors, you can proactively manage your system’s performance and ensure timely responses to any issues that occur.