Key Features
- Anomaly Detection: Monitors continuously analyze data to detect anomalies in various metrics, ensuring you are alerted to potential issues promptly.
- Customizable Alerts: Define specific thresholds for metrics such as latency, cost, and evaluation metrics. Alerts can be set to trigger when these thresholds are exceeded.
- Email Notifications: Receive notifications directly to your email. You can configure alerts to send notifications to up to five email addresses, ensuring the right people are informed.
-
Metric Flexibility: Choose from a wide range of metrics to monitor, including:
- Latency
- Cost
- Token Usage
- Evaluation Metrics (e.g., toxicity)
How to Set Up Alerts
1. Choose the Metric/Evaluation
Select the metric you want to monitor from the dropdown menu. This includes:- System metrics (latency, cost, token usage)
- All types of evaluations:
- Pass/fail evaluations
- Numeric evaluations
- Deterministic evaluations
2. Define the Alert
Choose between two types of threshold settings:-
Auto Thresholding:
- This option will detect anomalies that are [
greater than
,less than
,greater or equal
to,less than or equal to
,equal to
] certain user-defined standard deviations.
- This option will detect anomalies that are [
-
Manual Thresholding:
- For pass/fail evaluations:
- Trigger an alert when the fail rate is [
greater than
,less than
,greater or equal to
,less than or equal to
,equal to
] a certain user-defined percentage.
- Trigger an alert when the fail rate is [
- For numeric evaluations:
- Trigger an alert when the evaluation value is [
greater than
,less than
,greater or equal to
,less than or equal to
,equal to
] a certain user-defined value.
- Trigger an alert when the evaluation value is [
- For deterministic metrics:
- Trigger an alert when certain values of the deterministic metric percentage is [
greater than
,less than
,greater or equal to
,less than or equal to
,equal to
] a certain user-defined percentage.
- Trigger an alert when certain values of the deterministic metric percentage is [
- For pass/fail evaluations: