LLM API cost alerts

• OpenAI hard limit: Settings → Limits → Usage limits → Monthly budget (cuts off API)
• Anthropic spend limit: Account → Plans & Billing → Spend limit
• Soft alert every 5 min: cron script fetches the usage API → if > daily_target × 1.2 → Telegram
• Attribution: tag every LLM call with user_id + endpoint in a JSON log for post-mortems
• Prompt-loop defense: max_tokens (50-500 for chat, 4K for long-form), 30 s timeout, retry no more than 1×

# Cron: every 5 min — heartbeat to enterno.io with current daily spend
# /etc/cron.d/llm-cost-watch
*/5 * * * * www-data /usr/bin/python3 /opt/llm-cost-check.py

# llm-cost-check.py (simplified)
import requests, os
from datetime import date

spent = fetch_today_usage()  # your billing
budget = 50.0  # USD/day

if spent > budget * 1.2:
    requests.post('https://enterno.io/api/heartbeat',
        params={'token': os.environ['HEARTBEAT_TOKEN'],
                'status': 'critical',
                'msg': f'LLM spend ${spent:.2f} > 120% of ${budget}/day'})
else:
    requests.post('https://enterno.io/api/heartbeat',
        params={'token': os.environ['HEARTBEAT_TOKEN'], 'status': 'ok'})

• LLM API latency monitoring
• What is a heartbeat monitor

To set up LLM API cost alerts effectively, configure a budget cap using your cloud provider's billing console and implement anomaly detection using monitoring tools like AWS CloudWatch or Google Cloud Monitoring. Set thresholds for alert notifications when spending approaches your budget cap, and integrate alerting mechanisms such as email or SMS for real-time updates.

Establishing a budget cap is crucial to managing costs effectively when using LLM APIs. Here’s how to configure a budget cap using AWS as an example:

1. Access the AWS Billing Console: Log into your AWS account and navigate to the Billing Dashboard.
2. Create a Budget: Select 'Budgets' from the side menu and then click on 'Create budget.'
3. Define Budget Type: Choose 'Cost budget' and click 'Set your budget.'
4. Set Budget Amount: Specify your budget limit. For instance, set a monthly budget of $500 for LLM API calls.
5. Configure Alerts: Under 'Set alerts,' specify the notification threshold. For example, set an alert for when costs exceed 80% of your budget, which would be $400 in this case.
6. Choose Notification Channels: You can opt to receive alerts via email or SMS. Enter your contact information to receive notifications.
7. Review and Create: Review your settings and click 'Create budget' to finalize the setup.

This setup allows you to monitor LLM API usage against your budget to prevent unexpected costs.

In addition to budget caps, implementing anomaly detection can help identify unexpected spikes in LLM API usage, which could lead to increased costs. Here’s how to set up anomaly detection using Google Cloud Monitoring:

1. Access Google Cloud Console: Navigate to the Google Cloud Console and select 'Monitoring.'
2. Create an Alert Policy: Click 'Alerting' and then 'Create Policy.'
3. Select Condition Type: Choose 'Metrics' as the condition type. For LLM API costs, select the relevant metric, such as 'Total API Cost.'
4. Configure Anomaly Detection: In the condition configuration, select 'Anomaly Detection' and specify the threshold. For example, set a threshold that triggers an alert if costs exceed the average by 20% over a defined period.
5. Set Notification Channels: Choose how you want to be notified (e.g., email, SMS, or Slack) when an anomaly is detected.
6. Review and Save: Review your settings and click 'Save' to activate the alert policy.

This approach allows you to proactively manage costs and react quickly to unusual spending patterns, ensuring your LLM API usage remains within budget.