Monitoring LLM API Costs: Preventing Bill Surprises with Custom Metrics

It was a Monday morning, and my coffee tasted particularly bitter. I'd just opened the cloud billing dashboard, and a number stared back at me that made my stomach drop. Our LLM API usage for the past month had skyrocketed, blowing past our usual spend by a staggering 300%. There was no obvious feature launch, no massive user influx, just a quiet, insidious creep that had gone unnoticed until the bill arrived. My heart pounded – how could I have let this happen? As the lead developer, the buck stopped with me.

This wasn't just a financial hit; it was a wake-up call. We were leveraging powerful generative AI models, and while the capabilities were transformative, the costs were opaque by default. The problem wasn't just *that* we were spending money, but *where* and *why*. I needed a system, and I needed it fast, to gain visibility and control over our LLM API expenditure before the next bill arrived.

The Blind Spot: Why Standard Billing Isn't Enough

Most LLM providers offer some form of billing dashboard, but they are often high-level, showing total spend or token counts across your entire account. For a growing application with multiple features leveraging different models for various tasks, this aggregated view is nearly useless for debugging a cost spike. I couldn't tell if the increase was due to:

A specific feature being overused?
An inefficient prompt generating too many output tokens? (A topic I delved into deeply in How I Tuned Prompt Engineering for Maximum AI Cost Savings.)
A bug causing excessive API calls?
A new model costing significantly more than anticipated?

I needed granular data, tied directly to our application's operational context. My initial, naive approach of just summing tokens client-side was quickly abandoned. It was unreliable, prone to errors if an API call failed before logging, and didn't account for varying token prices across models or input/output differences.

Building My LLM API Cost Monitoring System

My goal was clear: capture every relevant detail of an LLM API call, centralize it, and then extract actionable metrics for real-time monitoring and alerting. I decided on a multi-stage approach:

Centralized Structured Logging: Intercept every API call and log its details.
Log-Based Metrics Extraction: Transform these logs into quantifiable metrics.
Visualization & Alerting: Create dashboards and set up alerts for anomalies.
Cost Attribution: Link costs back to specific features and components.

Phase 1: Intercepting and Logging Every API Call

The first step was to instrument our LLM API client. We use a custom Go wrapper around the various LLM provider SDKs (OpenAI, Anthropic, etc.). This was the perfect place to inject our logging logic. For every request and response, I decided to capture the following:

timestamp: When the call occurred.
model_name: The specific LLM model used (e.g., gpt-4-turbo, claude-3-opus-20240229).
input_tokens: The number of tokens in the prompt.
output_tokens: The number of tokens generated in the response.
api_key_id: An identifier for the API key used (useful for multi-tenant setups or isolating specific service usage).
feature_name: The application feature or component making the call (e.g., "blog_post_generator", "headline_rewriter", "summary_service"). This was crucial for attribution.
estimated_cost_usd: A calculated estimate of the cost for *this specific call*.
success: Boolean indicating if the API call was successful.
error_message: If an error occurred.

Calculating estimated_cost_usd on the fly was important for immediate feedback. I maintained a simple in-memory map of token prices (which I updated periodically from provider documentation). This allowed me to log a cost estimate directly with each event.


package llmclient

import (
    "context"
    "encoding/json"
    "fmt"
    "log"
    "time"

    "github.com/google/uuid"
    "go.uber.org/zap" // Assuming Zap for structured logging
)

// TokenPrices maps model names to their input/output token costs per 1K tokens
var TokenPrices = map[string]struct {
    InputPerK float64
    OutputPerK float64
}{
    "gpt-4-turbo":          {InputPerK: 10.00 / 1000, OutputPerK: 30.00 / 1000}, // Example: $10/1K input, $30/1K output
    "claude-3-opus-20240229": {InputPerK: 15.00 / 1000, OutputPerK: 75.00 / 1000}, // Example: $15/1K input, $75/1K output
    // ... add other models
}

type LLMCallLog struct {
    Timestamp      time.Time `json:"timestamp"`
    ModelName      string    `json:"modelName"`
    InputTokens    int       `json:"inputTokens"`
    OutputTokens   int       `json:"outputTokens"`
    APIKeyID       string    `json:"apiKeyId"`
    FeatureName    string    `json:"featureName"`
    EstimatedCostUSD float64   `json:"estimatedCostUSD"`
    Success        bool      `json:"success"`
    ErrorMessage   string    `json:"errorMessage,omitempty"`
}

// LLMClientWrapper wraps the actual LLM API client
type LLMClientWrapper struct {
    logger *zap.Logger
    // ... actual LLM SDK client
}

func NewLLMClientWrapper(logger *zap.Logger) *LLMClientWrapper {
    return &LLMClientWrapper{
        logger: logger,
        // ... initialize actual client
    }
}

// CallLLM simulates an LLM API call and logs its details
func (w *LLMClientWrapper) CallLLM(ctx context.Context, model string, prompt string, featureName string) (string, error) {
    startTime := time.Now()
    callLog := LLMCallLog{
        Timestamp:   startTime,
        ModelName:   model,
        FeatureName: featureName,
        APIKeyID:    "prod-key-001", // Or retrieve dynamically
    }

    // Simulate API call and token counting
    // In a real scenario, these would come from the LLM provider's response
    simulatedInputTokens := len(prompt) / 4 // Rough estimate: 4 chars per token
    simulatedOutputTokens := 200 // Simulate 200 output tokens

    callLog.InputTokens = simulatedInputTokens
    callLog.OutputTokens = simulatedOutputTokens

    // Calculate estimated cost
    if prices, ok := TokenPrices[model]; ok {
        callLog.EstimatedCostUSD = (float64(simulatedInputTokens) * prices.InputPerK) +
                                 (float64(simulatedOutputTokens) * prices.OutputPerK)
    } else {
        w.logger.Warn("Unknown model for cost calculation", zap.String("model", model))
    }

    // Simulate API success/failure
    var response string
    var err error
    if uuid.New().ID()%5 == 0 { // Simulate 20% error rate
        err = fmt.Errorf("simulated API error for model %s", model)
        callLog.Success = false
        callLog.ErrorMessage = err.Error()
        response = ""
    } else {
        response = "This is a simulated LLM response."
        callLog.Success = true
    }

    // Log the event
    w.logger.Info("LLM API Call",
        zap.Time("timestamp", callLog.Timestamp),
        zap.String("modelName", callLog.ModelName),
        zap.Int("inputTokens", callLog.InputTokens),
        zap.Int("outputTokens", callLog.OutputTokens),
        zap.String("apiKeyId", callLog.APIKeyID),
        zap.String("featureName", callLog.FeatureName),
        zap.Float64("estimatedCostUSD", callLog.EstimatedCostUSD),
        zap.Bool("success", callLog.Success),
        zap.String("errorMessage", callLog.ErrorMessage),
    )

    return response, err
}

// Example usage
func main() {
    logger, _ := zap.NewProduction()
    defer logger.Sync()

    client := NewLLMClientWrapper(logger)

    // Simulate calls for different features
    client.CallLLM(context.Background(), "gpt-4-turbo", "Write a blog post about LLM monitoring.", "blog_post_generator")
    client.CallLLM(context.Background(), "claude-3-opus-20240229", "Generate 5 catchy headlines.", "headline_rewriter")
    client.CallLLM(context.Background(), "gpt-4-turbo", "Summarize this article.", "summary_service")
}

I pushed these structured logs to Google Cloud Logging. The beauty of structured logging is that each field (modelName, featureName, estimatedCostUSD, etc.) becomes queryable and indexable. This foundation was critical.

Phase 2: Extracting Metrics with Log-Based Metrics

Once the logs were flowing into Cloud Logging, the next step was to turn them into time-series metrics. Google Cloud Monitoring offers "log-based metrics," which allow you to define custom metrics based on patterns in your logs. This was a game-changer.

I created several custom metrics:

llm/total_estimated_cost: A distribution metric tracking estimatedCostUSD for every successful call.
llm/total_input_tokens: A counter metric for inputTokens.
llm/total_output_tokens: A counter metric for outputTokens.

Crucially, I added labels to these metrics based on our log fields: modelName and featureName. This allowed me to break down the metrics by these dimensions. For example, a log-based metric for total estimated cost might look something like this (simplified configuration):


# metrics.yaml for Google Cloud Logging
# Metric for total estimated cost
- name: "llm_total_estimated_cost"
  description: "Total estimated cost of LLM API calls"
  valueExtractor: "jsonPayload.estimatedCostUSD"
  metricKind: "DELTA" # or GAUGE, depending on how you want to aggregate
  valueType: "DOUBLE"
  labelExtractors:
    model_name: "jsonPayload.modelName"
    feature_name: "jsonPayload.featureName"

# Metric for total input tokens
- name: "llm_total_input_tokens"
  description: "Total input tokens for LLM API calls"
  valueExtractor: "jsonPayload.inputTokens"
  metricKind: "DELTA"
  valueType: "INT64"
  labelExtractors:
    model_name: "jsonPayload.modelName"
    feature_name: "jsonPayload.featureName"

# ... similar for output tokens

This configuration effectively transformed my raw logs into structured time-series data, ready for analysis. The flexibility of log-based metrics meant I didn't need to introduce an extra metrics agent or a complex ETL pipeline just for this.

Phase 3: Visualization and Alerting

With metrics flowing, the next step was to make them visible and actionable. I built a dedicated dashboard in Google Cloud Monitoring. Key charts included:

Daily Estimated Cost: A bar chart showing total daily spend, broken down by featureName. This immediately highlighted which part of our application was the primary cost driver.
Hourly Token Usage by Model: Line charts showing input and output tokens per hour, segmented by modelName. This helped identify if a particular model was suddenly seeing a surge in use or if its output verbosity had increased.
Cost per Feature Trend: A time-series graph showing the rolling 7-day average cost for each major feature.

Here's a conceptual view of a dashboard widget:


// Example Dashboard Widget Configuration (Conceptual JSON for GCP Monitoring)
{
  "displayName": "Daily Estimated LLM API Cost by Feature",
  "widgetType": "TIMESERIES",
  "dataSources": [
    {
      "name": "projects/my-gcp-project/metricDescriptors/logging.googleapis.com/user/llm_total_estimated_cost",
      "groupBy": [
        "metric.labels.feature_name"
      ],
      "aligner": "ALIGN_SUM",
      "reducer": "REDUCE_SUM",
      "perSeriesAligner": "ALIGN_DELTA",
      "aggregation": "ALIGN_RATE",
      "interval": "86400s" // Daily aggregation
    }
  ],
  "chartOptions": {
    "mode": "STACKED_BAR"
  }
}

The ability to drill down by featureName and modelName allowed me to quickly pinpoint the source of the original cost spike: a new experimental feature, still in alpha, had inadvertently been left running with an aggressive retry policy, causing a feedback loop that generated thousands of unnecessary calls to a high-cost model. Without this granular visibility, it would have been a needle in a haystack.

Beyond dashboards, I set up alerting policies. I configured alerts for:

Daily Cost Threshold: If the total estimated daily cost for *any* feature exceeded a predefined threshold (e.g., $50/day for a beta feature, $200/day for a core feature).
Hourly Token Anomaly: If the hourly input or output token count for a specific model deviated significantly (e.g., 2 standard deviations) from its historical average.

These alerts are now piped to our Slack channel, giving us near real-time notifications of potential issues. This proactive approach is a stark contrast to my previous "wait for the bill" strategy.

Phase 4: Refining Cost Attribution and Control

With basic monitoring in place, I started thinking about more advanced cost attribution. We sometimes run experiments with different prompt variations or model parameters. To understand the cost impact of these, I extended my logging to include an experiment_id or prompt_template_version. This allowed us to correlate specific prompt engineering choices with actual token usage and estimated costs, directly informing our optimization efforts. This goes hand-in-hand with our efforts in How I Tuned Prompt Engineering for Maximum AI Cost Savings.

Understanding cost attribution also directly feeds into our adaptive rate limiter for AI APIs. By knowing which features consume the most, we can set more intelligent rate limits, prioritizing critical services and throttling less essential ones during peak usage or when budgets are tight.

For context, here's a snapshot of what our daily estimated LLM spend might look like on a good day, broken down by feature:

Blog Post Generator: $35.20 (gpt-4-turbo, high output)
Headline Rewriter: $8.15 (gpt-3.5-turbo, low output, high volume)
Summary Service: $12.50 (claude-3-sonnet, medium volume)
Internal Tooling: $4.80 (various models, ad-hoc usage)
Total Daily Spend: $60.65

This kind of breakdown is invaluable. It tells us exactly where our money is going and helps us make informed decisions about feature development and optimization. You can find up-to-date pricing information directly from providers, for example, OpenAI's pricing page: https://openai.com/pricing.

What I Learned / The Challenge

The biggest lesson I took from this experience is that proactive, granular monitoring is non-negotiable for LLM-powered applications. The costs can be highly variable and opaque, making it easy to incur significant unexpected expenses. Relying solely on high-level vendor dashboards is a recipe for bill shock.

Another challenge is the dynamic nature of LLM pricing. Providers frequently adjust their rates, introduce new models with different cost structures, or change tokenization methods. My simple TokenPrices map needs regular updates, and in a more mature system, this might be fetched from a configuration service or even an API if providers offered one for pricing data. This highlights the ongoing maintenance required for a custom monitoring solution.

Building this custom solution also revealed the overhead involved. While log-based metrics are powerful, they require careful planning of log structure and metric definitions. It's a trade-off between the flexibility and control of a custom system versus the simplicity (but lack of granularity) of vendor-provided tools. For now, the control is worth the effort.

Looking Ahead

While my current system has significantly improved our cost visibility and control, I'm already thinking about the next iterations. I want to integrate this cost data more deeply into our development lifecycle. Imagine a CI/CD pipeline that could estimate the cost impact of a new prompt template or a change in model before it even hits production. This would require more sophisticated cost prediction models, perhaps leveraging historical data and more detailed tokenization estimates.

I'm also keeping a close eye on how LLM providers evolve their own cost management tools. As the industry matures, I anticipate more native, granular cost attribution and alerting features. Until then, building our own robust monitoring system remains a critical aspect of responsible AI development and cost management for our project.

The days of coffee-bitter billing surprises are, hopefully, behind us.

Current Date: 2026-03-02

This post reflects my personal experiences and solutions at the time of writing. Best practices and available technologies may evolve.

Citations:

OpenAI Pricing: https://openai.com/pricing

Search This Blog

TechFrontier | AI Engineering, Go & Cloud Cost Optimization