LLM API Cost Optimization: Reducing Context Window Expenses

I remember it vividly. It was a Tuesday morning, and I was doing my routine check of our cloud billing dashboard. My coffee almost went cold when I saw the graph: a sharp, alarming spike in our LLM API costs. It wasn't just a blip; it was a sustained surge, pushing us well past our projected monthly budget within the first week. My heart sank. What had gone wrong?

My first thought was a runaway process, perhaps an infinite loop of API calls. I immediately checked our request logs and custom metrics. While the request count had indeed increased slightly, it didn't fully explain the exponential jump in cost. The culprit, I soon discovered, wasn't the *number* of calls, but the *size* of the calls – specifically, the monstrous context windows we were sending to the LLM. We were paying a premium for every single token, and suddenly, we were sending vastly more than anticipated.

This wasn't just a financial hit; it was a technical challenge that threatened the sustainability of our project. My mission became clear: drastically reduce our LLM API context window expenses without compromising the quality or functionality of our core features. This devlog entry details my journey, the strategies I employed, the code changes I made, and the tangible results I achieved.

The Hidden Cost of Context: Understanding the Problem

Most LLM providers charge based on token usage for both input (prompt) and output (completion). While output tokens can sometimes be optimized with techniques like streaming (which I've discussed in a previous post, Optimizing LLM API Latency: Async, Streaming, and Pydantic in Production), input tokens, especially within the context window, are often the silent killers of a budget. My monitoring, while robust for overall cost, hadn't initially highlighted the *composition* of those costs effectively enough to catch this early.

I realized we had fallen into a common trap: assuming larger context windows were always better. For certain complex tasks, they are invaluable. However, for many of our use cases, we were sending entire documents, long conversation histories, or comprehensive data extracts when only a small fraction was truly relevant to the immediate query. This was akin to sending a whole library to answer a single question – expensive and inefficient.

Here’s a simplified breakdown of the cost structure I was grappling with:


# Example: Hypothetical LLM API pricing (actual numbers vary by provider and model)
MODEL_PRICING = {
    "gpt-3.5-turbo": {
        "input_token_cost_per_1k": 0.0005,
        "output_token_cost_per_1k": 0.0015
    },
    "gpt-4": {
        "input_token_cost_per_1k": 0.03,
        "output_token_cost_per_1k": 0.06
    }
}

def calculate_cost(model_name, input_tokens, output_tokens):
    input_cost = (input_tokens / 1000) * MODEL_PRICING[model_name]["input_token_cost_per_1k"]
    output_cost = (output_tokens / 1000) * MODEL_PRICING[model_name]["output_token_cost_per_1k"]
    return input_cost + output_cost

# Scenario: Sending a large document (5000 input tokens) vs. a concise prompt (500 input tokens)
# Assuming 200 output tokens for both
large_context_cost = calculate_cost("gpt-4", 5000, 200)
small_context_cost = calculate_cost("gpt-4", 500, 200)

print(f"Cost with large context (5000 input tokens): ${large_context_cost:.4f}") # Output: $0.1620
print(f"Cost with small context (500 input tokens): ${small_context_cost:.4f}")   # Output: $0.0210
# This is a 7.7x difference for just one call!

The numbers spoke for themselves. A small difference in input token count could lead to a massive difference in recurring costs, especially when multiplied by thousands of API calls per day. This reinforced the need for a multi-pronged approach to context window optimization.

My Multi-Pronged Strategy for Cost Reduction

1. Aggressive Prompt Engineering & Condensation

My first line of defense was to scrutinize every prompt we sent. We had a tendency to be overly verbose, including instructions or examples that weren't strictly necessary for every call. My goal was to make prompts as lean as possible without sacrificing output quality.

A. Few-Shot to Zero-Shot (Where Possible)

While few-shot prompting often yields better results for complex tasks, it comes at a significant token cost due to the examples provided. I identified areas where the model's understanding of the task was robust enough to perform well with zero-shot prompting, or at least with significantly fewer examples. This required careful A/B testing and evaluation of output quality.


# Before: Few-shot example
FEW_SHOT_PROMPT = """
Extract the key entities (person, organization, location) from the following text.

Example 1:
Text: "Elon Musk founded SpaceX in California."
Entities: Person: Elon Musk, Organization: SpaceX, Location: California

Example 2:
Text: "Sundar Pichai is the CEO of Google, headquartered in Mountain View."
Entities: Person: Sundar Pichai, Organization: Google, Location: Mountain View

Text: "{input_text}"
Entities:
"""

# After: Zero-shot (or minimal instruction)
ZERO_SHOT_PROMPT = """
Identify and list all distinct people, organizations, and locations mentioned in the following text. Respond in a structured JSON format.

Text: "{input_text}"
"""

# Token cost reduction is immediate here, especially if many examples are used.

B. Instruction Tuning & Conciseness

I focused on making instructions crystal clear and concise. Often, I found that we were repeating instructions or using overly flowery language. Direct, imperative statements worked just as well, if not better, and saved tokens.

I also leveraged the model's inherent capabilities. Instead of explicitly instructing the model to "summarize this 500-word article into 3 sentences, ensuring all key points are covered," I found that "Summarize the following article concisely:" often produced similar quality with fewer tokens in the instruction itself, letting the model infer the 'concise' part.

C. Contextual Pruning

For conversational agents or long-document processing, I implemented a 'contextual pruning' strategy. Instead of sending the entire conversation history, I developed a simple heuristic:

Always keep the last N turns of the conversation.
Summarize older turns if they exceed a certain token limit, retaining key information.
Prioritize specific entities or topics identified in the current turn, ensuring their presence in the context.

This wasn't full RAG (which I'll discuss next), but a simpler form of managing the immediate conversational buffer. It's a delicate balance, as aggressive pruning can lead to loss of coherence, so careful testing was paramount.

2. Implementing Semantic Caching

One of the biggest culprits of redundant token usage was repeated queries. Users often asked similar questions, or our internal processes would re-evaluate the same content multiple times. My solution was to implement a semantic cache.

Unlike a traditional key-value cache that relies on exact string matches, a semantic cache stores the embeddings of prompts and their corresponding LLM responses. When a new prompt comes in, we generate its embedding and compare it to the embeddings of cached prompts. If a sufficiently similar prompt (above a defined similarity threshold) is found, we return the cached response instead of hitting the LLM API.

Here's a simplified conceptual flow:


import hashlib
from typing import Dict, Any

# Assume these are available from your vector DB client and embedding model
# from vector_db_client import VectorDBClient
# from embedding_model import EmbeddingModel

# For demonstration, let's mock these
class MockVectorDBClient:
    def __init__(self):
        self.store = {} # {embedding_hash: {"prompt_embedding": ..., "response": ...}}

    def search(self, query_embedding, threshold=0.9):
        # In a real scenario, this would be an approximate nearest neighbor search
        for key, value in self.store.items():
            # Mocking similarity check
            if hashlib.sha256(str(query_embedding).encode()).hexdigest() == key:
                return value["response"] # Exact match for simplicity
        return None

    def add(self, prompt_embedding, response):
        key = hashlib.sha256(str(prompt_embedding).encode()).hexdigest()
        self.store[key] = {"prompt_embedding": prompt_embedding, "response": response}

class MockEmbeddingModel:
    def encode(self, text):
        return [ord(char) for char in text] # Simple mock embedding

vector_db = MockVectorDBClient()
embedding_model = MockEmbeddingModel()

def get_llm_response_with_cache(prompt: str, llm_api_call_func: callable) -> str:
    prompt_embedding = embedding_model.encode(prompt)

    # 1. Check cache
    cached_response = vector_db.search(prompt_embedding, threshold=0.9)
    if cached_response:
        print("Cache hit!")
        return cached_response

    # 2. If not in cache, call LLM
    print("Cache miss. Calling LLM...")
    llm_response = llm_api_call_func(prompt) # This would be your actual LLM API call

    # 3. Store in cache
    vector_db.add(prompt_embedding, llm_response)
    return llm_response

# Example usage:
def mock_llm_api(prompt):
    print(f"  --> LLM processing: '{prompt[:30]}...'")
    return f"LLM response for: {prompt}"

print(get_llm_response_with_cache("What is the capital of France?", mock_llm_api))
print(get_llm_response_with_cache("What is the capital of France?", mock_llm_api)) # Should be a cache hit
print(get_llm_response_with_cache("Tell me the capital of France.", mock_llm_api)) # Might be a cache hit with real embeddings

The implementation involved:

Choosing an embedding model (e.g., OpenAI's text-embedding-ada-002, or an open-source alternative for cost savings).
Integrating with a vector database (e.g., Pinecone, Weaviate, or even a local FAISS index for smaller scales).
Careful tuning of the similarity threshold. Too high, and you miss potential hits; too low, and you return irrelevant answers.
Developing a cache invalidation strategy, especially for dynamic content.

This strategy significantly reduced redundant LLM calls, especially for frequently asked questions or common summarization tasks. It's a powerful technique, but it adds architectural complexity. For more on monitoring costs and preventing bill surprises, my colleague recently wrote about it: Monitoring LLM API Costs: Preventing Bill Surprises with Custom Metrics.

3. Retrieval-Augmented Generation (RAG) for Dynamic Context

For tasks requiring knowledge beyond the LLM's training data or for processing large, domain-specific documents, RAG became indispensable. Instead of trying to cram entire manuals or datasets into the context window, I adopted a retrieval-first approach.

The core idea of RAG is to:

Break down large documents into smaller, semantically meaningful chunks.
Store these chunks (along with their embeddings) in a vector database.
When a user query comes in, retrieve only the most relevant chunks from the vector database using semantic search.
Feed *only* these retrieved chunks (along with the user query) to the LLM as context.

This dramatically reduces the input token count because the LLM only sees the specific information it needs to answer the question, rather than the entire source document. For example, if a user asks about "the warranty period for product X," the system retrieves only the relevant section from the product manual, not the entire manual.


# Simplified RAG flow (conceptual)
# Assume 'vector_db' and 'embedding_model' from the caching example are available
# Assume 'document_store' is a collection of pre-chunked documents

def retrieve_relevant_chunks(query: str, num_chunks: int = 3) -> list:
    query_embedding = embedding_model.encode(query)
    # In a real system, this would search the vector DB for top-k similar document chunks
    # For demonstration, let's just pick some mock chunks
    mock_chunks = [
        "Product X warranty covers manufacturing defects for 2 years.",
        "To claim warranty for Product X, visit our support portal.",
        "Product Y features include a 12MP camera and 5G connectivity." # Irrelevant, should be filtered by real search
    ]
    # A real vector DB search would return top N most similar chunks
    return mock_chunks[:num_chunks] # Return top N relevant chunks

def generate_rag_prompt(query: str, retrieved_context: list) -> str:
    context_str = "\n".join([f"- {chunk}" for chunk in retrieved_context])
    return f"""
    Based on the following context, answer the user's question.
    Context:
    {context_str}

    Question: {query}
    Answer:
    """

def rag_pipeline(user_query: str, llm_api_call_func: callable) -> str:
    relevant_context = retrieve_relevant_chunks(user_query)
    prompt = generate_rag_prompt(user_query, relevant_context)
    print(f"Generated RAG Prompt (first 200 chars):\n{prompt[:200]}...")
    return llm_api_call_func(prompt)

# Example usage
print("\n--- RAG Pipeline ---")
print(rag_pipeline("What is the warranty for Product X?", mock_llm_api))

The chunking strategy is crucial here. Too small, and context is lost; too large, and you defeat the purpose of RAG. I found that a chunk size of 250-500 tokens with a small overlap (10-20%) worked well for most of our text-based documents. External resources like the Pinecone RAG guide provided excellent insights into best practices for this architecture.

4. Dynamic Model Selection and Context Window Sizing

Not all tasks require the most powerful (and expensive) LLMs with massive context windows. I implemented a dynamic routing mechanism that selected the appropriate model based on the complexity of the task and the estimated input token length.

Smaller, Cheaper Models for Simple Tasks: For tasks like basic sentiment analysis, simple summarization of short texts, or rephrasing, I routed requests to models like gpt-3.5-turbo or even specialized, fine-tuned smaller models. These models are significantly cheaper per token.
Dynamic Context Window Sizing: For tasks that *do* require larger context, I implemented logic to estimate the token count of the input. If the estimated tokens were below a certain threshold (e.g., 2000 tokens), I'd use a model with a standard context window. If it was higher, and the task was critical, I'd route to a model with a larger context window (e.g., gpt-4-turbo), but only after applying all other optimization techniques (pruning, RAG, etc.).


# Simple example of dynamic model selection
import tiktoken # For accurate token counting

encoding = tiktoken.encoding_for_model("gpt-4")

def estimate_tokens(text: str) -> int:
    return len(encoding.encode(text))

def get_optimized_llm_config(prompt: str, task_complexity: str) -> Dict[str, Any]:
    estimated_input_tokens = estimate_tokens(prompt)

    if task_complexity == "simple" and estimated_input_tokens < 1000:
        return {"model": "gpt-3.5-turbo", "max_tokens": 500}
    elif task_complexity == "moderate" and estimated_input_tokens < 4000:
        return {"model": "gpt-3.5-turbo-16k", "max_tokens": 1000}
    elif task_complexity == "complex" and estimated_input_tokens < 8000:
        return {"model": "gpt-4-turbo", "max_tokens": 2000}
    else:
        # Fallback for very long or very complex prompts, potentially with warning/alert
        return {"model": "gpt-4-turbo", "max_tokens": 4000} # Max context for this example

# This logic would be integrated into our LLM abstraction layer
# where we make the actual API calls.

This approach allowed me to use the right tool for the job, avoiding the "one model fits all" mentality that was leading to overspending. The key was to accurately categorize tasks and estimate token counts.

The Results: A Significant Drop in LLM Expenses

After implementing these strategies, the impact on our billing was almost immediate and incredibly satisfying. Within two weeks, our LLM API costs dropped by over 60% from their peak, stabilizing at a level significantly below our pre-spike baseline. The average input token count per API call decreased by roughly 70% across the board for our most frequently used endpoints.

Here's a simplified view of the cost metrics I tracked (hypothetical numbers for illustration):


{
  "metric_name": "LLM_API_INPUT_TOKENS_PER_DAY",
  "data": [
    {"date": "2026-02-15", "value": 15000000},
    {"date": "2026-02-16", "value": 18000000},
    {"date": "2026-02-17", "value": 25000000},
    {"date": "2026-02-18", "value": 32000000}, # Cost spike detected
    {"date": "2026-02-19", "value": 30000000},
    {"date": "2026-02-20", "value": 28000000}, # Initial prompt engineering
    {"date": "2026-02-21", "value": 20000000}, # Semantic caching rollout
    {"date": "2026-02-22", "value": 12000000}, # RAG implementation for key features
    {"date": "2026-02-23", "value": 10000000},
    {"date": "2026-02-24", "value": 9500000},
    {"date": "2026-02-25", "value": 9000000}  # Stabilized at ~65% reduction from peak
  ]
}

This wasn't just about saving money; it was about building a more resilient and sustainable architecture for our LLM-powered features. It proved that thoughtful design and continuous optimization are critical when working with usage-based APIs.

What I Learned / The Challenge

The biggest lesson for me was the absolute necessity of granular cost monitoring. While I had overall cost monitoring in place, I needed to dive deeper into *what* was driving those costs. Tracking input and output tokens per API call, per feature, and per user became paramount. This shift in focus allowed me to pinpoint the exact areas of waste.

The challenge wasn't just technical; it was also about trade-offs. Aggressive optimization can sometimes impact quality or introduce complexity. For instance, too much prompt condensation might lead to a less nuanced response, or an overly strict semantic cache could miss valid, slightly rephrased queries. Balancing cost savings with performance and output quality required continuous iteration, A/B testing, and close collaboration with our product team to define acceptable thresholds.

Another learning curve was the operational overhead of managing these systems. Semantic caching and RAG introduce new components (vector databases, embedding models) that need to be deployed, monitored, and maintained. This added complexity, but the cost savings and improved performance justified the effort.

LLM API Cost Optimization: Reducing Context Window Expenses

My first thought was a runaway process, perhaps an infinite loop of API calls. I immediately checked our request logs and custom metrics. While the request count had indeed increased slightly, it didn't fully explain the exponential jump in cost. The culprit, I soon discovered, wasn't the number of calls, but the size of the calls – specifically, the monstrous context windows we were sending to the LLM. We were paying a premium for every single token, and suddenly, we were sending vastly more than anticipated.

The Hidden Cost of Context: Understanding the Problem

Most LLM providers charge based on token usage for both input (prompt) and output (completion). While output tokens can sometimes be optimized with techniques like streaming (which I've discussed in a previous post, Optimizing LLM API Latency: Async, Streaming, and Pydantic in Production), input tokens, especially within the context window, are often the silent killers of a budget. My monitoring, while robust for overall cost, hadn't initially highlighted the composition of those costs effectively enough to catch this early.

Here’s a simplified breakdown of the cost structure I was grappling with:


# Example: Hypothetical LLM API pricing (actual numbers vary by provider and model)
MODEL_PRICING = {
    "gpt-3.5-turbo": {
        "input_token_cost_per_1k": 0.0005,
        "output_token_cost_per_1k": 0.0015
    },
    "gpt-4": {
        "input_token_cost_per_1k": 0.03,
        "output_token_cost_per_1k": 0.06
    }
}

def calculate_cost(model_name, input_tokens, output_tokens):
    input_cost = (input_tokens / 1000) * MODEL_PRICING[model_name]["input_token_cost_per_1k"]
    output_cost = (output_tokens / 1000) * MODEL_PRICING[model_name]["output_token_cost_per_1k"]
    return input_cost + output_cost

# Scenario: Sending a large document (5000 input tokens) vs. a concise prompt (500 input tokens)
# Assuming 200 output tokens for both
large_context_cost = calculate_cost("gpt-4", 5000, 200)
small_context_cost = calculate_cost("gpt-4", 500, 200)

print(f"Cost with large context (5000 input tokens): ${large_context_cost:.4f}") 
print(f"Cost with small context (500 input tokens): ${small_context_cost:.4f}")   
# This is a 7.7x difference for just one call!

My Multi-Pronged Strategy for Cost Reduction

1. Aggressive Prompt Engineering & Condensation

A. Few-Shot to Zero-Shot (Where Possible)


# Before: Few-shot example
FEW_SHOT_PROMPT = """
Extract the key entities (person, organization, location) from the following text.

Example 1:
Text: "Elon Musk founded SpaceX in California."
Entities: Person: Elon Musk, Organization: SpaceX, Location: California

Example 2:
Text: "Sundar Pichai is the CEO of Google, headquartered in Mountain View."
Entities: Person: Sundar Pichai, Organization: Google, Location: Mountain View

Text: "{input_text}"
Entities:
"""

# After: Zero-shot (or minimal instruction)
ZERO_SHOT_PROMPT = """
Identify and list all distinct people, organizations, and locations mentioned in the following text. Respond in a structured JSON format.

Text: "{input_text}"
"""

# Token cost reduction is immediate here, especially if many examples are used.

B. Instruction Tuning & Conciseness

C. Contextual Pruning

For conversational agents or long-document processing, I implemented a 'contextual pruning' strategy. Instead of sending the entire conversation history, I developed a simple heuristic:

Always keep the last N turns of the conversation.
Summarize older turns if they exceed a certain token limit, retaining key information.
Prioritize specific entities or topics identified in the current turn, ensuring their presence in the context.

2. Implementing Semantic Caching

Here's a simplified conceptual flow:


import hashlib
from typing import Dict, Any

# Assume these are available from your vector DB client and embedding model
# from vector_db_client import VectorDBClient
# from embedding_model import EmbeddingModel

# For demonstration, let's mock these
class MockVectorDBClient:
    def __init__(self):
        self.store = {} # {embedding_hash: {"prompt_embedding": ..., "response": ...}}

    def search(self, query_embedding, threshold=0.9):
        # In a real scenario, this would be an approximate nearest neighbor search
        for key, value in self.store.items():
            # Mocking similarity check
            if hashlib.sha256(str(query_embedding).encode()).hexdigest() == key:
                return value["response"] # Exact match for simplicity
        return None

    def add(self, prompt_embedding, response):
        key = hashlib.sha256(str(prompt_embedding).encode()).hexdigest()
        self.store[key] = {"prompt_embedding": prompt_embedding, "response": response}

class MockEmbeddingModel:
    def encode(self, text):
        return [ord(char) for char in text] # Simple mock embedding

vector_db = MockVectorDBClient()
embedding_model = MockEmbeddingModel()

def get_llm_response_with_cache(prompt: str, llm_api_call_func: callable) -> str:
    prompt_embedding = embedding_model.encode(prompt)

    # 1. Check cache
    cached_response = vector_db.search(prompt_embedding, threshold=0.9)
    if cached_response:
        print("Cache hit!")
        return cached_response

    # 2. If not in cache, call LLM
    print("Cache miss. Calling LLM...")
    llm_response = llm_api_call_func(prompt) # This would be your actual LLM API call

    # 3. Store in cache
    vector_db.add(prompt_embedding, llm_response)
    return llm_response

# Example usage:
def mock_llm_api(prompt):
    print(f"  --> LLM processing: '{prompt[:30]}...'")
    return f"LLM response for: {prompt}"

print(get_llm_response_with_cache("What is the capital of France?", mock_llm_api))
print(get_llm_response_with_cache("What is the capital of France?", mock_llm_api)) # Should be a cache hit
print(get_llm_response_with_cache("Tell me the capital of France.", mock_llm_api)) # Might be a cache hit with real embeddings

The implementation involved:

Choosing an embedding model (e.g., OpenAI's text-embedding-ada-002, or an open-source alternative for cost savings).
Integrating with a vector database (e.g., Pinecone, Weaviate, or even a local FAISS index for smaller scales).
Careful tuning of the similarity threshold. Too high, and you miss potential hits; too low, and you return irrelevant answers.
Developing a cache invalidation strategy, especially for dynamic content.

3. Retrieval-Augmented Generation (RAG) for Dynamic Context

The core idea of RAG is to:

Break down large documents into smaller, semantically meaningful chunks.
Store these chunks (along with their embeddings) in a vector database.
When a user query comes in, retrieve only the most relevant chunks from the vector database using semantic search.
Feed only these retrieved chunks (along with the user query) to the LLM as context.


# Simplified RAG flow (conceptual)
# Assume 'vector_db' and 'embedding_model' from the caching example are available
# Assume 'document_store' is a collection of pre-chunked documents

def retrieve_relevant_chunks(query: str, num_chunks: int = 3) -> list:
    query_embedding = embedding_model.encode(query)
    # In a real system, this would search the vector DB for top-k similar document chunks
    # For demonstration, let's just pick some mock chunks
    mock_chunks = [
        "Product X warranty covers manufacturing defects for 2 years.",
        "To claim warranty for Product X, visit our support portal.",
        "Product Y features include a 12MP camera and 5G connectivity." # Irrelevant, should be filtered by real search
    ]
    # A real vector DB search would return top N most similar chunks
    return mock_chunks[:num_chunks] # Return top N relevant chunks

def generate_rag_prompt(query: str, retrieved_context: list) -> str:
    context_str = "\n".join([f"- {chunk}" for chunk in retrieved_context])
    return f"""
    Based on the following context, answer the user's question.
    Context:
    {context_str}

    Question: {query}
    Answer:
    """

def rag_pipeline(user_query: str, llm_api_call_func: callable) -> str:
    relevant_context = retrieve_relevant_chunks(user_query)
    prompt = generate_rag_prompt(user_query, relevant_context)
    print(f"Generated RAG Prompt (first 200 chars):\n{prompt[:200]}...")
    return llm_api_call_func(prompt)

# Example usage
print("\n--- RAG Pipeline ---")
print(rag_pipeline("What is the warranty for Product X?", mock_llm_api))

4. Dynamic Model Selection and Context Window Sizing

Smaller, Cheaper Models for Simple Tasks: For tasks like basic sentiment analysis, simple summarization of short texts, or rephrasing, I routed requests to models like gpt-3.5-turbo or even specialized, fine-tuned smaller models. These models are significantly cheaper per token.
Dynamic Context Window Sizing: For tasks that do require larger context, I implemented logic to estimate the token count of the input. If the estimated tokens were below a certain threshold (e.g., 2000 tokens), I'd use a model with a standard context window. If it was higher, and the task was critical, I'd route to a model with a larger context window (e.g., gpt-4-turbo), but only after applying all other optimization techniques (pruning, RAG, etc.).


# Simple example of dynamic model selection
import tiktoken # For accurate token counting

encoding = tiktoken.encoding_for_model("gpt-4")

def estimate_tokens(text: str) -> int:
    return len(encoding.encode(text))

def get_optimized_llm_config(prompt: str, task_complexity: str) -> Dict[str, Any]:
    estimated_input_tokens = estimate_tokens(prompt)

    if task_complexity == "simple" and estimated_input_tokens < 1000:
        return {"model": "gpt-3.5-turbo", "max_tokens": 500}
    elif task_complexity == "moderate" and estimated_input_tokens < 4000:
        return {"model": "gpt-3.5-turbo-16k", "max_tokens": 1000}
    elif task_complexity == "complex" and estimated_input_tokens < 8000:
        return {"model": "gpt-4-turbo", "max_tokens": 2000}
    else:
        # Fallback for very long or very complex prompts, potentially with warning/alert
        return {"model": "gpt-4-turbo", "max_tokens": 4000} # Max context for this example

# This logic would be integrated into our LLM abstraction layer
# where we make the actual API calls.

The Results: A Significant Drop in LLM Expenses

Here's a simplified view of the cost metrics I tracked (hypothetical numbers for illustration):


{
  "metric_name": "LLM_API_INPUT_TOKENS_PER_DAY",
  "data": [
    {"date": "2026-02-15", "value": 15000000},
    {"date": "2026-02-16", "value": 18000000},
    {"date": "2026-02-17", "value": 25000000},
    {"date": "2026-02-18", "value": 32000000}, # Cost spike detected
    {"date": "2026-02-19", "value": 30000000},
    {"date": "2026-02-20", "value": 28000000}, # Initial prompt engineering
    {"date": "2026-02-21", "value": 20000000}, # Semantic caching rollout
    {"date": "2026-02-22", "value": 12000000}, # RAG implementation for key features
    {"date": "2026-02-23", "value": 10000000},
    {"date": "2026-02-24", "value": 9500000},
    {"date": "2026-02-25", "value": 9000000}  # Stabilized at ~65% reduction from peak
  ]
}

What I Learned / The Challenge

The biggest lesson for me was the absolute necessity of granular cost monitoring. While I had overall cost monitoring in place, I needed to dive deeper into what was driving those costs. Tracking input and output tokens per API call, per feature, and per user became paramount. This shift in focus allowed me to pinpoint the exact areas of waste.

Search This Blog

TechFrontier | AI Engineering, Go & Cloud Cost Optimization