Lowering LLM API Costs: A Deep Dive into Function Calling and Tool Use Optimization

I distinctly remember the email notification. It wasn't the usual "your daily spend is trending" alert; it was a stark, red-flagged "your projected monthly bill has exceeded 200% of your average!" My heart sank a little. As the lead developer, I knew exactly where to look: our shiny new LLM integrations. While they were delivering incredible value, the cost curve was starting to look more like a rocket launch than a gentle ascent.

We had been rapidly integrating large language models into our content generation pipeline, and the initial results were fantastic. However, as the complexity of our tasks grew – think multi-stage content creation, detailed data extraction for fact-checking, and dynamic content adaptation – so did our LLM API bill. My initial approach, while functional, was proving to be a token-guzzling beast.

The Problem: Unoptimized LLM Usage and Soaring Token Counts

Our core workflow involved several steps for generating a blog post, each often requiring an LLM call:

Topic Ideation: Generate 5-10 potential blog titles and outlines based on a high-level theme.
Outline Expansion: Take a chosen outline and expand it into detailed sections and sub-points.
Content Generation: Write paragraphs for each section.
Data Extraction/Fact-Checking: Parse external data (e.g., search results, internal knowledge base snippets) and ask the LLM to extract specific entities or verify facts.
Refinement/SEO Optimization: Rewrite sections for clarity, tone, and SEO keywords.

The biggest culprits for cost escalation were steps 3 and 4. For content generation, we often fed the LLM a large context window, including previous sections, style guides, and user preferences. But step 4, data extraction, was a particular pain point. I was essentially prompting the LLM with huge chunks of unstructured text (sometimes entire web pages or long JSON payloads) and then asking it to "extract X, Y, and Z into a JSON format." This meant:

Massive Input Tokens: Every single character of that unstructured text was sent to the LLM, even if only a few pieces of information were needed.
Inefficient Output Tokens: The LLM would then generate a JSON string, which, while structured, still consumed tokens for keys, values, and formatting.
Fragile Parsing: Despite asking for JSON, LLMs, especially older models, could sometimes drift, leading to malformed output that required robust (and sometimes brittle) post-processing on our end.
Multiple Calls for Complex Tasks: If an extraction was too complex, I'd break it into multiple LLM calls, increasing both latency and cost.

I knew there had to be a more efficient way. My initial thought was to simply switch to cheaper models, but that felt like a band-aid. The fundamental inefficiency in how we were interacting with the LLM remained.

The "Aha!" Moment: LLMs as Orchestrators, Not Just Generators

The turning point came when I started diving deeper into the concept of "function calling" or "tool use" offered by modern LLMs. My initial understanding was that this was just for letting the LLM *call external APIs*. While true, I realized its much more profound implication: the LLM could orchestrate actions and structure data, without necessarily generating the data itself.

Instead of asking the LLM to *generate* a JSON object containing extracted data, I could ask it to *call a function* that *would receive* that data as arguments. This subtle shift had massive implications for token usage and reliability.

Before: The Token-Heavy Extraction Prompt

Here's a simplified example of how I might have extracted information from a news article snippet before:


// Python-like pseudocode
article_snippet = """
    In a landmark announcement today, Acme Corp (NASDAQ: ACME) revealed its Q3 earnings,
    exceeding analyst expectations with a revenue of $1.2 billion, a 15% increase year-over-year.
    The company's CEO, Jane Doe, stated that "innovation in AI and sustainable energy solutions"
    were key drivers. Shares surged by 8% in after-hours trading.
    The report also highlighted a new partnership with Global Tech Solutions,
    effective January 1, 2027, to develop next-generation quantum computing.
"""

prompt = f"""
Given the following article snippet, extract the company name, its stock ticker,
Q3 revenue, year-over-year revenue increase percentage, CEO name, and any new partnerships
with their effective date. Return the information as a JSON object.

Article Snippet:
{article_snippet}

JSON Output:
"""

// Call LLM API with this prompt
// Expects: {"company_name": "Acme Corp", "stock_ticker": "ACME", ...}

In this scenario, the `article_snippet` (potentially very long) is part of the input prompt, consuming tokens. The LLM then generates the entire JSON string, consuming output tokens. If the article was 5000 tokens long, and the JSON output was 200 tokens, that's 5200 tokens per call just for extraction!

After: The Token-Saving Power of Function Calling

With function calling, I defined a "tool" that represented the action of "recording extracted article data." The LLM's job then shifted from *generating* the data to *deciding* to call this tool and providing the arguments to it.


// Python-like pseudocode demonstrating tool definition
tool_definitions = [
    {
        "name": "record_article_data",
        "description": "Records key information extracted from a news article.",
        "parameters": {
            "type": "object",
            "properties": {
                "company_name": {
                    "type": "string",
                    "description": "The name of the company mentioned."
                },
                "stock_ticker": {
                    "type": "string",
                    "description": "The stock ticker of the company, if available."
                },
                "q3_revenue": {
                    "type": "number",
                    "description": "The reported Q3 revenue in USD."
                },
                "yoy_revenue_increase_percent": {
                    "type": "number",
                    "description": "The year-over-year revenue increase percentage."
                },
                "ceo_name": {
                    "type": "string",
                    "description": "The name of the CEO mentioned."
                },
                "new_partnership": {
                    "type": "string",
                    "description": "Description of any new partnerships announced."
                },
                "partnership_effective_date": {
                    "type": "string",
                    "description": "The effective date of the new partnership, in YYYY-MM-DD format."
                }
            },
            "required": ["company_name", "q3_revenue"] // Example required fields
        }
    }
]

// The prompt now focuses on the task, with the article snippet in the content
prompt_with_tools = f"""
Please analyze the following news article snippet and extract the relevant information.
If you find details about a company's Q3 earnings, CEO, or new partnerships, please
use the 'record_article_data' tool to capture them.

Article Snippet:
{article_snippet}
"""

// Call LLM API with prompt_with_tools and tool_definitions
// The LLM's response will contain a 'tool_calls' field if it decides to use the tool.

When the LLM receives this prompt and the `tool_definitions`, it doesn't *generate* the JSON output. Instead, it processes the `article_snippet` and, recognizing the instruction to "use the 'record_article_data' tool," it generates a `tool_call` object. This object contains the function name and the arguments, which are precisely the extracted pieces of information.

The key difference? The LLM only needs to generate the *arguments* to the function, not the entire JSON structure, including keys. This dramatically reduces output tokens. More importantly, the output is *guaranteed* to conform to the `tool_definitions` schema, eliminating fragile parsing on our end. We just receive structured data directly!

Here's a conceptual LLM response for the tool call:


// Example LLM response payload (simplified)
{
  "candidates": [
    {
      "content": {
        "parts": [
          {
            "functionCall": {
              "name": "record_article_data",
              "args": {
                "company_name": "Acme Corp",
                "stock_ticker": "ACME",
                "q3_revenue": 1200000000,
                "yoy_revenue_increase_percent": 15,
                "ceo_name": "Jane Doe",
                "new_partnership": "Global Tech Solutions for next-generation quantum computing",
                "partnership_effective_date": "2027-01-01"
              }
            }
          }
        ]
      }
    }
  ]
}

Our application then simply executes the `record_article_data` function with these `args`. The data is clean, structured, and ready to use.

Tool Use Optimization Strategies

Beyond the fundamental shift to function calling, I discovered several ways to optimize tool use further:

1. Precise Schema Definitions

The clearer and more specific your tool's `parameters` schema is, the better the LLM will be at extracting exactly what you need. I found that adding detailed `description` fields for each parameter was crucial. Specifying `type` (e.g., `string`, `number`, `boolean`, `array`) and `enum` values where appropriate significantly improved accuracy and reduced hallucination of argument values.

For example, instead of just `{"type": "string", "description": "Date"}`, I'd use `{"type": "string", "description": "The effective date, in YYYY-MM-DD format."}`. This guidance helps the LLM format the data correctly.

2. Chaining Tool Calls for Complex Workflows

Function calling isn't just for single extractions. It's incredibly powerful for orchestrating multi-step processes. For instance, our topic ideation workflow (step 1 above) could be refined:


// Tool to generate a single blog post idea
tool_definitions = [
    {
        "name": "suggest_blog_post_idea",
        "description": "Suggests a single blog post idea with a title and a brief outline.",
        "parameters": {
            "type": "object",
            "properties": {
                "title": {"type": "string", "description": "A compelling title for the blog post."},
                "outline_points": {
                    "type": "array",
                    "items": {"type": "string"},
                    "description": "Key bullet points for the blog post outline."
                }
            },
            "required": ["title", "outline_points"]
        }
    }
]

prompt = "Generate 3 distinct blog post ideas about 'sustainable AI development'."

// Initial LLM call might generate 3 separate tool_calls for 'suggest_blog_post_idea'
// Or, I could have a wrapper tool "generate_multiple_ideas" that takes an array of ideas.

This allows the LLM to provide a structured list of ideas, which can then be fed into subsequent stages of our content pipeline. The LLM decides *how many* ideas to generate based on the prompt, and each idea is perfectly structured.

3. Strategic Tool Selection

Not every task benefits from function calling. Simple summarization or creative writing might still be best handled by direct text generation. The trick is to identify where structured output or external actions are beneficial. If I need to extract structured data, perform a lookup, or trigger a specific workflow based on LLM understanding, function calling is the way to go.

4. Error Handling and Fallbacks

Even with precise schemas, LLMs aren't perfect. They might sometimes hallucinate arguments or decide not to call a tool when they should. My error handling now includes:

Schema Validation: Always validate the arguments received from the LLM against the expected schema before executing the function.
Retry Mechanisms: If validation fails, I implement a retry with a slightly modified prompt, perhaps explicitly telling the LLM it failed and why.
Human-in-the-Loop: For critical extractions, a human review step can catch edge cases.

This attention to detail, reminiscent of our work on diagnosing and resolving Go Cloud Run memory leaks for AI workloads, ensures system robustness even when dealing with the non-deterministic nature of LLMs.

The Impact: Reduced Costs and Improved Reliability

The results were tangible and impressive. By shifting from LLM-generated JSON to LLM-orchestrated function calls:

Token Usage Plummeted: For complex extraction tasks, I saw a 30-50% reduction in total tokens per request. This was primarily due to the LLM only generating the concise arguments rather than the verbose JSON structure.
API Bill Reduction: Within the first month, our LLM API bill for these specific workflows dropped by approximately 25%. This wasn't just a marginal saving; it was a significant cut that brought our spending back into a sustainable range.
Increased Reliability: The structured output from function calls virtually eliminated parsing errors on our end. My code became simpler, as I no longer needed complex regex or robust JSON parsers to handle malformed LLM output.
Faster Iteration: Prompt engineering became less about "how do I get the LLM to output perfect JSON" and more about "how do I define the perfect tool for the LLM to use." This allowed me to iterate on new features much faster.
Simplified Codebase: Our application code for handling LLM responses became cleaner and more predictable. Instead of parsing arbitrary text, we were dealing with well-defined function calls and arguments.

This optimization wasn't just about cost; it was about building a more robust and scalable system. It also highlighted the importance of understanding the underlying mechanics of how LLMs process information, a lesson I also learned when optimizing Cloud Run cold starts from seconds to milliseconds – sometimes, the biggest gains come from fundamental architectural shifts, not just tweaking parameters.

For a deeper dive into the technical specifications of how these models handle function calling, I highly recommend checking out the official Google Cloud Vertex AI documentation on function calling or similar documentation from other providers like OpenAI.

What I Learned / The Challenge

The biggest challenge wasn't implementing function calling itself – most modern LLM APIs make it quite straightforward. The real challenge was shifting my mental model. I had to stop thinking of the LLM purely as a text generator and start seeing it as an intelligent agent capable of making decisions about *when* and *how* to interact with my application's capabilities. It's about leveraging the LLM's reasoning power for orchestration, not just content creation.

I learned that the most effective way to reduce LLM API costs isn't always about finding cheaper models or aggressively truncating prompts. Often, it's about fundamentally changing the interaction paradigm to leverage the LLM's strengths more efficiently. Function calling allows the LLM to operate at a higher level of abstraction, leading to more precise, reliable, and ultimately, more cost-effective interactions.

Search This Blog

TechFrontier | AI Engineering, Go & Cloud Cost Optimization