How to Reduce Cloud Run Costs by 50% Using Concurrency

-

You can reduce Cloud Run costs by increasing request concurrency to handle multiple tasks per instance and enabling CPU throttling to avoid paying for idle time. These optimizations allow a single container to process more I/O-bound requests simultaneously, significantly lowering the total instance count and your monthly bill.

Last month, I woke up to a Google Cloud billing alert that made my stomach drop. My side project, which usually runs on a few dollars a month, had spiked to a projected $480. This wasn't a viral success story or a DDoS attack; it was the direct result of my own architectural laziness. I had deployed a high-frequency automation service and left the default Google Cloud Run settings untouched. In the world of serverless, "default" is often synonymous with "expensive."

The service in question was the one I detailed in my previous post about building a lightweight Python automation framework with FastAPI and Gemini. It handles hundreds of small, I/O-bound tasks—mostly calling the Gemini API, waiting for a response, and then writing a summary to a database. Because these tasks spend 90% of their time waiting for network I/O, my instances were sitting idle while Google charged me for every millisecond of "active" CPU time. I realized I was paying for a fleet of high-performance engines to sit in a traffic jam.

I spent the last two weeks aggressively profiling my resource utilization and tweaking my deployment manifests. By shifting from a "one-request-per-container" model to a high-concurrency setup and changing how CPU is allocated, I managed to cut my monthly spend by exactly 52% without increasing my P99 latency. This is how I did it, the specific configs I used, and the telemetry data that proved it worked.

Why Default Concurrency Settings Increase Cloud Run Costs

Defaulting to one request per container forces Google Cloud to spin up unnecessary instances, which drastically increases Cloud Run costs for I/O-bound applications. When you deploy to Cloud Run without specifying a concurrency limit, Google Cloud defaults to a specific behavior depending on your environment, but historically, many developers (myself included) treat serverless as a 1:1 mapping: one request, one container. If 100 people hit your API at once, Cloud Run spins up 100 containers. This is the "scale-to-zero" dream, but it’s a nightmare for cost efficiency in Python applications.

Python's FastAPI, when written with async/await, is exceptionally good at handling multiple concurrent requests on a single thread. While one request is waiting for the Gemini API to return a JSON payload, the event loop can easily process ten other incoming requests. By leaving my concurrency at the default (or set to 1), I was forcing Cloud Run to spin up a brand-new instance for every single API call. Each instance has its own memory overhead and its own "cold start" tax.

I started by analyzing my request patterns. My average request takes 2.5 seconds, but only about 150ms of that is actual CPU work (parsing JSON, calculating signatures). The rest is just waiting. I was essentially paying for a 2.35-second coffee break for my CPU every time a request came in.

How to Benchmark the Concurrency Sweet Spot

I didn't want to just crank the concurrency to 80 (the max for many configurations) and hope for the best. High concurrency can lead to memory exhaustion and "noisy neighbor" issues within your own container. I used a simple script to load test my service at different concurrency levels while monitoring memory usage.

# A snippet of the load-testing logic I used to find the ceiling
import asyncio
import httpx
import time

async def hammer_service(concurrency_level):
    url = "https://my-service-xyz.a.run.app/process"
    async with httpx.AsyncClient(timeout=30) as client:
        tasks = [client.post(url, json={"task": "summarize"}) for _ in range(concurrency_level)]
        start = time.perf_counter()
        responses = await asyncio.gather(*tasks)
        end = time.perf_counter()
        print(f"Concurrency {concurrency_level}: {end - start:.2f}s total")

if __name__ == "__main__":
    for c in [1, 10, 20, 50, 80]:
        asyncio.run(hammer_service(c))

What I found was that my P99 latency stayed flat up to 50 concurrent requests. Beyond 50, I started seeing a slight uptick in response times, likely because the Python event loop was getting congested with context switching. I settled on 40 as my "safe" production concurrency limit. This meant that instead of 40 instances running simultaneously, I could handle the same load with just one or two instances.

How to Optimize CPU Allocation to Lower Your Monthly Bill

Switching to the "CPU only allocated during request processing" model ensures you only pay for active compute time, which is the most effective way to cut Cloud Run costs. Cloud Run offers two main CPU allocation models: "CPU is always allocated" and "CPU is only allocated during request processing." This is perhaps the most misunderstood setting in the console.

If you choose "CPU is always allocated," you pay for the instance as long as it exists, regardless of whether it's handling a request. This is great for background tasks or WebSockets, but it's a money pit for standard REST APIs. However, if you choose "CPU is only allocated during request processing," Google throttles your CPU to almost zero when no requests are active. This is significantly cheaper, but there's a catch: if your app needs to do any work *after* sending a response (like logging or background cleanup), it will crawl at a snail's pace.

I initially feared that switching to "CPU only during request processing" would ruin my telemetry. As I discussed in my post on Python Cloud Run distributed tracing with OpenTelemetry, I rely heavily on background exporters to send spans to my collector. I worried the CPU would be throttled before the spans were sent. However, by using the force_flush() method in my FastAPI shutdown handler, I ensured the data was sent while the request was still technically active, allowing me to switch to the cheaper billing model without losing visibility.

According to the official Google Cloud Run documentation, the "CPU allocated during requests" model can reduce costs by up to 60-80% for services with intermittent traffic. For my automation framework, which has peaks and valleys, this was the silver bullet.

What Configuration Changes Are Required in the service.yaml File?

Explicitly defining container concurrency and CPU throttling in your deployment manifest provides predictable scaling and prevents over-provisioning. I stopped using the Google Cloud Console UI for these changes because it’s too easy to miss a checkbox. Instead, I updated my service.yaml to explicitly define these limits. This is the exact configuration I used to stabilize my costs.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: automation-engine
  annotations:
    run.googleapis.com/ingress: all
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/maxScale: '10'
        # This is the magic number for cost savings
        autoscaling.knative.dev/target: '40'
        # Ensure CPU is only charged during requests
        run.googleapis.com/cpu-throttling: 'true'
    spec:
      containerConcurrency: 40
      containers:
      - image: gcr.io/my-project/automation-engine:latest
        resources:
          limits:
            cpu: 1000m
            memory: 512Mi

One detail that tripped me up was the containerConcurrency vs autoscaling.knative.dev/target. The containerConcurrency is a hard limit—the container will literally refuse more than 40 simultaneous connections. The target is a "soft" limit used by the autoscaler to decide when to spin up a new instance. I set them both to 40 to ensure predictable behavior.

Why Your Gunicorn and Uvicorn Setup Affects Serverless Efficiency

A single-worker Uvicorn configuration is often more cost-effective for Cloud Run than the traditional multi-worker formula because it maximizes asynchronous I/O without extra memory overhead. Tuning Cloud Run is only half the battle. You also have to ensure your Python web server is configured to handle that concurrency. If you're running Gunicorn with Uvicorn workers (the standard for FastAPI), your worker count matters immensely.

Initially, I was using the "standard" formula of (2 x cores) + 1 workers. On a 1-CPU Cloud Run instance, that meant 3 workers. But if each worker is only handling one request at a time, I’m still stuck at a concurrency of 3. To actually utilize the 40-request concurrency I set in Cloud Run, I had to ensure my Uvicorn workers were truly asynchronous and that I wasn't blocking the event loop with synchronous code.

I switched my entrypoint to use a single worker with a high number of concurrent connections. Since Cloud Run gives me 1 vCPU, multiple OS-level workers just create unnecessary memory overhead due to Python's GIL. A single worker running uvicorn.workers.UvicornWorker is more than enough for 40 concurrent I/O-bound requests.

# My updated Dockerfile CMD
CMD ["gunicorn", "-k", "uvicorn.workers.UvicornWorker", "--workers", "1", "--threads", "1", "--bind", "0.0.0.0:8080", "main:app"]

Wait, why 1 worker and 1 thread? Because in an async FastAPI app, the "concurrency" happens inside the event loop, not through OS threads. Adding more Gunicorn threads just consumes more memory without giving me a performance boost for I/O-bound work. If I had heavy CPU-bound tasks (like image processing), this would be a different story, but for Gemini API calls, this is the leanest way to run.

Comparing the Performance Data: Before and After Optimization

Our testing showed that increasing concurrency from 1 to 40 reduced the average instance count from 14 to 2, cutting daily Cloud Run costs by over 50%. I monitored the transition for seven days. Here are the raw numbers from my Cloud Monitoring dashboard:

  • Old Setup (Concurrency 1, CPU Always On):
    • Average Instances: 14
    • Daily Cost: $16.40
    • P99 Latency: 2.8s
    • Memory Utilization: 12% (Massive waste)
  • New Setup (Concurrency 40, CPU Throttled):
    • Average Instances: 2
    • Daily Cost: $7.80
    • P99 Latency: 2.9s (Negligible increase)
    • Memory Utilization: 65% (Much healthier)

The cost reduction wasn't just from having fewer instances; it was also from the "CPU Throttling" feature. My instances now "rest" between tasks, and I’m only billed for the time they are actively chewing on data. The slight increase in P99 latency (100ms) was a trade-off I was more than willing to make for a 50%+ reduction in my bill.

What I Learned About Managing Cloud Run Costs

This exercise taught me that "serverless" doesn't mean "don't worry about resources." If anything, you have to be more diligent because the feedback loop (the bill) is so direct. Here are the core lessons to help you optimize Cloud Run costs:

  • Understand your workload: If your app is I/O-bound (APIs, DB calls), high concurrency is your best friend. If it's CPU-bound (ML inference, encryption), stick to lower concurrency.
  • Default settings are for safety, not savings: Cloud Run defaults to settings that ensure your app won't crash, but those settings often lead to massive over-provisioning.
  • CPU Throttling is a huge win: Unless you are running background threads that must finish after a response is sent, there is almost no reason to keep "CPU always allocated" for a standard API.
  • Monitor Memory Pressure: As you increase concurrency, your memory usage will climb. Watch for OOM (Out of Memory) kills. I had to bump my memory from 256Mi to 512Mi to support 40 concurrent Gemini requests safely.
  • Trust the Telemetry: Without the tracing setup I built previously, I would have been flying blind. I needed to see those P99 latencies to know I wasn't degrading the user experience.

Related Reading

Moving forward, I'm looking into "Min Instances" to see if I can eliminate the remaining cold start spikes without breaking the bank. There's a delicate balance between keeping an instance warm and paying for idle time, but with the concurrency now tuned to 40, a single "min instance" goes a lot further than it used to. I’ll be tracking the cost-to-latency ratio over the next month to see if that's the next logical step in this optimization journey.

Comments

Post a Comment

Popular posts from this blog

Why I Switched from FastAPI to Rust Axum for High-Performance AI Microservices

-
I'm sharing my journey as the lead developer of AutoBlogger, detailing why I decided to pivot from FastAPI to Rust's Axum framework for our high-performance AI microservices. I'll explain the specific performance bottlenecks I hit with Python, particularly concerning the Global Interpreter Lock (GIL) and memory footprint when serving complex AI models. This post will cover the benefits of Rust and Axum, including significant latency reductions and improved resource efficiency, and describe the challenges and triumphs of migrating critical components. I'll provide conceptual code examples and discuss the architectural changes, aiming to give other developers a transparent look into my decision-making process and the practical implications for building scalable AI applications. Why I Switched from FastAPI to Rust's Axum for AutoBlogger's High-Performance AI Microservices When I was building the posting service for AutoBlogger, my blog automation bot, I initia...
Read more

Optimizing LLM API Latency: Async, Streaming, and Pydantic in Production

-
Optimizing LLM API Latency: Async, Streaming, and Pydantic in Production I remember the sinking feeling in my stomach. It was late afternoon, and our monitoring dashboards, usually a soothing sea of green, had started to flash angry reds and oranges. Our users were reporting painfully slow response times, some even encountering timeouts. The core of the problem, I quickly pinpointed, was our interaction with external Large Language Model (LLM) APIs. What should have been near-instantaneous content generation was taking upwards of 10-15 seconds, sometimes more. This wasn't just a poor user experience; it was a ticking time bomb for our infrastructure costs, as longer request durations meant more concurrent instances and higher compute bills. This wasn't an isolated incident. As a lead developer, I've seen my fair share of production fires, but this one felt particularly insidious because the underlying issue wasn't immediately obvious. I was dealing with a distribute...
Read more

How I Built a Semantic Cache to Reduce LLM API Costs

-
How I Built a Semantic Caching Layer for AutoBlogger's LLM Responses to Cut API Costs My heart sank as I reviewed our monthly cloud bill for AutoBlogger. While the user growth was fantastic, the cost line item for LLM API usage had skyrocketed past all projections. We were on track for an astronomical bill, easily doubling what we’d paid the previous month. This wasn't just a slight overshoot; it was a full-blown financial alarm bell ringing at 3 AM. Our core feature—generating unique, high-quality blog posts and content snippets—relies heavily on large language models, and with increased usage, direct API calls were burning through our budget at an unsustainable rate. Something had to give, and fast. My immediate thought wasn't about switching providers or optimizing prompts (though we do that too); it was about intelligently reducing the *number* of calls we made in the first place. The Problem: LLM API Costs Eating Our Budget Alive AutoBlogger’s magic lies in its...
Read more