Replacing Celery with FastAPI Background Tasks for AI Automation

-

FastAPI Background Tasks provide a lightweight alternative to Celery by executing functions after a response is sent within the same process. This migration can reduce cloud infrastructure costs by over 70% for small-scale AI automation workflows on Google Cloud Run.

I was staring at my Google Cloud Billing console at 2:14 AM on a Tuesday when I realized I had built a monster. My "simple" AI automation agent, which was supposed to handle a few hundred PDF processing tasks a day, was costing me $422 a month just in infrastructure overhead. Most of that wasn't even the LLM tokens—it was the managed Redis instance and the three extra Cloud Run services I had spun up to act as Celery workers. Even worse, I was dealing with a recurring "Ghost Task" bug where tasks would re-run indefinitely because the worker's visibility timeout was shorter than the Gemini API's response time during a peak period.

I didn't need a distributed task queue. I needed a way to fire a function and return a 202 Accepted response to my webhook. I realized I had fallen into the classic trap of over-engineering for "scale" that didn't exist yet. Last month, I deleted the entire Celery stack and moved everything into native FastAPI Background Tasks. The result? My GCP bill dropped by 30%, my codebase is 500 lines lighter, and I haven't had a single "stuck task" since. If you are running small-to-medium automation jobs on Cloud Run, you might be making the same mistake I was.

Why Celery Increases Infrastructure Costs in Serverless Environments

Running Celery on serverless platforms like Google Cloud Run often requires expensive "always-on" compute and managed brokers that inflate monthly bills. When we think of background tasks in Python, Celery is the default choice. It’s powerful, battle-tested, and has a feature for everything. But in a serverless environment like Google Cloud Run, Celery is often a liability. To run Celery properly, you need a broker (usually Redis or RabbitMQ) and a persistent worker process. On GCP, this means paying for Memorystore (which starts at about $35/month for the smallest instance) and keeping a Cloud Run service "always on" to listen for tasks.

My specific breaking point was a tool-calling loop. I was using Gemini to analyze complex financial documents, a process I detailed in my previous post on Scaling AI Agent Logic with Gemini Tool Use and Structured Outputs. Because these LLM calls can take 30 to 60 seconds, I couldn't run them in the main request-response cycle. I shoved them into Celery. But Cloud Run’s request-based scaling doesn't play nice with Celery workers. If the worker is processing a task but isn't receiving an HTTP request, Cloud Run might throttle its CPU to near zero, causing the task to hang or fail silently.

To fix this, I had to enable "CPU always allocated," which effectively turned my serverless function into a very expensive, underutilized VM. I was paying for 24/7 compute to handle tasks that only ran for 2 hours a day in total. That is not why I moved to the cloud.

How to Implement FastAPI Background Tasks for Efficient Processing

FastAPI Background Tasks leverage the Starlette framework to schedule post-response execution without requiring a separate worker process or message broker. FastAPI provides a BackgroundTasks class that allows you to schedule a function to run after the response has been sent to the client. It uses starlette.background under the hood. It’s not a separate process; it runs in the same event loop (for async def functions) or in a separate thread (for standard def functions) as your API.

Here is the core pattern I switched to. Notice how much cleaner the dependency injection is compared to setting up a Celery app instance with broker URLs and result backends.

from fastapi import FastAPI, BackgroundTasks, Depends
from typing import Dict
import uuid
import time

app = FastAPI()

# A simple in-memory or database-backed status tracker
task_status: Dict[str, str] = {}

def process_ai_automation(task_id: str, payload: dict):
    """
    This is the heavy lifting. In my case, it's calling Gemini 
    and updating a database.
    """
    try:
        task_status[task_id] = "processing"
        # Simulate heavy LLM work
        time.sleep(10) 
        # Integration with GCP logic here
        task_status[task_id] = "completed"
    except Exception as e:
        task_status[task_id] = f"failed: {str(e)}"

@app.post("/analyze")
async def start_analysis(payload: dict, background_tasks: BackgroundTasks):
    task_id = str(uuid.uuid4())
    task_status[task_id] = "queued"
    
    # This is the magic. The API returns immediately, 
    # but the function keeps running.
    background_tasks.add_task(process_ai_automation, task_id, payload)
    
    return {"task_id": task_id, "status": "accepted"}

The beauty of this is that it respects the Cloud Run lifecycle. As long as the background task is running, the instance stays active. However, there is a massive caveat: you must ensure your Cloud Run instance doesn't shut down before the task finishes. This is where most developers get burned.

How to Ensure Task Reliability with Firestore State Management

A "State-First" architecture using Firestore provides the necessary persistence to make ephemeral background tasks resilient against container shutdowns. The biggest argument against BackgroundTasks is that they are ephemeral. If the container crashes or scales down, the task is gone. Celery solves this with persistent brokers. To bridge this gap without the Celery overhead, I implemented a "State-First" pattern using Firestore.

Before adding the task to the BackgroundTasks queue, I write a record to Firestore with a PENDING status. The background task updates this to COMPLETED or FAILED. I then set up a simple Cloud Scheduler job that runs every 30 minutes to check for PENDING tasks older than 10 minutes. If it finds any, it re-queues them via a private internal API call. This gives me the "at-least-once" delivery guarantee of Celery without the $400/month price tag.

I also had to be careful about performance. If you're interested in how I tune these types of services on GCP, check out my guide on How to Optimize Go API Performance on Google Cloud Run—while that post focuses on Go, the principles of concurrency and CPU allocation in Cloud Run apply directly to Python as well.

Should You Use Async or Sync for FastAPI Background Tasks?

Properly selecting between synchronous and asynchronous function definitions prevents CPU-bound background tasks from blocking the FastAPI event loop. One thing I learned the hard way: FastAPI runs async def background tasks in the main event loop. If your background task is CPU-bound (like heavy data processing) and you use async def, you will block the entire API from responding to new requests. Always use standard def for CPU-heavy background tasks so FastAPI runs them in a separate thread pool. If you're calling external APIs (like the Gemini API or a database), async def is fine, provided you use an asynchronous client.

# Scenario A: CPU Bound (Use def)
def sync_cpu_work(data: dict):
    # This runs in a separate thread. 
    # It won't block the main event loop.
    result = perform_heavy_math(data)
    save_to_db(result)

# Scenario B: I/O Bound (Use async def)
async def async_io_work(data: dict):
    # This runs in the event loop. 
    # Use await for every I/O call.
    async with httpx.AsyncClient() as client:
        response = await client.post("https://api.gemini.com/v1/...", json=data)
    await save_to_db_async(response.json())

Comparing Costs: Celery vs. FastAPI Background Tasks on GCP

The transition to a native FastAPI approach resulted in a 73% reduction in total monthly infrastructure costs on Google Cloud Platform. After running the new architecture for 30 days, I pulled the logs and billing data. The difference was stark. My previous Celery-based architecture required a minimum of two n1-standard-1 instances (or equivalent Cloud Run min-instances) to ensure the worker was always ready.

Component Celery + Redis Stack (Monthly) FastAPI Background Tasks (Monthly)
Managed Redis (Memorystore) $35.00 $0.00
Worker Compute (Always-on) $120.00 $0.00
API Compute (Request-based) $45.00 $52.00*
Networking & Overhead $12.00 $4.00
Total $212.00 $56.00

*Note: The API compute cost increased slightly because the instances stay alive longer to finish background tasks.

Beyond the cost, the latency of starting a task dropped from ~150ms (network round trip to Redis + worker pickup) to effectively 0ms, as the task is registered in-memory instantly. For a developer, the biggest win was the debugging experience. I no longer had to check three different log streams (API, Redis, Worker) to figure out why a document wasn't processed. Everything is in one sequential log trace.

How to Monitor FastAPI Background Tasks with OpenTelemetry

Integrating OpenTelemetry and Cloud Monitoring is essential for maintaining visibility into task success rates after removing Celery's built-in monitoring tools. The danger of moving away from Celery is losing the built-in monitoring tools like Flower. When I first switched, I had a silent failure where a background task would crash due to a database connection timeout, and the API would still return a 202. The client thought everything was fine, but the work never finished.

I solved this by integrating custom OpenTelemetry metrics. Every time a task starts, fails, or succeeds, I increment a counter that feeds into Cloud Monitoring. I wrote a detailed breakdown of this approach in my post on GCP Custom Metrics for AI Agent Anomaly Detection. If you're going to use BackgroundTasks, you must have an alert set up for task failures, or you will lose data.

Here is the middleware-style wrapper I use to ensure every background task is logged and metered properly:

import logging
from functools import wraps

logger = logging.getLogger("background_tasks")

def monitored_task(name: str):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            logger.info(f"Starting background task: {name}")
            try:
                result = func(*args, **kwargs)
                logger.info(f"Task {name} completed successfully")
                return result
            except Exception as e:
                logger.error(f"Task {name} failed: {str(e)}", exc_info=True)
                # Here you would push to Google Cloud Metrics
                # push_metric(name, "failure")
                raise
        return wrapper
    return decorator

@monitored_task("pdf_summary_job")
def process_pdf(file_id: str):
    # Actual logic here
    pass

When is Celery Necessary for High-Scale Task Management?

Distributed task queues like Celery or Google Cloud Tasks are still required for massive task bursts or strict exactly-once delivery guarantees. I am not saying Celery is obsolete. I still use it for my heavy-duty scraping projects that require thousands of concurrent workers and complex rate-limiting. You should stay on Celery (or move to a dedicated queue like Google Cloud Tasks) if:

  • You need strict Exactly-Once processing: FastAPI BackgroundTasks are not designed for this. If the container dies, the task dies.
  • You have massive bursts: If you suddenly drop 50,000 tasks into the queue, you want a broker that can persist them while your workers scale up. FastAPI will try to handle them in-memory, which will lead to OOM (Out of Memory) errors.
  • You need task prioritization: If some tasks must jump the queue, Celery's priority features are essential.

But for the "AI Agent" use case—where you're doing 5-10 minute jobs that are triggered by a user action—Celery is a heavy tax you don't need to pay. For more details on the limitations, I highly recommend reading the official FastAPI BackgroundTasks documentation, which clarifies that these tasks are intended for things like sending emails or small processing jobs.

Key Takeaways for Modern Python Backend Architecture

Simplifying the backend stack by removing unnecessary distributed layers leads to faster deployments and improved system maintainability. The last few months of simplifying my stack taught me three major lessons:

  • In-memory is often enough: We are conditioned to think everything needs to be distributed. For small automation jobs, the overhead of distribution is often higher than the risk of a container restart.
  • Cloud Run lifecycle awareness is critical: If you use BackgroundTasks, you must understand how your cloud provider handles instance termination. On Cloud Run, the instance stays alive as long as it's processing, but it has a hard limit (usually 60 minutes).
  • State should live in the database, not the queue: By making my tasks idempotent and tracking their state in Firestore, I made my system more resilient than it ever was with Celery. I can now "replay" failed tasks just by changing a status flag in a UI.
  • The "Boring" stack wins: Deleting Redis and the Celery workers reduced my deployment complexity. My CI/CD pipeline runs faster, and I have fewer moving parts to secure.

Related Reading

Moving forward, I’m looking into Google Cloud Tasks as a middle ground. It offers the persistence of a queue without the need to manage a Redis instance or a persistent worker pool. It’s essentially "serverless Celery." But for now, the FastAPI BackgroundTasks approach is holding up perfectly under my current load. My next challenge is refactoring my notification engine to use this same pattern, which should shave another $20 off my monthly bill. If you're drowning in Celery boilerplate, try deleting it. You might be surprised at how little you actually needed it.

Comments

Post a Comment

Popular posts from this blog

Why I Switched from FastAPI to Rust Axum for High-Performance AI Microservices

-
I'm sharing my journey as the lead developer of AutoBlogger, detailing why I decided to pivot from FastAPI to Rust's Axum framework for our high-performance AI microservices. I'll explain the specific performance bottlenecks I hit with Python, particularly concerning the Global Interpreter Lock (GIL) and memory footprint when serving complex AI models. This post will cover the benefits of Rust and Axum, including significant latency reductions and improved resource efficiency, and describe the challenges and triumphs of migrating critical components. I'll provide conceptual code examples and discuss the architectural changes, aiming to give other developers a transparent look into my decision-making process and the practical implications for building scalable AI applications. Why I Switched from FastAPI to Rust's Axum for AutoBlogger's High-Performance AI Microservices When I was building the posting service for AutoBlogger, my blog automation bot, I initia...
Read more

Optimizing LLM API Latency: Async, Streaming, and Pydantic in Production

-
Optimizing LLM API Latency: Async, Streaming, and Pydantic in Production I remember the sinking feeling in my stomach. It was late afternoon, and our monitoring dashboards, usually a soothing sea of green, had started to flash angry reds and oranges. Our users were reporting painfully slow response times, some even encountering timeouts. The core of the problem, I quickly pinpointed, was our interaction with external Large Language Model (LLM) APIs. What should have been near-instantaneous content generation was taking upwards of 10-15 seconds, sometimes more. This wasn't just a poor user experience; it was a ticking time bomb for our infrastructure costs, as longer request durations meant more concurrent instances and higher compute bills. This wasn't an isolated incident. As a lead developer, I've seen my fair share of production fires, but this one felt particularly insidious because the underlying issue wasn't immediately obvious. I was dealing with a distribute...
Read more

How I Built a Semantic Cache to Reduce LLM API Costs

-
How I Built a Semantic Caching Layer for AutoBlogger's LLM Responses to Cut API Costs My heart sank as I reviewed our monthly cloud bill for AutoBlogger. While the user growth was fantastic, the cost line item for LLM API usage had skyrocketed past all projections. We were on track for an astronomical bill, easily doubling what we’d paid the previous month. This wasn't just a slight overshoot; it was a full-blown financial alarm bell ringing at 3 AM. Our core feature—generating unique, high-quality blog posts and content snippets—relies heavily on large language models, and with increased usage, direct API calls were burning through our budget at an unsustainable rate. Something had to give, and fast. My immediate thought wasn't about switching providers or optimizing prompts (though we do that too); it was about intelligently reducing the *number* of calls we made in the first place. The Problem: LLM API Costs Eating Our Budget Alive AutoBlogger’s magic lies in its...
Read more