Why I Chose Go for Building a High-Performance LLM API Proxy

Migrating an LLM API proxy from Python to Go reduces memory usage by up to 90% and significantly lowers latency for streaming connections. Go’s goroutines handle thousands of concurrent Server-Sent Events (SSE) more efficiently than Python’s event loop, leading to substantial infrastructure cost savings.

Three months ago, my team’s production LLM gateway hit a wall. We were running a FastAPI-based proxy on Google Cloud Run to handle requests to various model providers. On paper, it worked. But as soon as we scaled to 500 concurrent users—each maintaining a long-lived streaming connection for real-time text generation—the service started behaving erratically. Our p99 latency for the initial "Time to First Token" (TTFT) jumped from 200ms to over 3 seconds. Worse, our Cloud Run memory usage spiked to 2GB per instance, triggering aggressive auto-scaling that sent our GCP bill into a tailspin.

I realized we had hit the ceiling of Python’s asynchronous model for this specific LLM API proxy use case. While asyncio is powerful, managing thousands of concurrent, long-lived HTTP streams while performing middle-box tasks like request validation, rate limiting, and cost tracking is a heavy lift for a single-threaded event loop. After a frantic weekend of load testing, I decided to rewrite the entire proxy layer in Go. This wasn't a decision made because Go is "trendy," but because the architectural requirements of an LLM proxy—high concurrency, low memory overhead, and robust streaming support—map perfectly to Go's primitives.

How Python Async Increases Memory Costs for an LLM API Proxy

Python’s asynchronous model often incurs a high memory footprint per connection when managing long-lived LLM streams. The primary issue I encountered with our Python implementation was the memory footprint per connection. When you are proxying a streaming response from an LLM provider (like OpenAI or Anthropic), you aren't just passing bytes through. You are often intercepting the stream to calculate token usage or to implement a multi-tier caching system for LLM API responses. In Python, each await point in a stream adds overhead to the event loop's task management.

In our FastAPI setup, the memory overhead per connection was roughly 4MB. That doesn't sound like much until you have 1,000 concurrent streams. That’s 4GB of RAM just for the connection state, excluding the application logic. On Cloud Run, this forced us to use higher-tier instances (2 vCPUs, 4GB RAM), which are significantly more expensive. When I benchmarked a prototype in Go, the memory footprint per connection dropped to less than 200KB. This was the first sign that I was on the right track.

How to Architect a High-Performance Go LLM API Proxy

Utilizing Go’s net/http/httputil package allows for efficient request interception and token counting without blocking the main response flow. The core of the new system relies on Go’s net/http/httputil package. Most developers use it for simple reverse proxies, but for an LLM API proxy, you need to customize the Transport and the ModifyResponse hooks to handle things like token counting and dynamic routing. Here is a simplified version of the proxy structure I implemented:

package main

import (
	"bytes"
	"context"
	"io"
	"net/http"
	"net/http/httputil"
	"net/url"
	"time"
)

type LLMProxy struct {
	target *url.URL
	proxy  *httputil.ReverseProxy
}

func NewLLMProxy(targetHost string) *LLMProxy {
	url, _ := url.Parse(targetHost)
	p := &LLMProxy{
		target: url,
		proxy:  httputil.NewSingleHostReverseProxy(url),
	}

	// Customizing the transport to handle connection pooling
	p.proxy.Transport = &http.Transport{
		MaxIdleConns:        100,
		IdleConnTimeout:     90 * time.Second,
		MaxIdleConnsPerHost: 100,
	}

	p.proxy.ModifyResponse = p.handleStreamingResponse
	return p
}

func (p *LLMProxy) handleStreamingResponse(res *http.Response) error {
	// This is where we intercept the stream for token counting
	if res.Header.Get("Content-Type") == "text/event-stream" {
		// We use a TeeReader to read the stream without consuming it
		originalBody := res.Body
		reader, writer := io.Pipe()
		res.Body = reader

		go func() {
			defer originalBody.Close()
			defer writer.Close()
			
			// Custom logic to parse SSE chunks and count tokens
			// without blocking the main response flow
			io.Copy(writer, originalBody)
		}()
	}
	return nil
}

func (p *LLMProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	// Injecting API Keys and modifying headers
	r.Host = p.target.Host
	r.Header.Set("Authorization", "Bearer " + getSecretKey())
	
	p.proxy.ServeHTTP(w, r)
}

The beauty of this approach is in the io.Pipe() and the goroutine. In Go, starting a goroutine is incredibly cheap, requiring only about 2KB of stack space. I can spin up a dedicated routine for every single streaming request to handle side-car tasks like logging or usage tracking without adding latency to the client's response. This is a massive win over the Python event loop, where long-running CPU-bound tasks (like parsing JSON chunks in a stream) can block the entire loop if not handled with extreme care.

How Go Reduces Cloud Run Costs for LLM API Proxy Workloads

Switching to Go enables the use of smaller Cloud Run instances, reducing monthly infrastructure bills by up to 70%. One of my main goals was to reduce our monthly GCP bill. In a previous post, I discussed predicting LLM API costs as a pre-production strategy, but infrastructure costs are often the "hidden" part of that equation. By switching to Go, I was able to downsize our Cloud Run instances from 2 vCPU / 4GB RAM to 1 vCPU / 512MB RAM.

The performance density of Go is staggering. On a single 512MB instance, we are now handling 2,000 concurrent requests with a p99 overhead of only 15ms. This efficiency directly impacts the bottom line. Because Cloud Run charges per request and per allocated resource-second, reducing the memory footprint by 8x and improving the execution speed allowed us to handle the same traffic for about 30% of the original cost.

Why Context Cancellation is Essential for LLM API Proxy Efficiency

Propagating context cancellation ensures that upstream requests are terminated immediately when a client disconnects, preventing wasted token costs. In the world of LLMs, users often cancel requests mid-stream. If your proxy doesn't handle these cancellations properly, you end up paying for tokens that the user never saw. Go’s context package makes this trivial. When a client closes the connection, the r.Context().Done() channel signals, allowing us to immediately terminate the upstream request to the model provider.

func (p *LLMProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
	defer cancel()

	// Pass the context to the upstream request
	outReq := r.WithContext(ctx)
	
	// If the client disconnects, outReq context is cancelled, 
	// and httputil.ReverseProxy handles closing the upstream connection.
	p.proxy.ServeHTTP(w, outReq)
}

This simple pattern saved us thousands of dollars in "ghost" token usage. In our old Python setup, we had a bug where the httpx client didn't always propagate the cancellation correctly, leading to upstream requests running to completion even after the user had closed their browser tab. For more on managing these types of costs, check out my guide on optimizing LLM API costs for batch processing workloads.

How to Handle Server-Sent Events (SSE) in an LLM API Proxy

Proper SSE handling in Go requires disabling response buffering to maintain the real-time interactivity of LLM responses. LLM providers almost exclusively use SSE for streaming. Handling this in Go requires a bit more boilerplate than in Python, but the control you get is worth it. You have to be careful not to buffer the response. If your proxy buffers the SSE stream, the user will see a long pause followed by a massive wall of text, destroying the "typing" effect that makes LLMs feel interactive.

The httputil.ReverseProxy handles this by default by flushing the response periodically, but I found that I needed to explicitly set the FlushInterval to -1 or a very small value to ensure immediate delivery of tokens. Additionally, you should consult the official Go net/http documentation to understand how the ResponseWriter hijacking works if you need to do more advanced manipulation of the raw TCP stream.

Performance Benchmarks: Go vs. Python for LLM API Proxy Tasks

Load testing proves that Go handles nearly ten times the concurrent streams as Python while consuming a fraction of the CPU and memory. I ran a series of load tests using k6 to validate the migration. The test simulated 1,000 concurrent users requesting 50 tokens each with a 100ms delay between tokens (simulating real LLM latency).

Metric	Python (FastAPI)	Go (Standard Library)	Improvement
Avg CPU Usage (1k users)	85%	12%	7.0x
Memory Usage (1k users)	3.2 GB	240 MB	13.3x
TTFT p99 (ms)	450ms	42ms	10.7x
Max Concurrent Streams	~1,200 (OOM)	10,000+	8.3x+

The numbers speak for themselves. The Python service would eventually crash due to Out-Of-Memory (OOM) errors when connections lingered too long. The Go service, meanwhile, barely broke a sweat. The garbage collector in Go is also much better suited for this type of workload. Since we are creating many small objects (SSE chunks), Go’s generational GC handles the churn with minimal pauses, whereas Python’s reference counting and cyclic GC often led to noticeable latency spikes during cleanup.

Key Challenges When Migrating an LLM API Proxy to Go

While Go requires more verbose code for JSON handling, the resulting type safety prevents common runtime errors in production. The migration wasn't without its headaches. The biggest challenge was reimplementing the complex response parsing logic we had in Python. We used a lot of Pydantic models for validating LLM outputs on the fly. In Go, I had to write custom JSON unmarshalers and use struct tags heavily. It’s more verbose, and I miss the "magic" of Pydantic sometimes, but the type safety in Go has caught several edge cases where an upstream provider changed their response schema slightly.

Another lesson: Don't over-engineer the concurrency. In the beginning, I tried to use a complex worker pool for processing the streams. I quickly realized that Go's scheduler is much better at managing thousands of goroutines than any manual pool I could write. I reverted to a simple "one goroutine per request" model, and performance actually improved because I removed the overhead of channel synchronization in the worker pool.

Summary of Benefits for Using Go in an LLM API Proxy

The migration to Go demonstrates that concurrency primitives and memory efficiency are the primary drivers for scaling middleware. Key takeaways from this project include:

Concurrency Primitives Matter: For I/O-bound tasks like proxying thousands of streaming connections, Go’s goroutines provide a massive scalability advantage over Python’s single-threaded event loop.
Memory Efficiency is Cost Efficiency: On serverless platforms like Cloud Run, reducing your memory footprint directly translates to lower monthly bills. Go’s ability to handle high concurrency with minimal RAM is its "killer feature" for middleware.
Streaming requires care: When proxying SSE, ensure you are not buffering responses. Use io.Pipe or io.TeeReader to inspect streams without blocking the data flow to the client.
Type Safety saves production: While Go is more verbose than Python, its strict typing and compile-time checks have prevented several runtime failures during upstream API updates.
Context is King: Always propagate context cancellation to upstream providers. It prevents wasted compute and saves significant costs on token usage.

Search This Blog

TechFrontier | AI Engineering, Go & Cloud Cost Optimization