Go Cloud Run: Debugging and Fixing Persistent Connection Leaks

It was a Monday morning, and my coffee hadn't even kicked in when the alerts started screaming. Our core data processing service, running on Google Cloud Run, was showing a sharp increase in latency, 5xx errors, and, most alarmingly, a sudden spike in active instances and CPU utilization. This wasn't just a blip; it was a sustained climb, pushing our monthly bill estimates into uncomfortable territory. My heart sank – another production fire to extinguish.

This particular service is a critical component of our content generation pipeline, responsible for fetching data from various external APIs, processing it, and then pushing it to downstream ML models. It's written in Go, a language I love for its performance and concurrency primitives, but also one that can hide subtle resource management pitfalls if you're not careful. This time, the culprit turned out to be a classic: persistent connection leaks.

The Initial Symptoms: Latency, Errors, and a Soaring Bill

My first glance at Cloud Monitoring (formerly Stackdriver) confirmed the worst. The service's average request latency had jumped from a healthy 50ms to over 500ms, with p99 latency pushing into multiple seconds. Error rates, usually negligible, were now hovering around 5-10%. The most telling metric, however, was the instance count. Our service typically scaled down to a handful of instances during off-peak hours, but now it was stubbornly holding onto 30-40 instances, even with minimal traffic. CPU utilization per instance was also higher than usual, despite the service theoretically having idle capacity.

The cost projection dashboard painted an even grimmer picture. Based on the current trend, we were looking at a 3x increase in billing for this specific service by month-end. This wasn't just a performance regression; it was a financial drain that needed immediate attention.

Where I Looked First (and Why I Was Wrong)

My initial instinct, as it often is, was to check the usual suspects:

Database Connections: Was our PostgreSQL database overloaded? Were we opening too many connections without closing them? A quick check of database metrics showed normal connection counts and query performance.
External API Rate Limits: Were we hitting rate limits with our upstream providers? Our retry logic and circuit breakers usually handle this gracefully, and external API dashboards showed no unusual activity.
Application Logic Bugs: Had a recent deployment introduced a computationally expensive bug? I rolled back to the previous known-good version, but the problem persisted. This told me it wasn't a recent code change, but something more fundamental or a slow-burn issue that finally hit a breaking point.

I was stumped. The application itself seemed to be processing requests, but slowly, and consuming far more resources than it should. It felt like a slow memory leak, but my previous battle with Go memory leaks in my AI data pipeline had taught me to look beyond just heap usage. There had to be another resource being slowly exhausted.

The "Aha!" Moment: Open File Descriptors and Network Connections

Since Cloud Run instances are ephemeral and managed, getting direct shell access for traditional debugging tools like netstat or lsof isn't straightforward in a production environment. However, Cloud Monitoring does expose some key system metrics. I started digging deeper into the "System" metrics for my Cloud Run service.

That's when I saw it: a steadily increasing trend in "Open File Descriptors" and "TCP Connections (ESTABLISHED)" that didn't correlate with active request traffic. Even during periods of low incoming requests, these metrics would climb, albeit slowly, never really dropping back down. This was the smoking gun. It strongly pointed towards connections being opened but never properly closed or reused, leading to resource exhaustion.

In Go, an HTTP client connection, a database connection, or even a file handle are all represented by file descriptors at the OS level. If these aren't released, the OS eventually runs out of available descriptors, leading to errors and performance degradation. On a shared environment like Cloud Run, this means your container gets throttled or restarted, and new instances are spun up to compensate, driving up costs.

The Culprit: Mismanaged HTTP Client and Connection Pooling

Our service makes extensive use of external HTTP APIs. My investigation quickly narrowed down to how we were making these HTTP requests. A simplified, problematic pattern looked something like this:


package main

import (
	"fmt"
	"io"
	"net/http"
	"time"
)

func fetchExternalData(url string) ([]byte, error) {
	// BAD PRACTICE: Creating a new http.Client for every request
	// This creates a new http.Transport and connection pool for each call,
	// leading to connection leaks and inefficient resource use.
	client := &http.Client{
		Timeout: 10 * time.Second,
	}

	resp, err := client.Get(url)
	if err != nil {
		return nil, fmt.Errorf("failed to make GET request: %w", err)
	}
	defer resp.Body.Close() // ALWAYS defer Body.Close()!

	body, err := io.ReadAll(resp.Body)
	if err != nil {
		return nil, fmt.Errorf("failed to read response body: %w", err)
	}

	return body, nil
}

// In a real application, this would be called repeatedly.
func main() {
	// ... Imagine this function being called hundreds of times per second
	// by a Cloud Run instance.
	data, err := fetchExternalData("https://api.example.com/data")
	if err != nil {
		fmt.Println("Error:", err)
		return
	}
	fmt.Println("Fetched data length:", len(data))
}

The core issue here is creating a new http.Client for every single request. While defer resp.Body.Close() is correctly used (which is crucial to prevent the *response body* from leaking), creating a new http.Client implicitly creates a new http.Transport. An http.Transport is responsible for managing HTTP connection pooling and reuse. When you create a new client for every request, you're effectively discarding the connection pool after each request, preventing connections from being reused and allowing them to linger until they time out naturally, consuming file descriptors in the process.

The Solution: Reusing `http.Client` with a Configured `http.Transport`

The fix involved a fundamental change in how we initialized and used our HTTP client. Instead of creating a new client per request, we needed to create a single, shared http.Client instance, ideally with a custom http.Transport configured for optimal connection reuse and timeouts. This is a well-documented best practice in Go's net/http package. You can find more details in the official Go documentation for http.Transport.

Here's how I refactored the client:


package main

import (
	"fmt"
	"io"
	"net/http"
	"time"
)

// Declare a global or package-level http.Client.
// Initialize it once, ideally during application startup.
var httpClient *http.Client

func init() {
	// Configure a custom Transport for better control over connection pooling
	// and timeouts. This is critical for long-running services.
	transport := &http.Transport{
		MaxIdleConns:        100,              // Max idle connections across all hosts
		MaxIdleConnsPerHost: 20,               // Max idle connections per host
		IdleConnTimeout:     90 * time.Second, // How long an idle connection is kept alive
		TLSHandshakeTimeout: 10 * time.Second,
		ExpectContinueTimeout: 1 * time.Second,
	}

	httpClient = &http.Client{
		Timeout:   30 * time.Second, // Overall request timeout
		Transport: transport,
	}
}

func fetchExternalDataOptimized(url string) ([]byte, error) {
	// GOOD PRACTICE: Use the shared http.Client
	resp, err := httpClient.Get(url)
	if err != nil {
		return nil, fmt.Errorf("failed to make GET request with optimized client: %w", err)
	}
	defer resp.Body.Close()

	// Crucial: Ensure the response body is fully read OR discarded
	// This allows the connection to be reused in the pool.
	_, err = io.Copy(io.Discard, resp.Body)
	if err != nil {
		// Log this error, but don't necessarily fail the request if body read isn't critical
		fmt.Printf("Warning: Failed to fully discard response body for %s: %v\n", url, err)
	}

	body, err := io.ReadAll(resp.Body) // io.ReadAll will read from the already-discarded body, returning empty if discarded fully
	if err != nil {
		return nil, fmt.Errorf("failed to read response body after discard: %w", err)
	}

	return body, nil
}

func main() {
	// ... The rest of your application logic, now using fetchExternalDataOptimized
	data, err := fetchExternalDataOptimized("https://api.example.com/data")
	if err != nil {
		fmt.Println("Error:", err)
		return
	}
	fmt.Println("Fetched data length (optimized):", len(data))
}

The key changes were:

Global/Package-level httpClient: Initialized once during application startup using an init() function.
Custom http.Transport: Configured with sensible MaxIdleConns, MaxIdleConnsPerHost, and IdleConnTimeout values. These settings tell Go how many connections to keep in the pool and for how long, allowing efficient reuse.
io.Copy(io.Discard, resp.Body): Even after calling defer resp.Body.Close(), it's a good practice to fully read (or discard) the response body if you don't need its content. This ensures the underlying connection can be returned to the connection pool for reuse. Without this, the connection might not be fully closed or returned to the pool until the garbage collector eventually processes the resp.Body, which can lead to delays and resource exhaustion.

While this specific example focuses on http.Client, similar principles apply to database connections (e.g., using sql.DB.SetMaxOpenConns and SetMaxIdleConns for proper pooling) and other network resources. My previous blog post, How I Fixed Memory Leaks in My Go AI Data Pipeline, also delves into general Go resource management, and there's definitely overlap in the mindset required for debugging these kinds of issues.

Impact on Cloud Run: Performance and Cost Recovery

After deploying the fix, the change was almost immediate and dramatic. Within minutes, the Cloud Monitoring dashboards began to normalize:

Instance Count: Dropped from 30-40 instances back down to 3-5 during peak times, and scaled down to 0-1 during idle periods.
CPU Utilization: Significantly reduced, indicating less overhead per request.
Latency: Returned to its usual low-double-digit milliseconds.
Error Rate: Back to near zero.
Open File Descriptors / TCP Connections: Stabilized and showed healthy fluctuations correlating with actual traffic, rather than a continuous climb.

The cost savings were equally impressive. By reducing the number of active instances and their CPU usage, we brought the projected monthly cost for this service back down to its normal levels, effectively saving us an estimated $800-$1000 per month just for this single service. This experience reinforced my belief that optimizing resource utilization, especially in serverless environments, directly translates to significant cost reductions. This aligns well with the strategies I discussed in My Journey to 70% Savings: Optimizing Machine Learning Inference on AWS Lambda, where proper resource management was key to significant cost reductions.

What I Learned / The Challenge

This debugging saga was a potent reminder that while Go is excellent for performance, it doesn't absolve developers from understanding fundamental resource management, especially when dealing with network I/O. The challenge lies in the subtlety of these leaks: they don't always crash your application immediately. Instead, they manifest as a slow, insidious degradation of performance and a steady increase in operational costs. In a serverless environment like Cloud Run, where you pay for compute time and memory, such leaks are particularly costly because they prevent instances from scaling down efficiently and keep them busy with lingering resources.

My key takeaways from this experience are:

Don't recreate http.Client: Always reuse a single instance of http.Client, ideally configured with a custom http.Transport.
Always defer resp.Body.Close(): This is non-negotiable for every HTTP request.
Discard response bodies: If you don't need the full response body, explicitly discard it using io.Copy(io.Discard, resp.Body) before closing. This ensures the underlying connection is immediately available for reuse in the pool.
Monitor low-level system metrics: Don't just look at application-level metrics. Keep an eye on system metrics like open file descriptors, TCP connections, and memory usage. These often reveal the true nature of resource exhaustion.
Understand your platform: Cloud Run's autoscaling and billing model amplifies the impact of resource leaks. What might be a minor performance hit on a long-running VM can become a major cost center in a serverless function.

Search This Blog

TechFrontier | AI Engineering, Go & Cloud Cost Optimization