Diagnosing a slow Go API: measure, profile and verify

Start by separating CPU work from waiting

A slow Go API needs a measured diagnosis before a rewrite. Record latency percentiles, throughput and errors under a defined workload, then investigate the resource that explains the delay. CPU profiles help with active computation; traces and dependency timings help explain waiting. Verify a proposed change against the same workload and an independent correctness check.

My starting point would be the operation the user experiences: which endpoint, which input, which percentile and which concurrency level? A service can have a healthy average while a smaller group of requests waits much longer. Changing code before describing that group makes the result difficult to assess.

Write down a baseline you can reproduce

Choose representative request sizes and a fixed observation window. Record the application revision, Go version, CPU and memory limits, replica count and dependency conditions. Keep successful and failed requests visible. A reduction in latency caused by returning more errors is not an improvement.

Start with one hypothesis. For example, a larger response may allocate more memory, or requests may wait for an exhausted database connection pool. Compare equivalent traffic before and after the change. If the workload or resource limits change too, record that explicitly rather than attribute the whole difference to one function.

Collect end-to-end latency and error rate separately from a microbenchmark. A benchmark can compare a small implementation detail, but it cannot reproduce network queues, database contention, retries or the cost of a full request. Keep the scope of each measurement attached to the result.

• Identify the endpoint, input size and concurrency.
• Record latency percentiles, throughput and errors.
• Keep the Go version, revision and resource limits with the results.
• Write the expected behavior before changing the implementation.

Choose the diagnostic tool from the symptom

For CPU saturation, collect a CPU profile during representative work and inspect the expensive call paths. For allocation pressure, inspect allocation samples and garbage-collection activity. For long requests with spare CPU, trace the request across dependencies and inspect connection waits, deadlines and synchronization.

A profile answers a narrower question than “why is the API slow?” A function can occupy a large share of sampled CPU without explaining the time spent waiting on another service. Treat a missing hot spot as a reason to revisit the hypothesis, rather than add concurrency blindly.

Collect profiles separately when practical: diagnostic tools have overhead and can influence one another. The official Go diagnostics reference explains the available profiles and their limits. For a production investigation, agree on collection scope, access and acceptable overhead with the team first.

Worked example: repeated string construction

This is an independent educational demonstration, not a client case study or a production performance claim. The downloadable Go file compares repeated string concatenation with a strings.Builder. Both functions produce the same comma-separated output. Table-driven tests compare their results with strings.Join, including empty inputs, empty values and Unicode.

The benchmark uses two input sizes and reports allocations. Keeping the result in a package variable prevents the result from simply being discarded. Run repeated samples and inspect both time and allocation measurements. The example intentionally isolates one operation; it does not simulate an HTTP request.

The commands below require a local Go toolchain. No benchmark numbers are asserted here. Run the example locally before drawing conclusions. Save the linked file in its own directory so generated profiles and test binaries stay separate from application files.

# Save the linked encode_test.go in an empty working directory.
go version
go test -run TestOutput -bench BenchmarkEncode -benchmem -count=5 encode_test.go
# Capture one profile type at a time.
go test -run '^$' -bench 'BenchmarkEncode/concatenate/1024$' -benchtime=3s -cpuprofile=cpu.out encode_test.go
go tool pprof -top cpu.out
go test -run '^$' -bench 'BenchmarkEncode/concatenate/1024$' -benchtime=3s -memprofile=heap.out encode_test.go
go tool pprof -alloc_space -top heap.out

Interpret the example without promising an API speedup

If your measurements show fewer allocations or less time for the builder variant at a particular size, that supports a local implementation choice for that workload. It does not establish that an API will improve by the same proportion. The actual endpoint may spend almost all of its time elsewhere.

In a service, first verify that response construction appears in the evidence. Keep a correctness test around the representation, make a small change, and repeat the endpoint workload. Compare tail latency, errors and resource use together. If no material endpoint improvement appears, keep that result in the investigation notes.

A useful handover states the observed condition, the change, how it was checked and what remains uncertain. Include the original data rather than only a percentage. Another engineer should be able to repeat the investigation and decide whether the tradeoff still holds after a later release.

Independent expectations for Go tests

Keep diagnostic endpoints away from public traffic

When using net/http/pprof, keep diagnostic handlers on an explicitly controlled listener or administrative route. A separate port alone is not access control. Confirm the bind address, network policy and authentication appropriate to the environment before collecting profiles.

Profiles and traces may reveal internal implementation details. Limit access and retention, and collect only what the investigation needs. The downloadable example uses go test profile files and does not start an HTTP server or expose a diagnostic endpoint.

Turn the investigation into a focused engineering engagement

For a consulting discussion, bring a description of the affected endpoint, a sanitized sample workload, available measurements and the outcome you need. We can use that context to define whether the scope is diagnosis, an implementation change, or support for your team’s measurement process.

This fits my focus on Go backends and Kubernetes platforms. My availability is limited alongside full-time employment; scope and timing need to be agreed before implementation. The linked consulting page describes relevant employment experience and how I approach a focused engagement.

Related guide: observable Kubernetes reconciliation