C++ Performance Optimization Case Study

Introduction

The starting point is a familiar complaint: “the API feels slow.” This case study walks through one endpoint with the loop measure → analyze → optimize → verify. I deliberately do not quote millisecond figures here: absolute latencies depend on the hardware, the data and the compiler, and a number from someone else’s machine is no substitute for running the harness on yours. What carries over is the order of the steps and the habit of re-measuring after each one.

Two habits matter more than any specific number. First, the profile decides the order of work, not intuition: the change that looked most “technical” (parallel serialization) was the least important, and the boring one (an index) delivered most of the gain. Second, every change is followed by a measurement under the same conditions, which is the only way to know that a change helped at all; optimizations that make code more complex without measurably improving it should be reverted. Along the way, the post also points out where the “optimized” code has its own costs and correctness risks, because a faster endpoint that returns broken JSON is not an improvement.

---

Problem: API too slow

Situation

A REST endpoint that returns a user list was reported as slow:

# end-to-end timing of one request; repeat it several times
$ curl -o /dev/null -s -w "time_total: %{time_total}s\n" http://api.example.com/users

A single curl timing includes the network and connection setup, so it only tells you that the request is slow, not where the time goes. It is still worth recording a few runs as the end-to-end starting point.

Requirements

• Goal: p50 latency under 50ms
• Constraint: keep the existing API contract

---

Measurement: benchmark baseline

Simple benchmark harness

// benchmark.cpp
#include <algorithm>
#include <chrono>
#include <iostream>
#include <string>
#include <vector>
using namespace std::chrono;
class Benchmark {
    std::vector<double> samples_;
    
public:
    template<typename Func>
    void run(const std::string& name, Func&& func, int iterations = 100) {
        samples_.clear();
        
        for (int i = 0; i < 10; ++i) {
            func();
        }
        
        for (int i = 0; i < iterations; ++i) {
            auto start = steady_clock::now();
            func();
            auto end = steady_clock::now();
            
            auto duration = duration_cast<microseconds>(end - start).count();
            samples_.push_back(duration / 1000.0); // ms
        }
        
        std::sort(samples_.begin(), samples_.end());
        double p50 = samples_[samples_.size() / 2];
        double p95 = samples_[samples_.size() * 95 / 100];
        double p99 = samples_[samples_.size() * 99 / 100];
        
        std::cout << name << ":\n"
                  << "  p50: " << p50 << "ms\n"
                  << "  p95: " << p95 << "ms\n"
                  << "  p99: " << p99 << "ms\n";
    }
};

Baseline

Run the harness against the unchanged getUserList a few times and write down p50, p95 and p99. That baseline is what every later step is compared against, on the same machine, with the same data and the same build flags.

A few properties make this harness trustworthy enough for comparisons. The ten warm-up runs fill caches and let lazy initialization happen before timing starts. It reports percentiles rather than an average, because latency distributions are skewed and a mean hides the tail that users notice. And it uses steady_clock, which cannot jump when the system clock is adjusted. Its limits are worth knowing too: 100 samples make p99 a single data point, so it is noisy; the compiler may optimize away work whose result is unused (use the result or a DoNotOptimize-style barrier); and it measures the function in isolation, without network, parsing and concurrency. Before trusting a before/after comparison, run each version several times; if two runs of the same build differ by 10%, a 10% “improvement” means nothing. For end-to-end numbers, a load generator such as wrk or k6 against the running server is the complement.

---

Profiling: hotspots with perf

perf

$ g++ -O2 -g -fno-omit-frame-pointer -std=c++17 *.cpp -o server
$ perf record -g ./server
$ perf report

What the profile points at

For code shaped like the one below, three entries sit at the top of the report, in this order:

1. UserManager::findUsersByRole, the per-request scan
2. std::string::string(std::string const&), string copies called from that scan
3. the JSON serialization function

Two details of the command line make this profile readable. -O2 profiles the code you actually ship; profiling a -O0 build shows hotspots that the optimizer would have removed anyway. -g adds symbols and line information without changing the generated code, so function names resolve. -g on perf record records call stacks, so perf report can show that the std::string copy constructor is being called from findUsersByRole; for reliable stacks through optimized code, also build with -fno-omit-frame-pointer or record with --call-graph dwarf. The percentages are shares of samples, which is why they add up to about 100%: they tell you where time goes, not how long anything takes. The practical reading rule is Amdahl’s law: removing a function that takes two-thirds of the samples can at best make the request about 3x faster, while perfecting a part that takes one-eighth can never gain more than about 14%. That arithmetic, applied to the shares in your own report, decides the order of the next three sections.

---

Bottleneck 1: full scan per request

Original code

class UserManager {
    std::vector<User> users_; // 10,000 users
public:
    std::vector<User> findUsersByRole(const std::string& role) {
        std::vector<User> result;
        
        // Costly nested work per user
        for (const auto& user : users_) {
            for (const auto& r : user.roles) {
                if (r == role) {
                    result.push_back(user);
                    break;
                }
            }
        }
        
        return result;
    }
};

Complexity

• n users, ~m roles per user on average
• String comparisons dominate

Strictly, this is O(n·m) per request, not O(n²): it scans all 10,000 users and compares every role string, even though the response contains only about 100 of them. The problem is less the loop shape than doing the full scan on every request for data that changes rarely. Each r == role is a string comparison (cheap when lengths differ, a memcmp when they match), and each match copies a whole User into the result, which is where the separate string-copy hotspot comes from. A linear scan like this is perfectly fine for a few hundred items; it becomes the bottleneck here because the collection is large and the query is frequent.

---

Optimization 1: hash map index

Improved code

class UserManager {
    std::vector<User> users_;
    std::unordered_map<std::string, std::vector<size_t>> roleIndex_;
public:
    void buildIndex() {
        roleIndex_.clear();
        for (size_t i = 0; i < users_.size(); ++i) {
            for (const auto& role : users_[i].roles) {
                roleIndex_[role].push_back(i);
            }
        }
    }
    
    std::vector<User> findUsersByRole(const std::string& role) {
        std::vector<User> result;
        
        if (auto it = roleIndex_.find(role); it != roleIndex_.end()) {
            result.reserve(it->second.size());
            for (size_t idx : it->second) {
                result.push_back(users_[idx]);
            }
        }
        
        return result;
    }
};

Result

Re-run the harness and the profile. The scan no longer dominates: findUsersByRole drops down the report, and the next bottleneck, string copying, becomes visible.

The index trades memory and maintenance for query speed. A lookup is now one hash of the role string plus a walk over exactly the matching users, so the cost is proportional to the result size rather than to the whole collection. The price is that the index must stay correct: every insert, delete or role change in users_ has to update roleIndex_, and storing indices into a std::vector makes that fragile, since erasing an element shifts every later index. If users can be removed, store stable IDs or keep users in a container with stable addresses, and rebuild or patch the index under the same lock that protects the users. In a multi-threaded server, readers of the index also need that lock (or a copy-on-write snapshot), otherwise a concurrent buildIndex() is a data race. What remains of this function’s cost is mostly the User copies in push_back, which the next section addresses.

---

Bottleneck 2: string copies

Issue

for (size_t idx : it->second) {
    result.push_back(users_[idx]); // full User copy
}
struct User {
    std::string id;
    std::string name;
    std::string email;
    std::vector<std::string> roles;
};

Copying a User means copying three strings and a vector of strings. Strings longer than the small-string buffer (typically 15 characters with libstdc++, 22 with libc++) each need a heap allocation, and email addresses usually exceed it, so every copied user costs several allocations and frees. That is exactly what the std::string::string(std::string const&) line in the profile was measuring: not one expensive operation, but a large number of small ones.

---

Optimization 2: string_view and moves

Return references

std::vector<const User*> findUsersByRole(const std::string& role) const {
    std::vector<const User*> result;
    
    if (auto it = roleIndex_.find(role); it != roleIndex_.end()) {
        result.reserve(it->second.size());
        for (size_t idx : it->second) {
            result.push_back(&users_[idx]);
        }
    }
    
    return result;
}

string_view in serialization

std::string serializeUserView(std::string_view id, std::string_view name) {
    std::ostringstream oss;
    oss << "{\"id\":\"" << id << "\","
        << "\"name\":\"" << name << "\"}";
    return oss.str();
}

Returning const User* removes the copies entirely, and it changes the contract: the pointers are valid only while users_ is not modified. Any push_back on the vector may reallocate and leave every returned pointer dangling, so the caller must finish using the result (here, serializing it) before the next write, typically while holding a read lock. That is fine inside one request handler, and dangerous if the result is stored or passed to another thread. std::span or a lightweight view type can make the “borrowed, do not keep” intent clearer than a vector of raw pointers.

The serialization snippet has a correctness problem that no benchmark will reveal: it writes name into the JSON without escaping. A user named Jane "JJ" Doe, or one containing a backslash or a newline, produces invalid JSON, and a crafted name can inject extra fields into the response. Any hand-written JSON output needs escaping of ", \ and control characters; a JSON library such as nlohmann/json, RapidJSON’s Writer, or glaze handles that and is usually fast enough. std::ostringstream is also one of the slower ways to build a string, with locale handling and virtual calls per insertion; std::string::append, std::format (C++20) or fmt::format_to into a buffer are faster.

Result

Re-measure. The std::string copy constructor should largely disappear from the profile, which leaves serialization at the top. A heap profiler such as Heaptrack confirms the change more directly than timing does: the allocation count per request drops.

---

Bottleneck 3: JSON serialization

Issue

std::string toJson(const std::vector<const User*>& users) {
    std::string json = "[";
    for (size_t i = 0; i < users.size(); ++i) {
        json += serializeUser(*users[i]); // repeated +=
        if (i < users.size() - 1) {
            json += ",";
        }
    }
    json += "]";
    return json;
}

Repeated string += reallocates whenever the capacity is exhausted, copying everything built so far each time.

That copying is not quadratic in practice: std::string grows its capacity geometrically (by a factor of about 1.5 to 2 depending on the library), so appending n characters costs amortized O(n) in total, with about log n reallocations. The real costs in this function are elsewhere: serializeUser returns a new temporary string for every user, which is allocated, appended and freed, and the few reallocations of the growing result still copy the whole buffer. Quadratic behavior does appear with json = json + part, which creates a new string on every iteration, and in code that inserts at the front.

---

Optimization 3: threading and reservation

Reserve capacity

std::string toJson(const std::vector<const User*>& users) {
    std::string json;
    json.reserve(users.size() * 100);
    
    json = "[";
    for (size_t i = 0; i < users.size(); ++i) {
        json += serializeUser(*users[i]);
        if (i < users.size() - 1) {
            json += ",";
        }
    }
    json += "]";
    return json;
}

reserve(users.size() * 100) removes the growth reallocations if 100 bytes per user is a reasonable estimate; too low and it still reallocates, too high and it wastes memory per request, so the estimate should come from measuring real responses. A bigger change would be to have serializeUser append into the output string (void serializeUser(const User&, std::string& out)) instead of returning a temporary, which removes one allocation per user. Note that json = "["; after reserve keeps the reserved capacity in all major implementations, although the standard does not strictly promise it; json += '['; avoids the question.

Parallel serialization

#include <thread>
#include <future>
std::string toJsonParallel(const std::vector<const User*>& users) {
    if (users.size() < 100) {
        return toJson(users);
    }
    
    size_t numThreads = std::thread::hardware_concurrency();
    size_t chunkSize = (users.size() + numThreads - 1) / numThreads;
    
    std::vector<std::future<std::string>> futures;
    
    for (size_t i = 0; i < numThreads; ++i) {
        size_t start = i * chunkSize;
        size_t end = std::min(start + chunkSize, users.size());
        
        if (start >= users.size()) break;
        
        futures.push_back(std::async(std::launch::async, [&, start, end]() {
            std::string chunk;
            chunk.reserve((end - start) * 100);
            
            for (size_t j = start; j < end; ++j) {
                chunk += serializeUser(*users[j]);
                if (j < end - 1) chunk += ",";
            }
            return chunk;
        }));
    }
    
    std::string result = "[";
    for (size_t i = 0; i < futures.size(); ++i) {
        result += futures[i].get();
        if (i < futures.size() - 1) result += ",";
    }
    result += "]";
    
    return result;
}

Result

This is the step to be most skeptical about, even if a single-request benchmark shows it helping. std::async(std::launch::async, ...) typically creates a new thread per call, so each request now starts and joins one thread per core, and creating a thread costs far more than serializing a handful of users; those threads also compete with every other request for the same cores. In a single-request benchmark, the machine’s idle cores make the parallel version look good; under real load, where every core is already busy serving other requests, parallelizing within a request mostly adds overhead and can reduce throughput. The threshold users.size() < 100 also means the 100-user case in the benchmark is exactly the first size that takes the parallel path.

If parallel serialization is worth keeping, it belongs on a shared thread pool with a much higher threshold, chosen by measuring under concurrent load (with wrk or similar), and it should be judged on throughput and p99 at the target request rate, not on the latency of one request in isolation. The lambda captures users by reference, which is safe only because every future is joined with get() before the function returns; an exception thrown in one task is rethrown by its get(), but the other tasks keep running until their own get().

---

What each stage removes

Stage-by-stage

Stage	Change	What it removes	New cost or risk
0	Baseline	—	—
1	Role index	Full scan and string compares per request	Index must be kept in sync with every write
2	Pointer views, less copying	Several heap allocations per returned user	Pointers are valid only until users_ changes
3	reserve + parallel JSON	Growth reallocations; serial serialization	Thread creation per request; worse under load

CPU profile shift

Before: findUsersByRole dominated; string copy + JSON next.
After: JSON + I/O + hash lookup more balanced.

A flat profile is the signal to stop. When no single function dominates, further gains require either architectural changes (caching whole responses, pagination, moving work to the database) or many small optimizations each worth a few percent, and each of those adds code to maintain. The original goal was p50 under 50 ms. If the index and the copy removal already meet it, which is what you should check before going further, the parallel step goes beyond the requirement, and that is the right moment to ask whether the extra complexity is worth it.

---

Lessons and more ideas

Takeaways

1. Don’t optimize without measurement
2. Fix algorithmic cost first
3. Remove needless copies (references, moves, string_view)
4. Parallelize last, after serial optimizations

Optimization flow

graph TD
    A[Performance issue] --> B[Measure and profile]
    B --> C{Find bottleneck}
    C --> D[Improve algorithmic complexity]
    D --> E[Memory / copies]
    E --> F[Compiler options]
    F --> G[Multithreading]
    G --> H[Verify and ship]

Tooling

Tool	Use	Example
perf	CPU hotspots	perf record -g ./app && perf report
gprof	Per-function time	g++ -pg ... && ./app && gprof app
Valgrind Callgrind	Call graphs	valgrind --tool=callgrind ./app
Heaptrack	Allocations	heaptrack ./app

perf samples the running program with low overhead and needs no special build, which makes it the default on Linux. gprof requires rebuilding with -pg, adds instrumentation overhead to every function call, and handles multithreaded programs and shared libraries poorly, so it is mostly of historical interest. Callgrind gives exact instruction counts and full call graphs but runs the program many times slower, which distorts anything involving I/O or timing; it is best for comparing two versions of a CPU-bound function. Heaptrack answers a different question, “who allocates how much and how often”, and it would have pointed directly at the User copies in this case. For visualizing perf data, flame graphs (Brendan Gregg’s scripts, or perf script into tools such as speedscope) make the call-stack share much easier to read than the flat report.

---

More ideas

Short TTL cache

class UserManager {
    std::unordered_map<std::string, std::vector<const User*>> cache_;
    std::chrono::steady_clock::time_point cacheTime_;
    
public:
    std::vector<const User*> findUsersByRole(const std::string& role) {
        auto now = std::chrono::steady_clock::now();
        
        if (now - cacheTime_ < std::chrono::seconds(5)) {
            if (auto it = cache_.find(role); it != cache_.end()) {
                return it->second;
            }
        }
        
        auto result = findUsersByRoleImpl(role);
        cache_[role] = result;
        cacheTime_ = now;
        
        return result;
    }
};

This sketch has the classic cache bugs, which is why caching belongs at the end of the list. There is one cacheTime_ for all roles, so a miss for any role refreshes the timestamp and keeps every other role’s entry alive past its five seconds. It caches pointers into users_, so after the vector reallocates, a cache hit returns dangling pointers. It is not thread-safe, although the method will be called from concurrent requests. And nothing invalidates the cache when a user’s roles change, so clients see stale data for up to the TTL, which may or may not be acceptable for the product. A correct version stores a timestamp per entry, caches values or stable IDs, protects the map with a mutex (or uses a concurrent cache), and is invalidated on writes. Since the index already made the query cheap, caching the serialized response per role would save more work than caching the lookup.

The database index (GIN on an array column, as in PostgreSQL) applies when users live in a database rather than in memory; it is the same idea as roleIndex_, maintained by the database. Response compression trades CPU time for transfer time: it helps for large responses over slow networks and costs CPU on every request, so it is usually better handled by a reverse proxy such as nginx than inside the application.

DB index

CREATE INDEX idx_user_roles ON users USING GIN(roles);

Compress responses

// gzip JSON to reduce transfer time
#include <zlib.h>

---

Closing thoughts

Performance work is measure → analyze → optimize → verify, repeated.

1. perf located the real hotspots
2. Indexing removed the largest cost, the per-request scan
3. Removing copies cut allocations
4. Parallel JSON is optional and the step most likely to backfire under load

The algorithm change addresses the largest share of the profile, so it is where most of the gain is to be found, and re-measuring after every step is what tells you which change actually helped on your system.

---

FAQ

Q1. When should we optimize? Only when you have measurable pain. “It feels slow” without numbers usually means more complex code for little gain.

Q2. Should we use -O3? Often -O2 is enough; -O3 can grow code and hurt caches. Measure.

Q3. Thread first? Optimize the serial path first. Parallel slow code is just “fast slow code.”

---

Related Articles

• Profiling C++ Before Optimizing: perf, gprof, Flame Graphs and Finding the Real Bottleneck
• C++ multithreading

---