C++ Benchmarking: steady_clock, Warmup, Statistics and Google Benchmark
Key takeaways
Most hand-written C++ benchmarks measure the wrong thing: an optimized-away loop, a copy included in the timing, or a single noisy run. This guide shows each mistake with real output, a small harness that avoids them, and how Google Benchmark handles the same problems.
Benchmarking versus profiling
A benchmark answers “how long does this piece of code take, and did my change make it faster?” A profiler answers “where does my program spend its time?” You normally need both, in that order: profile first to find the code that matters (see profiling C++ before optimizing), then benchmark that code before and after each change.
Writing a benchmark looks trivial: read the clock, run the code, read the clock again. The difficulty is that a modern compiler and CPU are very good at making code faster in ways that do not happen in your real program. The optimizer deletes work whose result is unused. Caches and branch predictors warm up across repetitions. The OS moves your thread between cores and changes clock frequency. A naive benchmark can report numbers that are off by orders of magnitude, in either direction, and still look perfectly plausible.
All numbers in this article come from g++ 10.3 with -O2 on Windows x86-64. Your absolute timings will differ; the patterns are what matter.
Pick the right clock
#include <chrono>
auto t0 = std::chrono::steady_clock::now();
// code under test
auto t1 = std::chrono::steady_clock::now();
double ms = std::chrono::duration<double, std::milli>(t1 - t0).count();Use std::chrono::steady_clock. It is guaranteed to be monotonic: it never goes backwards and is not adjusted when the system time changes. high_resolution_clock sounds better but is implementation-defined. It is usually an alias of system_clock or steady_clock, and on the toolchain used here it is not steady:
steady is_steady=1, hrc is_steady=0A non-steady clock can jump when NTP adjusts the system time mid-measurement, which produces a negative or huge duration once in a while. See the time_point article for how the clocks differ.
Also note duration<double, std::milli> instead of duration_cast<microseconds>. Casting to an integer duration truncates, and dividing an integer count by the number of iterations truncates again. This is the harness many tutorials show:
auto total = duration_cast<microseconds>(end - start);
return total.count() / iterations; // integer divisionFor std::vector<int> v(1000) repeated 1000 times, it prints:
Average: 0usThe work took far less than a microsecond per iteration, so the average truncated to zero. Keep durations in floating point until you format them.
Pitfall 1: the optimizer deletes your benchmark
static long long sumTo(int n) {
long long s = 0;
for (int i = 0; i < n; ++i) s += (long long)i * i;
return s;
}
auto t0 = clk::now();
for (int r = 0; r < 1000; ++r) sumTo(1000000); // result discarded
auto t1 = clk::now();
volatile long long sink = 0;
for (int r = 0; r < 1000; ++r) sink = sumTo(1000000 + (r & 1)); // result used
auto t2 = clk::now();-O2: discarded: 0 us kept: 223958 us
-O0: discarded: 889904 us kept: 901222 usAt -O2, the first loop costs nothing, because sumTo has no side effects and its result is unused, so the compiler removed all 1000 calls. At -O0 both loops run, but -O0 numbers are meaningless for performance work, because nobody ships unoptimized code. The only fix is to keep optimization on and make the result observable.
Writing to a volatile works but is blunt: it forces a store on every iteration and can change the code under test. Google Benchmark’s benchmark::DoNotOptimize(x) is the standard tool. With GCC and Clang it is essentially an empty inline asm statement that claims to read x, so the compiler must compute x but emits no extra instructions. A minimal version you can use without the library:
template <class T>
inline void doNotOptimize(T const& value) {
asm volatile("" : : "r,m"(value) : "memory"); // GCC/Clang only
}The opposite mistake is just as common. If the input is a compile-time constant, the compiler may compute the whole result at compile time and the benchmark measures nothing. Derive inputs from data the compiler cannot see: random data generated at run time, command-line arguments, or values passed through doNotOptimize first.
Pitfall 2: timing the setup instead of the work
Suppose you want to compare std::sort with std::stable_sort. Each sample needs a fresh unsorted copy of the data, because sorting already-sorted data is a different (much faster) workload. Copying a million ints is not free, so the copy must happen outside the timed region:
template <class Setup, class Body>
Stats measure(Setup setup, Body body, int warmup = 5, int samples = 30) {
using clk = std::chrono::steady_clock;
std::vector<double> ms;
for (int i = 0; i < warmup + samples; ++i) {
auto input = setup(); // not timed
auto t0 = clk::now();
body(input);
auto t1 = clk::now();
doNotOptimize(input);
if (i >= warmup) ms.push_back(std::chrono::duration<double, std::milli>(t1 - t0).count());
}
return summarize(ms);
}The warmup iterations run the same code but are discarded. They let the instruction cache, branch predictors, page mappings and CPU frequency settle, so the first few slow samples do not skew the results.
Pitfall 3: trusting one number
Every measurement is noisy: interrupts, other processes, frequency scaling, thermal limits. Collect many samples and summarize them:
struct Stats { double min, median, p95, mean, stddev; };
Stats summarize(std::vector<double> s) {
std::sort(s.begin(), s.end());
auto pct = [&](double p) { return s[static_cast<size_t>(p * (s.size() - 1))]; };
double mean = std::accumulate(s.begin(), s.end(), 0.0) / s.size();
double var = 0;
for (double x : s) var += (x - mean) * (x - mean);
return {s.front(), pct(0.5), pct(0.95), mean, std::sqrt(var / (s.size() - 1))};
}Sorting one million random ints, two runs of the same program:
sort min 44.23 median 47.41 p95 59.87 mean 49.28 sd 5.17 ms
stable_sort min 51.34 median 52.41 p95 53.59 mean 52.60 sd 0.76 ms
sort sorted min 8.48 median 9.10 p95 11.84 mean 9.60 sd 1.16 ms
sort min 43.91 median 49.75 p95 60.23 mean 50.58 sd 4.94 ms
stable_sort min 51.25 median 52.98 p95 59.93 mean 54.51 sd 4.24 ms
sort sorted min 8.34 median 8.64 p95 10.94 mean 8.91 sd 0.87 msSeveral lessons are visible in this small table:
- The median is more stable than the mean. Noise only ever makes a run slower, so outliers pull the mean up. The median of
stable_sortmoved by about 1% between runs; its mean moved by more than 3% because one run had a few slow samples. - Tails vary more than centers. The p95 of
stable_sortwent from 53.6 ms to 59.9 ms between runs. If you care about tail latency, you need many more samples than 30. - The minimum is a useful “best achievable” figure for pure CPU work, but it hides variance that a real program would feel.
- Input matters as much as algorithm. Sorting already-sorted data was about five times faster than sorting random data. A benchmark on unrepresentative input answers a question nobody asked.
Also, std::sort beat std::stable_sort here, but only by 6 to 10% on the median depending on the run, and the ranges overlap in individual samples. Before claiming one version is faster, rerun the whole benchmark several times and check that the difference is larger than the run-to-run variation.
Complexity guarantees are a separate question from measured speed. std::sort is O(N log N) comparisons in the worst case (required since C++11). std::stable_sort is O(N log N) when it can allocate a buffer and O(N log² N) when it cannot. std::partial_sort is about O(N log K) for the top K elements, and std::nth_element is linear on average. Those bounds tell you how the cost grows; only measurement tells you the constants on your hardware.
Pitfall 4: the measurement is not the workload
Microbenchmarks run the same small piece of code thousands of times in a tight loop with hot caches. Real programs interleave that code with everything else, so the caches are often cold. A lookup table that looks free in a benchmark can be slower in production, because in production its cache lines have been evicted by the time it is used.
A failure mode I have run into more than once: a hash map change shows a clear win in a microbenchmark that inserts and looks up the same few thousand keys, then makes no measurable difference, or makes things slightly worse, in the application. The benchmark’s whole data set fit in L2 cache, and the real one did not. Check a candidate optimization against your real data sizes, and confirm it with an end-to-end measurement or a profiler before keeping the added complexity. The cache optimization article covers why data size changes results so sharply.
Reducing noise on the machine
- Build like production. Use
-O2or-O3, plus-DNDEBUGif release builds use it, and the same-marchflags you ship with. - Keep the machine quiet. Close browsers and IDE indexers. On laptops, plug in power and pick a fixed performance mode.
- Pin the thread (
taskset -c 2 ./benchon Linux) to avoid migrations between cores with different cache contents or, on hybrid CPUs, different core types. - Watch frequency scaling. Turbo boost and thermal throttling change clock speed during a run. For comparisons, run both versions back to back, several times, in alternating order.
Google Benchmark
For anything beyond a quick check, Google Benchmark handles most of the above for you: it picks the iteration count automatically, provides DoNotOptimize and ClobberMemory, supports parameterized inputs, and reports statistics across repetitions.
#include <benchmark/benchmark.h>
#include <vector>
static void BM_PushBack(benchmark::State& state) {
for (auto _ : state) {
std::vector<int> v;
for (int i = 0; i < state.range(0); ++i) v.push_back(i);
benchmark::DoNotOptimize(v.data());
benchmark::ClobberMemory();
}
state.SetItemsProcessed(state.iterations() * state.range(0));
}
BENCHMARK(BM_PushBack)->Range(8, 8 << 10);
static void BM_PushBackReserved(benchmark::State& state) {
for (auto _ : state) {
std::vector<int> v;
v.reserve(state.range(0));
for (int i = 0; i < state.range(0); ++i) v.push_back(i);
benchmark::DoNotOptimize(v.data());
benchmark::ClobberMemory();
}
state.SetItemsProcessed(state.iterations() * state.range(0));
}
BENCHMARK(BM_PushBackReserved)->Range(8, 8 << 10);
BENCHMARK_MAIN();DoNotOptimize(v.data()) keeps the vector’s contents alive, and ClobberMemory() tells the compiler that memory may have been read, so the stores into the vector cannot be dropped. Range(8, 8 << 10) runs each benchmark at several sizes from 8 to 8192, which shows how the cost of reallocation changes with size instead of giving a single number. SetItemsProcessed adds an items-per-second column, which is easier to compare across sizes than raw time.
Building and running:
git clone https://github.com/google/benchmark.git
cmake -S benchmark -B benchmark/build -DCMAKE_BUILD_TYPE=Release -DBENCHMARK_DOWNLOAD_DEPENDENCIES=on
cmake --build benchmark/build --config Release
cmake --install benchmark/build --prefix "$HOME/.local"
g++ -std=c++17 -O2 bench.cpp -I"$HOME/.local/include" -L"$HOME/.local/lib" -lbenchmark -lpthread -o bench
./bench --benchmark_repetitions=10 --benchmark_report_aggregates_only=trueWith repetitions enabled, the report includes mean, median and standard deviation for each benchmark, so you can apply the same reasoning as above. Pay attention to warnings printed at startup. If the library itself was built in Debug mode, it says so, and if CPU frequency scaling is enabled it warns that measurements may be noisy. Both are worth fixing before trusting the numbers.
When a benchmark needs per-iteration setup, state.PauseTiming() and state.ResumeTiming() exist, but they have a noticeable overhead of their own. For small bodies, the overhead can dominate the measurement. Prefer preparing a batch of inputs before the loop, or measuring setup and work together and subtracting a benchmark of setup alone.