Deploying a C++ Service to Production: glibc Floors, Debug Symbols, systemd and SIGTERM [#50-5]

“It works on my machine” is a C++ problem first

Deploying a Python or Node service is mostly about the runtime being present on the host. Deploying C++ is about the binary itself: which shared libraries it expects, which glibc symbol versions it was linked against, whether it contains the symbols you need to read its core dump, and whether it reacts to the signals the process supervisor sends. None of this shows up in unit tests, and all of it shows up on the first production deploy.

This article walks through the decisions in the order they bite: making the binary start on the target, making crashes debuggable, keeping the process running under systemd, and shutting it down cleanly under Kubernetes. Container image construction itself (layer caching, vcpkg and Conan in Docker, distroless bases) is covered in #40-3; here Docker appears only where it changes those decisions.

The running example is the chat gateway from #31-1 and #50-1: a long-lived Asio server holding thousands of TCP connections, which makes graceful shutdown matter more than it does for a stateless HTTP handler.

---

Step 1: make the binary start on the target

The glibc floor

glibc versions its symbols. When you link on a machine with glibc 2.35, calls like memcpy or pthread_create may bind to GLIBC_2.34-tagged versions. Running that binary on a host with glibc 2.31 fails immediately:

./chat-gateway: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found

glibc is backward compatible (old binaries run on new systems) but not forward compatible. So the rule is: the build environment defines the oldest system you can run on. You can check what a binary needs before shipping it:

# Highest glibc symbol version the binary requires
objdump -T chat-gateway | grep -o 'GLIBC_[0-9.]*' | sort -Vu | tail -1

# Same idea for libstdc++
objdump -T chat-gateway | grep -o 'GLIBCXX_[0-9.]*' | sort -Vu | tail -1

# Shared libraries it will look for at runtime
ldd chat-gateway

Putting the first command in CI, and failing the build when the version exceeds your oldest supported target, turns a deploy-time outage into a red build.

I learned this rule the usual way: a service built on a developer’s freshly upgraded workstation, copied to a server running the previous LTS release, and refusing to start with exactly the error above. Nothing in the code had changed; the compiler host had. Since then I treat “which distro version do we build on” as a deployment decision, written down next to the list of target hosts, not a detail of whoever set up CI.

Static, dynamic, or in between

Option	What it removes	What to watch
Fully dynamic (default)	Nothing; smallest binary	Every .so and its version must exist on the target
-static-libstdc++ -static-libgcc	Dependency on the target’s libstdc++	Still needs a compatible glibc; do not pass C++ objects across a boundary to a plugin built with a different libstdc++
Fully static with glibc	All .so files	NSS (getaddrinfo, getpwnam) still loads shared modules at runtime; the linker warns about it; dlopen is effectively unusable
Fully static with musl	All .so files, reliably	Different allocator and DNS resolver behavior; benchmark allocation-heavy code and test resolution in your environment

For a server that runs in a container you control, the pragmatic answer is usually dynamic linking inside a runtime image that uses the same base as the build image. The glibc match is then guaranteed by construction, and security updates to OpenSSL arrive through the base image instead of a rebuild. Partial static linking of libstdc++ is a good fit when you ship a single binary to hosts you do not control.

A multi-stage build that keeps the floor fixed

FROM ubuntu:22.04 AS build
RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential cmake ninja-build libboost-dev libssl-dev \
    && rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY . .
RUN cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo \
    && cmake --build build \
    && cd build \
    && objcopy --only-keep-debug chat-gateway chat-gateway.debug \
    && strip --strip-debug chat-gateway \
    && objcopy --add-gnu-debuglink=chat-gateway.debug chat-gateway

FROM ubuntu:22.04
RUN apt-get update && apt-get install -y --no-install-recommends \
        libssl3 ca-certificates \
    && rm -rf /var/lib/apt/lists/* \
    && useradd --system --uid 10001 app
COPY --from=build /src/build/chat-gateway /usr/local/bin/chat-gateway
USER app
EXPOSE 5555 8080
ENTRYPOINT ["/usr/local/bin/chat-gateway"]

Three details in that file matter more than they look:

• Same base image in both stages. That is what pins the glibc floor.
• ENTRYPOINT in exec (JSON array) form. The shell form, ENTRYPOINT /usr/local/bin/chat-gateway, runs the program under /bin/sh -c, which becomes PID 1 and does not forward SIGTERM. A typo in the array form, such as CMD [./myapp] without quotes, is not valid JSON, so Docker silently falls back to shell form with a nonsense command.
• No curl in the runtime image, so a HEALTHCHECK CMD curl ... would always fail. Either install the tool you probe with, or let the orchestrator probe over the network (Kubernetes does not use Docker’s HEALTHCHECK at all).

---

Step 2: make crashes debuggable

A C++ service will eventually crash in production, and the difference between a ten-minute and a ten-day investigation is whether you can load the core dump with symbols. Shipping -g binaries to production makes them large; shipping stripped binaries without keeping the symbols makes cores nearly useless.

The standard answer is split debug info, which the Dockerfile above already does:

objcopy --only-keep-debug chat-gateway chat-gateway.debug   # symbols only
strip --strip-debug chat-gateway                             # ship this
objcopy --add-gnu-debuglink=chat-gateway.debug chat-gateway  # record the pairing

readelf -n chat-gateway | grep 'Build ID'                    # unique ID of this build

Archive chat-gateway.debug from CI, keyed by the build ID (GCC and Clang emit one by default on most Linux distributions via --build-id). When a core arrives, place the debug file next to the binary or under /usr/lib/debug/.build-id/, and gdb chat-gateway core will find it. RelWithDebInfo is a sensible build type because it keeps optimization while generating full debug info to split off.

Getting the core file is the other half:

• On a VM with systemd, set LimitCORE=infinity in the unit and let systemd-coredump collect cores. coredumpctl list and coredumpctl gdb <pid> then do the rest.
• In containers, kernel.core_pattern is a host-wide setting. If the host pipes cores to systemd-coredump or apport, the core ends up on the host, not in the container, and the container’s ulimit -c must also allow it (docker run --ulimit core=-1). On managed Kubernetes you often cannot change the host setting, which is a good reason to also log a symbolized backtrace from a crash handler or use a crash-reporting library that writes minidumps.

Enabling core dumps, reading backtraces and ASan are covered step by step in #49-1.

---

Step 3: keep it running with systemd

On plain VMs, systemd is the process supervisor, log collector and resource limiter in one. A unit for the gateway:

# /etc/systemd/system/chat-gateway.service
[Unit]
Description=Chat gateway (C++)
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
Type=simple
User=chat
ExecStart=/opt/chat/bin/chat-gateway --config /etc/chat/gateway.toml
Restart=on-failure
RestartSec=2s
KillSignal=SIGTERM
TimeoutStopSec=30s

LimitNOFILE=65536
LimitCORE=infinity
MemoryMax=2G

NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
StateDirectory=chat

[Install]
WantedBy=multi-user.target

What each group is for, and the mistakes it prevents:

• Wants= plus After=network-online.target. After=network.target alone does not wait for addresses to be configured, and a server that binds to a specific IP at startup fails intermittently on boot.
• Restart=on-failure with a start limit. A crash restarts the process after 2 seconds; five failures within 60 seconds put the unit into the failed state instead of looping forever. StartLimitIntervalSec and StartLimitBurst belong in [Unit] on current systemd; older examples put StartLimitInterval in [Service], which is deprecated. Alert on the failed state, since it means the service is down and systemd has stopped trying.
• LimitNOFILE. Every TCP connection is a file descriptor. The default soft limit of 1024 caps a chat gateway at roughly a thousand clients, and accept starts failing with EMFILE, which the accept loop must handle without spinning.
• MemoryMax (the modern name; MemoryLimit is the deprecated cgroup-v1 spelling). When exceeded, the kernel’s OOM killer ends the process and systemd restarts it, which is better than the whole host swapping.
• Sandboxing. ProtectSystem=strict makes the filesystem read-only except for directories you grant, such as the one created by StateDirectory=. It costs nothing for a network server and limits the damage of a memory-safety bug.

Log to stdout and stderr. The journal captures them with timestamps and the unit name, journalctl -u chat-gateway reads them, and the same binary logs correctly in a container without a file-path configuration. If you want structured logs, write one JSON object per line and let the collector parse it.

---

Step 4: shut down cleanly on SIGTERM

Both systemd and Kubernetes stop a service the same way: send SIGTERM, wait a grace period (TimeoutStopSec, or terminationGracePeriodSeconds, 30 seconds by default in Kubernetes), then send SIGKILL. Everything your process does between those two signals is its chance to finish in-flight work.

Handling the signal

A classic std::signal handler may only do async-signal-safe things, such as setting a lock-free std::atomic<bool>, and then something has to poll that flag. In an Asio program there is a cleaner option: asio::signal_set turns the signal into an ordinary completion handler on the event loop.

int main() {
    asio::io_context io;
    Server server(io, 5555);                 // accept loop + sessions, as in #31-1
    Health health(io, 8080);                 // /livez and /readyz endpoints
    asio::steady_timer deadline(io);

    asio::signal_set signals(io, SIGINT, SIGTERM);
    signals.async_wait([&](boost::system::error_code ec, int) {
        if (ec) return;
        health.set_ready(false);             // 1. stop receiving new traffic
        server.stop_accepting();             // 2. close the listening socket
        server.close_all([&] {               // 3. let sessions flush and close...
            deadline.cancel();               //    ...and exit as soon as they are done
            health.stop();
        });
        deadline.expires_after(std::chrono::seconds(25));   // below the grace period
        deadline.async_wait([&](boost::system::error_code ec) {
            if (!ec) io.stop();              // 4. drain took too long: give up
        });
    });

    health.set_ready(true);
    io.run();                                // returns when all work is finished
}

The shape matters more than the class names: fail readiness first, stop accepting, drain existing work, and keep your own deadline shorter than the supervisor’s so you exit on your terms instead of being killed mid-write. For a chat gateway, “drain” also means spreading disconnects out, so that every client does not reconnect to the remaining gateways in the same second (see the reconnect section of #50-1).

The PID 1 trap

In a container, your process is often PID 1. The kernel treats PID 1 specially: signals for which it has not installed a handler are ignored rather than taking their default action. A C++ program without a SIGTERM handler, which would die instantly as a normal process, simply ignores SIGTERM as PID 1. The container then sits for the full grace period and is SIGKILLed, on every single deploy.

This was the most confusing deploy problem I ran into with C++ services in containers: the same binary stopped instantly under systemd and took exactly 30 seconds to stop in Kubernetes, and nothing appeared in the logs because the process never noticed anything. Installing a handler fixes it, and so does running under a minimal init such as tini or docker run --init, which also reaps zombie processes if you spawn children.

---

Step 5: Kubernetes probes and termination

spec:
  terminationGracePeriodSeconds: 45
  containers:
  - name: chat-gateway
    image: registry.example.com/chat-gateway:3f2c1ab   # git SHA, never :latest
    ports:
    - containerPort: 5555
    - containerPort: 8080
    resources:
      requests: { cpu: "500m", memory: "512Mi" }
      limits:   { memory: "2Gi" }
    startupProbe:
      httpGet: { path: /livez, port: 8080 }
      periodSeconds: 2
      failureThreshold: 30          # up to 60 s to finish startup
    livenessProbe:
      httpGet: { path: /livez, port: 8080 }
      periodSeconds: 10
      failureThreshold: 3
    readinessProbe:
      httpGet: { path: /readyz, port: 8080 }
      periodSeconds: 5
    lifecycle:
      preStop:
        exec: { command: ["sleep", "5"] }

The reasoning behind each piece:

• Liveness answers “is the event loop alive?” and nothing else. Serve /livez from the same io_context as the real work, so a deadlocked loop fails it. Never check the database here: when the database has an outage, a liveness check that depends on it restarts every pod at once, which turns one incident into two.
• Readiness answers “should traffic come here?” It is the endpoint you flip to false at the start of shutdown, and it can reflect dependencies that make serving pointless.
• Startup probe covers slow initialization (loading a large config or warming a cache) without giving liveness a long initialDelaySeconds that also delays detection of real hangs later.
• preStop: sleep 5. Removing a terminating pod from Service endpoints and sending SIGTERM happen in parallel. Without the short sleep, the process can close its listener while the load balancer is still routing new connections to it. The sleep requires a sleep binary in the image; distroless images need a different approach, such as delaying the listener close inside the program. The sleep counts against terminationGracePeriodSeconds, so set the grace period to cover sleep plus drain.
• Memory limit but no CPU limit. A memory limit protects the node. CPU limits throttle through CFS quotas, which shows up as latency spikes in a latency-sensitive server; many teams set only a CPU request. If you do set a CPU limit, remember that std::thread::hardware_concurrency() knows nothing about the CFS quota; it reports the CPUs the process may be scheduled on, which is typically every core on the node, so a thread pool sized from it can be several times larger than the CPU you are allowed to use. Take the pool size from configuration.

Rolling updates with maxUnavailable: 0 and maxSurge: 1 add a new pod before removing an old one. Blue-green and canary rollouts, and how to choose between them, are covered in Kubernetes deployment strategies.

---

Step 6: make releases traceable

• Tag images with the git SHA, and deploy by that tag. :latest makes “what is running?” unanswerable and rollback a guess.
• Embed the version in the binary. Pass -DGIT_SHA=... from CMake and print it at startup and on /livez. When a core dump arrives, the SHA leads to the exact source and the build ID leads to the exact debug file.
• Run the sanitizers in CI, not in production. ASan and TSan builds catch the memory and threading bugs that would otherwise surface as the crashes in Step 2; their runtime overhead makes them a poor fit for production traffic.
• Keep the glibc check from Step 1 in the pipeline, so changing the build image cannot silently raise the minimum supported host.

Previous: Game engine basics (#50-3) Next: Message queues (#50-7)

---

Related Articles

• C++ DevContainer and Docker: reproducible toolchains and multi-stage builds (#40-3)
• Segmentation faults and core dumps: GDB and LLDB debugging (#49-1)
• Kubernetes deployment strategies
• Chat server architecture: scaling, pub/sub, reconnects (#50-1)