C++ Debugging: GDB, LLDB, Sanitizers, Leaks, and Multithreaded Bugs

Introduction: “It crashes in production and I can’t reproduce it.”

Debugging problems encountered in practice

In real C++ development, we encounter this problem:

• Production Crash — It works fine in the development environment, but when deployed, it crashes randomly.
• Memory leak — Memory usage continues to increase for 3 days, no idea where the leak is coming from
• Data Race — Strange values sometimes appear in a multi-threaded environment.
• Poor performance — There is a bottleneck in a specific function, but it is difficult to find without a profiler.
• Iterator Invalidation — Crash during container traversal, unable to trace where it was modified.

This article covers advanced GDB techniques, utilizing Sanitizer, multi-threaded debugging, and production environment debugging based on real-world scenarios.

Goal:

• GDB/LLDB advanced techniques (watchpoint, conditional breakpoint, core dump analysis)
• Full use of Sanitizer (ASan, TSan, UBSan, MSan)
• Memory leak tracking (Valgrind, Heaptrack, practical patterns)
• Multi-threaded debugging (data race, deadlock detection)
• Production Debugging (logging, core dump, remote debugging)
• Frequent mistakes and solutions
• Production Pattern

Required Environment: C++17 or higher, GDB 8.0+, Clang/GCC with Sanitizers

---

Table of Contents

1. Problem scenario: Debugging situations encountered in practice
2. GDB/LLDB Advanced Techniques
3. Full use of Sanitizer
4. Memory Leak Tracking
5. Multithread Debugging
6. Debugging the production environment
7. Complete debugging workflow
8. Frequently occurring mistakes and solutions
9. Best practices/best practices
10. Production Patterns
11. Organization and checklist

---

1.Problem Scenario: Debugging situations encountered in practice

Scenario 1: “Crashes only in production”

Situation: Works normally in development environment, but random crashes after deployment to production
Symptom: Segmentation fault, core dump generation
Cause: Bug hidden by release build optimization (uninitialized variable, UB)
→ Core dump analysis, UB detection required with UBSan

Scenario 2: “Memory keeps growing”

Situation: After the server operates for 72 hours, memory increases from 8GB to 14GB.
Symptoms: Slow memory leak, unrecoverable until restart.
Cause: shared_ptr circular reference, object not removed from container
→ Track leaks with Valgrind, Heaptrack, and ASan

Scenario 3: “Sometimes strange values come out in multithreading”

Situation: When run 10 times, incorrect results are output 1 or 2 times.
Symptoms: data race, non-deterministic behavior
Cause: Missing shared variable synchronization, lockless access.
→ Race detection with TSan, check status of each thread with GDB

Related article: Learn the basic concepts of thread programming at Multithreading Basics.

Scenario 4: “Crash while traversing the container”

Situation: Crash after calling erase during vector traversal
Symptoms: iterator invalidation, segmentation fault
Cause: Not updating iterator after erase
→ Track container modification points with GDB watchpoints

Scenario 5: “A deadlock occurs”

Situation: Multithreaded server occasionally hangs, CPU utilization is 0%
Symptom: All threads are waiting for lock.
Cause: Circular lock wait (A→B, B→A)
→ Check the stack for each thread with GDB and analyze lock order with TSan.

Scenario 6: “It only crashes on release builds”

Situation: Debug build is normal, Release build crashes.
Symptom: Exposure of hidden bugs due to optimization
Cause: Uninitialized variable, undefined behavior (UB)
→ UB detection with UBSan, intermediate optimization test with -O1

Scenario 7: “It only crashes on certain inputs”

Situation: Normal input is ok, only crashes on certain inputs (empty string, large number, etc.)
Symptom: Missing boundary condition (edge case) handling
Cause: Insufficient input validation, missing array range check
→ Automatic test case generation with fuzzing (AFL, libFuzzer)

“mermaid flowchart TB subgraph Problems[“Practical Debugging Problems”] P1[Production Crash] P2[Memory Leak] P3[Data Race] P4[invalidate iterator] P5[Deadlock] end subgraph Tools[“Debugging Tools”] T1[GDB/LLDB + core dump] T2[Valgrind/ASan] T3[TSan] T4[Watchpoint] T5[Thread Analysis] end P1 —> T1 P2 —> T2 P3 —> T3 P4 —> T4 P5 —> T5

---

## <a name="gdb-advanced"></a>2.GDB/LLDB Advanced Techniques

### Review of basic usage

```cpp
#include <iostream>
#include <vector>

int buggyFunction(int x) {
int*ptr = nullptr;
if (x > 10) {
ptr = new int(x);
}
return *ptr;// Crash if x <= 10
}

int main() {
std::cout << buggyFunction(5) << std::endl;
return 0;
}

# Compile (including debug symbols)
g++ -g -O0 buggy.cpp -o buggy

# Start GDB
gdb ./buggy

# Basic commands
(gdb) break buggyFunction # Breakpoint in function
(gdb) run # run
(gdb) next # next line (skip to function)
(gdb) step # next line (into function)
(gdb) print ptr # print variable
(gdb) backtrace # stack trace
(gdb) continue # Continue execution

Watchpoint: Track variable changes

Problem: “I don’t know when and where this variable changes”

#include <iostream>
#include <thread>
#include <vector>

int global_counter = 0;

void increment() {
for (int i = 0; i < 1000; ++i) {
++global_counter;// I want to track where things change
}
}

int main() {
std::vector<std::thread> threads;
for (int i = 0; i < 10; ++i) {
threads.emplace_back(increment);
}
for (auto& t : threads) {
t.join();
}
std::cout << "Counter: " << global_counter << std::endl;
return 0;
}

Caution: The number of hardware watchpoints is limited per CPU, so monitoring entire large structures may be slow or fail.

# Use GDB Watchpoint
(gdb) break main
(gdb)run
(gdb) watch global_counter # Stop when variable changes
(gdb)continue
# Stop whenever global_counter changes
(gdb) backtrace # Check which function made the change

Conditional Breakpoint: Conditional Breakpoint

Problem: “I want to stop only under certain conditions out of 1000 loops.”

#include <iostream>
#iinclude <vector>

int main() {
std::vector<int> data(1000);
for (int i = 0; i < 1000; ++i) {
data[i] = i * 2;
// I only want to check if i is 500
}
return 0;
}

# Conditional breakpoint
(gdb) break 7 if i == 500 # break only when i is 500
(gdb)run

# Complex conditions
(gdb) break myFunction if ptr == nullptr && x > 100

# Change conditions
(gdb) condition 1 i == 750 # Change the condition of breakpoint 1

Caution: If local variables only use registers due to some optimizations, conditional expressions may not work as expected, so -O0 is recommended.

Core dump analysis

If a crash occurs in production:

# Enable core dump
ulimit -c unlimited

# Run program (create core file in case of crash)
./myapp
# Segmentation fault (core dumped)

# Debugging with core dumps
gdb ./myapp core

# Check crash point
(gdb)backtrace
(gdb) frame 0
(gdb) info locals # Check local variables
(gdb) print this # Check object status

Multi-threaded debugging

#include <iostream>
#include <thread>
#include <vector>
#include <mutex>

std::mutex mtx;
int shared_data = 0;

void worker(int id) {
for (int i = 0; i < 100; ++i) {
std::lock_guard<std::mutex> lock(mtx);
++shared_data;
std::cout << "Thread " << id << ": " << shared_data << std::endl;
}
}

int main() {
std::vector<std::thread> threads;
for (int i = 0; i < 4; ++i) {
threads.emplace_back(worker, i);
}
for (auto& t : threads) {
t.join();
}
return 0;
}

# Multithreaded debugging
(gdb) info threads # list all threads
(gdb) thread 2 # Switch to thread 2
(gdb) backtrace # Stack of that thread
(gdb) thread apply all backtrace # Stack output for all threads

# Breakpoints per thread
(gdb) break worker thread 3 # break only on thread 3

Caution: In a non-blocking/fiber environment, the thread number may fluctuate, so it is recommended to use it with a repro script.

GDB script automation

# Create debug.gdb file
break main
run
print argc
print argv[0]
continue

# Run script
gdb -x debug.gdb ./myapp

# or within GDB
(gdb) source debug.gdb

Note: Relative paths are relative to GDB’s current working directory, so place the script in the build directory or use an absolute path.

Pretty Printing (STL container)

#include <vector>
#include <map>

int main() {
std::vector<int> vec = {1, 2, 3, 4, 5};
std::map<std::string, int> m = {{"a", 1}, {"b", 2}};
return 0;
}

# Beautiful STL output in GDB
(gdb) print vec
# $1 = std::vector of length 5, capacity 5 = {1, 2, 3, 4, 5}

(gdb) print m
# $2 = std::map with 2 elements = {["a"] = 1, ["b"] = 2}

# Enable pretty printer (if not present)
# Add to ~/.gdbinit:
# python
# import sys
# sys.path.insert(0, '/usr/share/gcc/python')
# from libstdcxx.v6.printers import register_libstdcxx_printers
# register_libstdcxx_printers(None)
# end

LLDB (macOS default debugger)

# LLDB basic usage (similar to GDB)
lldb ./myapp

# Comparison of main commands
(lldb) breakpoint set --name main # GDB: break main
(lldb) run # GDB: run
(lldb) next # GDB: next
(lldb) step # GDB: step
(lldb) print variable # GDB: print variable
(lldb)bt#GDB:backtrace
(lldb) continue # GDB: continue

#Watchpoint
(lldb) watchpoint set variable global_counter
(lldb) watchpoint list

---

3.Take full advantage of Sanitizer

AddressSanitizer (ASan): Memory error detection

Detectable bugs:

• Use-after-free -Heap buffer overflow
• Stack buffer overflow -Use-after-return -Memory leaks

#include <iostream>

int main() {
int* arr = new int[10];
delete[] arr;

// Use-after-free
std::cout << arr[0] << std::endl;// 💥 Detected by ASan

return 0;
}

# ASan-enabled compilation
g++ -fsanitize=address -g -O1 use_after_free.cpp -o test
# or Clang
clang++ -fsanitize=address -g -O1 use_after_free.cpp -o test

# run
./test

# Example output:
# =================================================================================
# ==12345==ERROR: AddressSanitizer: heap-use-after-free on address 0x...
# READ of size 4 at 0x... thread T0
# #0 0x... in main use_after_free.cpp:7
#...
# freed by thread T0 here:
# #0 0x... in operator delete
# #1 0x... in main use_after_free.cpp:5

Heap Buffer Overflow Detection:

#include <iostream>

int main() {
int* arr = new int[10];

// Buffer overflow
arr[10] = 42;// 💥 ASan detects (index out of range)

delete[] arr;
return 0;
}

Stack Buffer Overflow Detection:

#include <cstring>

int main() {
char buffer[10];
strcpy(buffer, "This is too long!");// 💥 Detected by ASan
return 0;
}

ThreadSanitizer (TSan): Data race detection

#include <iostream>
#include <thread>

int global_counter = 0;// unprotected shared variable

void increment() {
for (int i = 0; i < 100000; ++i) {
++global_counter;// 💥 TSan detects data race
}
}

int main() {
std::thread t1(increment);
std::thread t2(increment);

t1.join();
t2.join();

std::cout << "Counter: " << global_counter << std::endl;
return 0;
}

# Compile with TSan enabled
g++ -fsanitize=thread -g -O1 data_race.cpp -o test -pthread

# run
./test

# Example output:
# ==================
# WARNING: ThreadSanitizer: data race (pid=12345)
# Write of size 4 at 0x... by thread T2:
# #0 increment() data_race.cpp:7
# Previous write of size 4 at 0x... by thread T1:
# #0 increment() data_race.cpp:7

Modified version (using mutex):

#include <iostream>
#include <thread>
#include <mutex>

int global_counter = 0;
std::mutex mtx;

void increment() {
for (int i = 0; i < 100000; ++i) {
std::lock_guard<std::mutex> lock(mtx);
++global_counter;// ✅ Now safe
}
}

int main() {
std::thread t1(increment);
std::thread t2(increment);

t1.join();
t2.join();

std::cout << "Counter: " << global_counter << std::endl;
return 0;
}

UndefinedBehaviorSanitizer (UBSan): Detect undefined behavior

#include <iostream>
#include <limits>

int main() {
// integer overflow
int max = std::numeric_limits<int>::max();
int overflow = max + 1;// 💥 Detected by UBSan

// divide by 0
int x = 10;
int y = 0;
int result = x/y;// 💥 Detected by UBSan

// dereference null pointer
int*ptr = nullptr;
int value = *ptr;// 💥 Detected by UBSan

// bad cast
class Base { virtual ~Base() {} };
class Derived : public Base {};
Base* b = new Base();
Derived* d = static_cast<Derived*>(b);  // 💥 UBSan catches this

return 0;
}

# Compile with UBSan enabled
g++ -fsanitize=undefined -g -O1 ub.cpp -o test

# run
./test

# Example output:
# ub.cpp:7:20: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'
# ub.cpp:11:19: runtime error: division by zero

MemorySanitizer (MSan): Detect uninitialized memory

#include <iostream>

int main() {
int x;// not initialized

if (x > 10) { // 💥 MSan detects
std::cout << "x is large" << std::endl;
}

int* arr = new int[10];// not initialized
std::cout << arr[0] << std::endl;// 💥 MSan detects

delete[] arr;
return 0;
}

# Enable MSan (only Clang supported)
clang++ -fsanitize=memory -g -O1 uninit.cpp -o test

# run
./test

Use Sanitizer Combination

# ASan + UBSan combination (recommended)
g++ -fsanitize=address,undefined -g -O1 program.cpp -o test

# Do not use for production builds (large performance overhead)
# Use only in development/test environments

Sanitizer Option Settings

# ASan options
export ASAN_OPTIONS=detect_leaks=1:check_initialization_order=1

# TSan options
export TSAN_OPTIONS=second_deadlock_stack=1

# run
./test

---

4.Memory leak tracking

Valgrind: Memory Profiling

#include <iostream>

void leakyFunction() {
int* leak = new int[100];
// delete[] leak;// omission!💥
}

int main() {
for (int i = 0; i < 10; ++i) {
leakyFunction();
}
return 0;
}

# Detect memory leaks with Valgrind
g++ -g leak.cpp -o leak
valgrind --leak-check=full --show-leak-kinds=all ./leak

# Example output:
# ==12345== HEAP SUMMARY:
# ==12345== in use at exit: 4,000 bytes in 10 blocks
# ==12345== total heap usage: 10 allocs, 0 frees, 4,000 bytes allocated
# ==12345==
# ==12345== 4,000 bytes in 10 blocks are definitely lost in loss record 1 of 1
# ==12345== at 0x...: operator new
# ==12345== by 0x...: leakyFunction() (leak.cpp:4)
# ==12345== by 0x...: main (leak.cpp:9)

shared_ptr circular reference detection

#include <iostream>
#include <memory>

class Node {
public:
std::shared_ptr<Node> next;
~Node() {
std::cout << "Node destroyed" << std::endl;
}
};

int main() {
auto node1 = std::make_shared<Node>();
auto node2 = std::make_shared<Node>();

// create circular reference 💥
node1->next = node2;
node2->next = node1;

// destructor not called when main exits (memory leak)
return 0;
}

Solution: Use weak_ptr:

#include <iostream>
#include <memory>

class Node {
public:
std::weak_ptr<Node> next;// ✅ Change to weak_ptr
~Node() {
std::cout << "Node destroyed" << std::endl;
}
};

int main() {
auto node1 = std::make_shared<Node>();
auto node2 = std::make_shared<Node>();

node1->next = node2;
node2->next = node1;

// ✅ Now destroys normally
return 0;
}

###Detecting memory leaks with ASan

# ASan also detects memory leaks by default
g++ -fsanitize=address -g leak.cpp -o leak
./leak

# Output:
# =================================================================================
# ==12345==ERROR: LeakSanitizer: detected memory leaks
#
# Direct leak of 400 byte(s) in 1 object(s) allocated from:
# #0 0x... in operator new
# #1 0x... in leakyFunction() leak.cpp:4
# #2 0x... in main leak.cpp:9

Heaptrack: Heap memory profiling

# Install Heaptrack (Linux)
sudo apt install heaptrack

# Run program
heaptrack ./myapp

# Results analysis
heaptrack_gui heaptrack.myapp.12345.gz

---

5.Multithreaded Debugging

Data Race Practical Scenario

#include <iostream>
#include <thread>
#include <vector>

class BankAccount {
private:
int balance = 1000;

public:
void withdraw(int amount) {
// 💥 Data Race: balance read/write is not atomic
if (balance >= amount) {
std::this_thread::sleep_for(std::chrono::milliseconds(1));
balance -= amount;
}
}

int getBalance() const { return balance;}
};

int main() {
BankAccount account;
std::vector<std::thread> threads;

// 10 threads simultaneously withdraw 100 won each
for (int i = 0; i < 10; ++i) {
threads.emplace_back([&account]() {
for (int j = 0; j < 10; ++j) {
account.withdraw(100);
}
});
}

for (auto& t : threads) {
t.join();
}

// Expected: 1000 - (10 * 10 * 100) = -9000 or negative
// Real: Different value every time (data race)
std::cout << "Final balance: " << account.getBalance() << std::endl;

return 0;
}

Detected with TSan:

g++ -fsanitize=thread -g bank.cpp -o bank -pthread
./bank

# WARNING: ThreadSanitizer: data race

Solution: Use mutex:

#include <iostream>
#include <thread>
#include <vector>
#include <mutex>

class BankAccount {
private:
int balance = 1000;
std::mutex mtx;// ✅ Add mutex

public:
void withdraw(int amount) {
std::lock_guard<std::mutex> lock(mtx);// ✅ Acquire lock
if (balance >= amount) {
std::this_thread::sleep_for(std::chrono::milliseconds(1));
balance -= amount;
}
}

int getBalance() {
std::lock_guard<std::mutex> lock(mtx);
return balance;
}
};

Deadlock detection

#include <iostream>
#include <thread>
#include <mutex>

std::mutex mutex1, mutex2;

void thread1() {
std::lock_guard<std::mutex> lock1(mutex1);
std::this_thread::sleep_for(std::chrono::milliseconds(10));
std::lock_guard<std::mutex> lock2(mutex2);// 💥 Deadlock
std::cout << "Thread 1" << std::endl;
}

void thread2() {
std::lock_guard<std::mutex> lock2(mutex2);
std::this_thread::sleep_for(std::chrono::milliseconds(10));
std::lock_guard<std::mutex> lock1(mutex1);// 💥 Deadlock
std::cout << "Thread 2" << std::endl;
}

int main() {
std::thread t1(thread1);
std::thread t2(thread2);

t1.join();
t2.join();

return 0;
}

Deadlock analysis with GDB:

# If the program stops, stop with Ctrl+C
g++ -g -pthread deadlock.cpp -o deadlock
./deadlock
# (pause)
# In another terminal:
gdb -p $(pidof deadlock)

(gdb) info threads
(gdb) thread apply all backtrace

# Check what locks each thread is waiting for

Solution: std::scoped_lock (C++17):

#include <iostream>
#include <thread>
#include <mutex>

std::mutex mutex1, mutex2;

void thread1() {
std::scoped_lock lock(mutex1, mutex2);// ✅ Deadlock prevention
std::cout << "Thread 1" << std::endl;
}

void thread2() {
std::scoped_lock lock(mutex1, mutex2);// ✅ Always acquire locks in the same order
std::cout << "Thread 2" << std::endl;
}

int main() {
std::thread t1(thread1);
std::thread t2(thread2);

t1.join();
t2.join();

return 0;
}

Debugging iterator invalidation

#include <iostream>
#include <vector>
#include <algorithm>

int main() {
std::vector<int> vec = {1, 2, 3, 4, 5};

// 💥 Invalidate iterator
for (auto it = vec.begin(); it != vec.end(); ++it) {
if (*it % 2 == 0) {
vec.erase(it);// it is invalidated after erase
}
}

return 0;
}

Track with GDB Watchpoint:

(gdb) break main
(gdb)run
(gdb) watch vec._M_impl._M_start # vector internal pointer watch
(gdb)continue
# Halts when calling erase

Solution: erase-remove idiom:

#include <iostream>
#include <vector>
#include <algorithm>

int main() {
std::vector<int> vec = {1, 2, 3, 4, 5};

// ✅ erase-remove idiom
vec.erase(
    std::remove_if(vec.begin(), vec.end(),
        [](int x) { return x % 2 == 0; }),
    vec.end()
);

// Or C++20 erase_if:
// std::erase_if(vec, [](int x) { return x % 2 == 0; });

return 0;
}

---

6. Debugging in production

Structured logging

#include <iostream>
#include <fstream>
#include <chrono>
#include <iomanip>
#include <sstream>
#include <mutex>

class Logger {
public:
enum Level { DEBUG, INFO, WARNING, ERROR, FATAL };

private:
static std::mutex mtx_;
static std::ofstream file_;
static Level min_level_;

public:
static void init(const std::string& filename, Level min_level = INFO) {
file_.open(filename, std::ios::app);
min_level_ = min_level;
}

static void log(Level level, const std::string& message,
const char* file = __builtin_FILE(),
int line = __builtin_LINE(),
const char* func = __builtin_FUNCTION()) {
if (level < min_level_) return;

std::lock_guard<std::mutex> lock(mtx_);

auto now = std::chrono::system_clock::now();
auto time = std::chrono::system_clock::to_time_t(now);
auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(
now.time_since_epoch()) % 1000;

std::ostringstream oss;
oss << std::put_time(std::localtime(&time), "%Y-%m-%d %H:%M:%S")
<< "." << std::setfill('0') << std::setw(3) << ms.count()
<< " [" << levelToString(level) << "] "
<< "[" << file << ":" << line << "] "
<< "[" << func << "] "
<< message << std::endl;

std::string log_line = oss.str();
std::cout << log_line;
if (file_.is_open()) {
file_ << log_line;
file_.flush();
}
}

private:
static const char* levelToString(Level level) {
switch (level) {
case DEBUG:   return "DEBUG";
case INFO:    return "INFO";
case WARNING: return "WARN";
case ERROR:   return "ERROR";
case FATAL:   return "FATAL";
default:      return "UNKNOWN";
}
}
};

std::mutex Logger::mtx_;
std::ofstream Logger::file_;
Logger::Level Logger::min_level_ = Logger::INFO;

// Conveniently used as a macro
#define LOG_DEBUG(msg) Logger::log(Logger::DEBUG, msg, __FILE__, __LINE__, __func__)
#define LOG_INFO(msg) Logger::log(Logger::INFO, msg, __FILE__, __LINE__, __func__)
#define LOG_ERROR(msg) Logger::log(Logger::ERROR, msg, __FILE__, __LINE__, __func__)

int main() {
Logger::init("app.log", Logger::DEBUG);

LOG_INFO("Start program");
LOG_DEBUG("Debug Information");
LOG_ERROR("Error occurred");

return 0;
}

Automatic core dump collection

#!/bin/bash
# core_dump_setup.sh

# Enable core dump
ulimit -c unlimited

# Set core dump file location
echo "/var/crash/core.%e.%p.%t" | sudo tee /proc/sys/kernel/core_pattern

# Run program
./myapp

# Create a core dump in /var/crash/ when a crash occurs

Remote debugging (gdbserver)

# Server (production environment)
gdbserver :1234 ./myapp

# Client (development environment)
gdb ./myapp
(gdb) target remote server_ip:1234
(gdb) continue

Production crash reporting

#include <csignal>
#include <cstdlib>
#include <iostream>
#include <execinfo.h>
#include <unistd.h>

void signalHandler(int sig) {
std::cerr << "Error: signal " << sig << std::endl;

// output stack trace
void* array[10];
size_t size = backtrace(array, 10);

std::cerr << "Stack trace:" << std::endl;
backtrace_symbols_fd(array, size, STDERR_FILENO);

exit(1);
}

int main() {
// Register signal handler
signal(SIGSEGV, signalHandler);
signal(SIGABRT, signalHandler);

// program logic
int* ptr = nullptr;
*ptr = 42;// Output stack trace when a crash occurs

return 0;
}

---

7. Complete debugging workflow

Step-by-step process

flowchart TB
Start[Bug found] --> Reproduce[Reproducible?]
Reproduce -->|Yes| Minimal[Minimal repro]
Reproduce -->|No| Logging[Add logging]
Logging --> Reproduce

Minimal --> Hypothesis[Hypothesis]
Hypothesis --> Tool{Pick tool}

Tool -->|Memory bugs| ASan[AddressSanitizer]
Tool -->|Data races| TSan[ThreadSanitizer]
Tool -->|Undefined behavior| UBSan[UBSan]
Tool -->|General| GDB[GDB/LLDB]

ASan --> Verify[Verify]
TSan --> Verify
UBSan --> Verify
GDB --> Verify

Verify -->|Fixed| Test[Add tests]
Verify -->|Not fixed| Hypothesis

Test --> Done[Done]

Example: compound bug

#include <iostream>
#include <thread>
#include <vector>
#include <memory>

class Resource {public:
int*data;

Resource() : data(new int[100]) {
std::cout << "Resource created" << std::endl;
}

~Resource() {
delete[] data;
std::cout << "Resource destroyed" << std::endl;
}
};

std::shared_ptr<Resource> global_resource;

void worker() {
// 💥 Mixed bugs
for (int i = 0; i < 1000; ++i) {
if (!global_resource) {
global_resource = std::make_shared<Resource>();
}

// data race
global_resource->data[i % 100] = i;

if (i == 500) {
global_resource.reset();// other threads may be busy
}
}
}

int main() {
std::vector<std::thread> threads;
for (int i = 0; i < 4; ++i) {
threads.emplace_back(worker);
}

for (auto& t : threads) {
t.join();
}

return 0;
}

Debugging Steps:

# Step 1: Detect data race with TSan
g++ -fsanitize=thread -g bug.cpp -o bug -pthread
./bug
# WARNING: ThreadSanitizer: data race

# Step 2: Detect memory errors with ASan
g++ -fsanitize=address -g bug.cpp -o bug -pthread
./bug
# ERROR: AddressSanitizer: heap-use-after-free

# Step 3: Detailed analysis with GDB
g++ -g bug.cpp -o bug -pthread
gdb ./bug
(gdb) break worker
(gdb)run
(gdb) info threads
(gdb) thread apply all backtrace

Modified version:

#include <iostream>
#include <thread>
#include <vector>
#include <memory>
#include <mutex>

class Resource {
public:
int*data;

Resource() : data(new int[100]) {
std::cout << "Resource created" << std::endl;
}

~Resource() {
delete[] data;
std::cout << "Resource destroyed" << std::endl;
}
};

std::shared_ptr<Resource> global_resource;
std::mutex resource_mutex;

void worker() {
for (int i = 0; i < 1000; ++i) {
std::shared_ptr<Resource> local_resource;

{
std::lock_guard<std::mutex> lock(resource_mutex);
if (!global_resource) {
global_resource = std::make_shared<Resource>();
}
local_resource = global_resource;// ✅ Maintain local copy
}

// ✅ Now safely accessible
local_resource->data[i % 100] = i;
}
}

int main() {
std::vector<std::thread> threads;
for (int i = 0; i < 4; ++i) {
threads.emplace_back(worker);
}

for (auto& t : threads) {
t.join();
}

return 0;
}

---

8.Common mistakes and solutions

Mistake 1: Compiling without debug symbols

# ❌ Wrong method
g++ -O2 program.cpp -o program

# ✅ Correct way
g++ -g -O0 program.cpp -o program # Debug build
g++ -g -O2 program.cpp -o program # Release build (with debug symbols)

Mistake 2: Optimizing Variables Due to Optimization

int main() {
int x = 10;
int y = x + 5;// compiler optimizes for y = 15
return y;
}

# When you try to print x in GDB, you get the message “optimized out”
(gdb) print x
# $1 = <optimized out>

# Solved: Compile with -O0
g++ -g -O0 program.cpp -o program

Mistake 3: Sanitizer and Optimization Levels

# ❌ -O0 may hide some bugs
g++ -fsanitize=address -g -O0 program.cpp

# ✅ -O1 or -O2 recommended (Sanitizer official recommendation)
g++ -fsanitize=address -g -O1 program.cpp

Mistake 4: Missing -pthread in multithreaded programs

# ❌ Link error or runtime error
g++ -fsanitize=thread -g program.cpp

# ✅ Add -pthread
g++ -fsanitize=thread -g program.cpp -pthread

Mistake 5: Limit core dump size

# Check if no core dump is generated
ulimit -c
# If 0, it is disabled

# Set to unlimited
ulimit -c unlimited

# Permanent settings (added to /etc/security/limits.conf)
# * soft core unlimited
# * hard core unlimited

Mistake 6: Using Valgrind and ASan simultaneously

# ❌ Conflict occurs
g++ -fsanitize=address program.cpp -o program
valgrind ./program

# ✅ Use only one
# When using ASan
g++ -fsanitize=address program.cpp -o program
./program

# When using Valgrind
g++ -g program.cpp -o program
valgrind --leak-check=full ./program

---

9.Best practices/best practices

Development environment settings

# Add debug option to CMakeLists.txt
if(CMAKE_BUILD_TYPE MATCHES Debug)
add_compile_options(-g -O0)
add_compile_options(-fsanitize=address,undefined)
add_link_options(-fsanitize=address,undefined)
endif()

if(CMAKE_BUILD_TYPE MATCHES RelWithDebInfo)
add_compile_options(-g -O2)
endif()

Sanitizer integration into CI/CD

# .github/workflows/ci.yml
name: C.I.

on: [push, pull_request]

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- name: Build with ASan
run: |
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make

- name: Run tests
run: |cd build
export ASAN_OPTIONS=detect_leaks=1
ctest --output-on-failure

Assertion strategy

#include <cassert>
#include <iostream>

// Assertions only active in debug builds
#ifdef NDEBUG
#define DEBUG_ASSERT(condition, message) ((void)0)
#else
#define DEBUG_ASSERT(condition, message) \
if (!(condition)) { \
std::cerr << "Assertion failed: " << message << std::endl;\
std::cerr << "File: " << __FILE__ << ", Line: " << __LINE__ << std::endl;\
std::abort();\
}
#endif

// Assertions that are also enabled in release builds
#define RELEASE_ASSERT(condition, message) \
if (!(condition)) { \
std::cerr << "Fatal error: " << message << std::endl;\
std::cerr << "File: " << __FILE__ << ", Line: " << __LINE__ << std::endl;\
std::abort();\
}

int main() {
int x = 10;

// Only checked during development
DEBUG_ASSERT(x > 0, "x must be positive");

// Always check (important invariant)
RELEASE_ASSERT(x < 100, "x must be less than 100");

return 0;
}

Logging level strategy

// Development: DEBUG level
Logger::init("app.log", Logger::DEBUG);

// Staging: INFO level
Logger::init("app.log", Logger::INFO);

// Production: WARNING level
Logger::init("app.log", Logger::WARNING);

Test-driven debugging

#include <cassert>
#include <iostream>

// Write a bug reproducibility test
void test_divide_by_zero() {
try {
int result = divide(10, 0);
assert(false && "Should throw exception");
} catch (const std::invalid_argument& e) {
std::cout << "Test passed: " << e.what() << std::endl;
}
}

int main() {
test_divide_by_zero();
return 0;
}

---

10.production pattern

Pattern 1: Health check endpoint

#include <iostream>
#include <chrono>
#include <thread>
#include <atomic>

class HealthMonitor {
private:
std::atomic<bool> is_healthy_{true};
std::chrono::steady_clock::time_point last_heartbeat_;

public:
void heartbeat() {
last_heartbeat_ = std::chrono::steady_clock::now();
}

bool isHealthy() {
auto now = std::chrono::steady_clock::now();
auto elapsed = std::chrono::duration_cast<std::chrono::seconds>(
now - last_heartbeat_).count();

// Abnormal if there is no heartbeat for more than 10 seconds
return elapsed < 10;
}

void setUnhealthy() {
is_healthy_ = false;
}
};

// Serve /health endpoint from HTTP server
// GET /health -> {"status": "ok", "uptime": 12345}

Pattern 2: Metric collection

#include <iostream>
#include <atomic>
#include <chrono>

class Metrics {
private:
std::atomic<uint64_t> request_count_{0};
std::atomic<uint64_t> error_count_{0};
std::atomic<uint64_t> total_latency_ms_{0};

public:
void recordRequest(uint64_t latency_ms, bool is_error = false) {
++request_count_;
total_latency_ms_ += latency_ms;
if (is_error) {
++error_count_;
}
}

void report() {
uint64_t requests = request_count_.load();
uint64_t errors = error_count_.load();
uint64_t latency = total_latency_ms_.load();

std::cout << "Requests: " << requests << std::endl;
std::cout << "Errors: " << errors << std::endl;
if (requests > 0) {
std::cout << "Avg latency: " << (latency / requests) << "ms" << std::endl;
std::cout << "Error rate: " << (errors * 100.0 / requests) << "%" << std::endl;
}
}
};

Pattern 3: Graceful Shutdown

#include <csignal>
#include <atomic>
#include <iostream>
#include <thread>

std::atomic<bool> shutdown_requested{false};

void signalHandler(int signal) {
if (signal == SIGINT || signal == SIGTERM) {
std::cout << "Shutdown requested..." << std::endl;
shutdown_requested = true;
}
}

int main() {
signal(SIGINT, signalHandler);
signal(SIGTERM, signalHandler);

std::cout << "Server started. Press Ctrl+C to stop."<< std::endl;

while (!shutdown_requested) {
// main loop
std::this_thread::sleep_for(std::chrono::seconds(1));
}

std::cout << "Shutting down gracefully..." << std::endl;
// Clean up resources

std::cout << "Shutdown complete."<< std::endl;
return 0;
}

Pattern 4: Circular buffer logging

#include <array>
#include <string>
#include <mutex>

template<size_t N>
class CircularLogBuffer {
private:
std::array<std::string, N> buffer_;
size_t index_ = 0;
std::mutex mtx_;

public:
void log(const std::string& message) {
std::lock_guard<std::mutex> lock(mtx_);
buffer_[index_] = message;
index_ = (index_ + 1) % N;
}

void dump() {
std::lock_guard<std::mutex> lock(mtx_);
std::cout << "=== Last " << N << " log entries ===" << std::endl;
for (size_t i = 0; i < N; ++i) {
size_t idx = (index_ + i) % N;
if (!buffer_[idx].empty()) {
std::cout << buffer_[idx] << std::endl;
}
}
}
};

// Dump only the most recent log when crashing
CircularLogBuffer<100> crash_log;

---

11.Organize and Checklist

Guide to selecting debugging tools

Problem type	Recommended Tools	When to use
memory leak	ASan, Valgrind	Development/Test
data race	TSan	Multi-threaded development
undefined behavior	UBSan	All Builds
General Crash	GDB/LLDB	In development
Production Crash	core dump + GDB	Production
performance bottleneck	perf, gprof	Optimization steps
uninitialized memory	MSan	Clang Environment

Sanitizer performance comparison

tools	speed overhead	memory overhead	Detection range	Compiler Support
ASan	2x	2~3x	memory error, leak	GCC, Clang, MSVC
TSan	5~15x	5~10x	data race	GCC, Clang
UBSan	1.2x	Minimum	undefined behavior	GCC, Clang, MSVC
MSan	3x	2x	uninitialized memory	Clang only
Valgrind	10~50x	Minimum	Memory overall	All binaries

Recommended combination: ASan + UBSan (daily development), TSan (multithreaded), Valgrind (deep analysis)

Development environment checklist

# ✅ Debug build settings
- [ ] -g flag added
- [ ] Use -O0 or -O1
- [ ] Enable Sanitizer (-fsanitize=address,undefined)
- [ ] Maximize compilation warnings (-Wall -Wextra -Wpedantic)

# ✅ Test environment
- [ ] Write unit tests
- [ ] Integration of Sanitizer into CI/CD
- [ ] Code coverage measurement
- [ ] Fuzzing test (AFL, libFuzzer)

# ✅ Production ready
- [ ] Building a logging system (by level, file rotation)
- [ ] Enable core dump (ulimit -c unlimited)
- [ ] Health check endpoint
- [ ] Metric collection (number of requests, error rate, latency)
- [ ] Graceful shutdown (SIGTERM handler)
- [ ] Monitoring notification (Prometheus, Grafana)

Practical tip: Reduce debugging time

1. Always Develop with Sanitizer

# Add to .bashrc or .zshrc
alias g++debug='g++ -g -O1 -fsanitize=address,undefined -Wall -Wextra'

# use
g++debug myapp.cpp -o myapp

2. GDB configuration file (.gdbinit)

# ~/.gdbinit
set print pretty on
set print array on
set print array-indexes on
set pagination off

# Shorten frequently used commands
define pv
print $arg0
end

3. Quick Repro Script

#!/bin/bash
# reproduce_bug.sh

# Debug build
g++ -g -O0 -fsanitize=address bug.cpp -o bug

# Run multiple times (check for replay)
for i in {1..10};do
echo "Run $i"
./bug ||break
done

4. Dynamically change log level

// Control log level with environment variables
const char* log_level = std::getenv("LOG_LEVEL");
if (log_level && std::string(log_level) == "DEBUG") {
Logger::setLevel(Logger::DEBUG);
}

// When running
// LOG_LEVEL=DEBUG ./myapp

Debugging workflow summary

1. Reproducible: Reduce bugs to minimal reproducible code
2. Hypothesis: Establishing a causal hypothesis
3. Tools: Select appropriate debugging tools
4. Analysis: Cause analysis with GDB and Sanitizer
5. Edit: Bug fix
6. Validation: Write and run tests
7. Documentation: Record bug causes and solutions

Quick reference: Debugging checklist

# 🔍 Check immediately when a bug is found
□ Is it reproducible?(Not reproducible → Add logging)
□ Have you completed writing minimally reproducible code?
□ Check all compilation warnings?(gcc -Wall -Wextra)
□ Enable Sanitizer?(ASan + UBSan)

# 🛠️ Select tool
□ Memory error → ASan or Valgrind
□ Data Race → TSan
□ Crash → GDB + core dump
□ Performance bottleneck → perf, gprof

# ✅ After solving
□ Write test cases
□ Request a code review
□ Documentation (bug causes and solutions)

Troubleshooting: Quick problem solving

Symptoms	Cause	Solution
Segfault	nullptr dereference, array out of bounds	Enable ASan, check stack with GDB
memory leak	delete missing, circular reference	Valgrind or ASan leak detection
data race	Missing sync	Enable TSan, add mutex
Deadlock	Circular lock wait	Checking thread stack with GDB, using scoped_lock
Random crash	UB, uninitialized variable	Enable UBSan, MSan
slow build	PCH unused, single threaded	ccache, Ninja, parallel builds

Next steps

• Learn more in-depth debugging techniques in GDB Advanced Guide
• Take advantage of the advanced features of each Sanitizer in the Complete Guide to Sanitizer
• Learn how to solve complex memory problems in Debugging Memory Leaks
• Learn the basics of thread programming at Multithread Basics
• In-depth analysis of crash causes in Segfault Debugging

---

FAQ

Q1: Where should I start debugging?

A:

1. Reduce bugs to minimal, reproducible code
2. Compile and run with ASan + UBSan (automatically detects most bugs)
3. If you still have problems, step through with GDB

Q2: GDB vs LLDB which should I use?

A:

• Linux: GDB (standard)
• macOS: LLDB (built-in, Xcode integrated)
• Windows: Visual Studio Debugger or GDB (MinGW)
• The commands are almost similar, so you only need to learn one.

Q3: Can I use Sanitizer in production?

A:

• ❌ Not recommended: Large performance overhead (2 to 5 times), increased memory usage
• Use only in development/test environments
• Use logging + core dump + monitoring in production

Q4: Valgrind vs ASan which is better??

A:

• ASan: Faster (2-3 times slower), integrated at compile time, detects more bugs
• Valgrind: slow (10-50x slower), separate execution, precise memory tracking.
• Recommended: ASan for routine use, Valgrind for detailed analysis

Q5: How do I reproduce multithreaded bugs?

A:

• Compiled with TSan (automatically detects most races)
• Timing control with std::this_thread::sleep_for
• Increased contention by increasing the number of threads
• Create load with stress tool

Q6: What are the debugging learning resources?

A:

• Book: “The Art of Debugging with GDB, DDD, and Eclipse”
• Documentation: GDB official documentation, Sanitizer documentation
• Practice: Intentionally create bugs and practice debugging
• Open Source: Learn the bug fixing process from issue trackers of popular projects

Q7: What should I pay attention to when debugging in a production environment?

A:

• Minimize performance impact: Set logging level appropriately (WARNING or higher only)
• Protection of sensitive information: Do not leave passwords, API keys, etc. in logs
• Core dump size limit: Manage disk space with ulimit -c setting.
• Remote debugging security: Firewall settings when using gdbserver
• Build a repro environment: Test in the same environment as production

Q8: I turned on ASan, but memory usage is too high.

A: ASan increases memory usage by 2-3x.This is normal.

• Use only in development/test environments
• If memory is insufficient, split into small test cases
• Or use Valgrind (slower but uses less memory)
• CI/CD uses instances with sufficient memory

Q9: “optimized out” message appears in GDB

A: Variables have been removed as a compiler optimization.

# Solution: Disable optimization
g++ -g -O0 program.cpp # use -O0

# Or disable optimization only for specific functions
__attribute__((optimize("O0")))
void debugFunction() {
// This function is not optimized
}

Q10: How do I prevent deadlock?

A:

• Lock order consistency: Locks are always acquired in the same order.
• Use std::scoped_lock (C++17, deadlock prevention)
• Timeout Settings: Use try_lock_for()
• Minimize lock holding time: Avoid long operations within the lock.
• Verify with TSan: Always enable TSan during development

---

Good article to read together (internal link)

Here’s another article related to this topic.

• C++ GDB/LLDB |A bug that couldn’t be found even after printing 100 couts, solved in 5 minutes with a debugger
• C++ Runtime Verification: A Complete Guide to AddressSanitizer and ThreadSanitizer [#41-2]
• C++ Segmentation fault |core dump
• C++ Memory Leak |“Memory Leak” Guide
• C++ std::thread introduction |3 common mistakes such as missing joins and overuse of detach and solutions
• C++ Sanitizers |“Sanitizer” guide

---

Keywords covered in this article (related search terms)

This article will be helpful if you search for C++, debugging, gdb, lldb, sanitizer, valgrind, memory leak, multithreaded, debugging, etc.

---

Related articles

• C++ Debugging Practical Guide |Perfect use of gdb, LLDB, and Visual Studio
• 5 causes of C++ Segmentation fault and debugging methods |Tracking with GDB
• Finding C++ memory leaks |With Valgrind·ASan
• C++ GDB |
• C++ Memory Leak |