Core Dump Complete Guide | Generation, Analysis & Debugging

Core Dump Complete Guide | Generation, Analysis & Debugging

· 50분 읽기 · 수정 Advanced Guide
Core Dump Complete Guide | Generation, Analysis & Debugging

이 글의 핵심

Complete guide to Core Dump generation, GDB analysis, Segmentation Fault debugging, and production cases. Solve production crash issues with ulimit configuration, symbol files, and backtrace analysis.

Introduction: Why Core Dump is Needed

How do you debug when a program suddenly crashes in production? Core Dump is essential for solving hard-to-reproduce bugs, intermittent Segmentation Faults, and memory corruption issues. Value of Core Dump:

  • Complete memory snapshot at crash time
  • All variable values and stack traces
  • Post-mortem analysis without reproduction
  • Real data from production environment What This Guide Covers:
  • Core Dump generation configuration
  • Core Dump analysis with GDB
  • Practical debugging scenarios
  • Automation and monitoring

Reality in Production

When learning development, everything seems clean and theoretical. But production is different. Wrestling with legacy code, chasing tight deadlines, facing unexpected bugs. The content covered here was initially learned as theory, but through applying it to real projects, I realized “ah, that’s why it’s designed this way.” Particularly memorable are the trials and errors from my first project. I followed what I learned from books but couldn’t figure out why it didn’t work, spending days stuck. Eventually discovered the problem through senior developer code review, learning a lot in the process. This guide covers not just theory but also pitfalls and solutions you might encounter in practice.

Table of Contents

  1. Core Dump Basics
  2. Core Dump Generation Setup
  3. GDB Core Dump Analysis
  4. Practical Debugging Scenarios
  5. Automation and Collection
  6. Production Environment Setup
  7. Advanced Analysis Techniques
  8. Troubleshooting

1. Core Dump Basics

What is Core Dump?

Core Dump is saving memory contents to a file when a program terminates abnormally (crashes).

flowchart TB
    Program[Program Running] --> Crash["Crash Occurs\nSIGSEGV, SIGABRT etc"]
    
    Crash --> Kernel[Kernel Receives Signal]
    
    Kernel --> Check{Core Dump\nEnabled?}
    
    Check -->|Yes| Dump["Generate Memory Dump\ncore.12345"]
    Check -->|No| Exit[Program Terminates]
    
    Dump --> File["Core Dump File\n- Stack\n- Heap\n- Registers\n- Variable Values"]
    
    File --> Debug[Analyze with GDB]

Information in Core Dump

1. Stack Memory
   - Function call stack
   - Local variables
   - Function arguments
2. Heap Memory
   - Dynamically allocated memory
   - Objects created with malloc/new
3. Register State
   - PC (Program Counter)
   - SP (Stack Pointer)
   - General purpose registers
4. Global Variables
   - Static variables
   - Global objects
5. Shared Library Mappings
   - Loaded .so file list
   - Memory address mappings

Signals That Cause Crashes

// Signals that generate Core Dump
SIGQUIT    // Quit (Ctrl+\)
SIGILL     // Illegal Instruction
SIGABRT    // Abort (assert failure)
SIGFPE     // Floating Point Exception (divide by zero)
SIGSEGV    // Segmentation Fault (invalid memory access)
SIGBUS     // Bus Error (alignment error)
SIGTRAP    // Trace/Breakpoint Trap
SIGSYS     // Bad System Call

2. Core Dump Generation Setup

ulimit Configuration

# Check current Core Dump size limit
ulimit -c
# 0 → Core Dump disabled
# unlimited → No limit
# Enable Core Dump (current session)
ulimit -c unlimited
# Permanent setting (all users)
echo "* soft core unlimited" | sudo tee -a /etc/security/limits.conf
echo "* hard core unlimited" | sudo tee -a /etc/security/limits.conf
# Specific user only
echo "myuser soft core unlimited" | sudo tee -a /etc/security/limits.conf
# For systemd services
# /etc/systemd/system/myapp.service
[Service]
LimitCORE=infinity

Core Dump Save Location

# Check current setting
cat /proc/sys/kernel/core_pattern
# Default: core (creates 'core' file in current directory)
# Set custom pattern
sudo sysctl -w kernel.core_pattern=/var/crash/core.%e.%p.%t
# Permanent setting
echo "kernel.core_pattern=/var/crash/core.%e.%p.%t" | sudo tee -a /etc/sysctl.conf
# Pattern variables:
# %e - Executable name
# %p - PID
# %t - Timestamp (Unix time)
# %s - Signal number
# %u - UID
# %g - GID
# %h - Hostname
# Example: core.myapp.12345.1234567890
# Install systemd-coredump (Ubuntu)
sudo apt install systemd-coredump
# Configuration
sudo nano /etc/systemd/coredump.conf
[Coredump]
Storage=external
Compress=yes
ProcessSizeMax=2G
ExternalSizeMax=2G
MaxUse=10G
# Core Dump save location
# /var/lib/systemd/coredump/
# List Core Dumps
coredumpctl list
# Specific Core Dump info
coredumpctl info <PID>
# Extract Core Dump
coredumpctl dump <PID> -o core.dump
# Analyze directly with GDB
coredumpctl debug <PID>

3. GDB Core Dump Analysis

Basic Analysis Flow

# 1. Compile with debug symbols
g++ -g -O0 myapp.cpp -o myapp
# 2. Configure to generate Core Dump
ulimit -c unlimited
# 3. Run program (crash occurs)
./myapp
# Segmentation fault (core dumped)
# 4. Open Core Dump with GDB
gdb ./myapp core.12345
# Or use systemd-coredump
coredumpctl debug 12345

GDB Basic Commands

# After loading Core Dump
# 1. Backtrace (stack trace)
(gdb) bt
#0  0x00005555555551a9 in crash_function () at myapp.cpp:10
#1  0x00005555555551c8 in main () at myapp.cpp:15
# Detailed backtrace
(gdb) bt full
#0  0x00005555555551a9 in crash_function () at myapp.cpp:10
        ptr = 0x0
        value = 42
#1  0x00005555555551c8 in main () at myapp.cpp:15
        argc = 1
        argv = 0x7fffffffe0a8
# 2. Move to specific frame
(gdb) frame 0
(gdb) frame 1
# 3. Check source code
(gdb) list
5       void crash_function() {
6           int* ptr = nullptr;
7           int value = 42;
8           
9           // Segmentation Fault!
10          *ptr = value;
11      }
# 4. Check variable values
(gdb) print ptr
$1 = (int *) 0x0
(gdb) print value
$2 = 42
# 5. All local variables
(gdb) info locals
ptr = 0x0
value = 42
# 6. Register state
(gdb) info registers
rax            0x0                 0
rbx            0x555555557d60      93824992247136
rsp            0x7fffffffddd0      0x7fffffffddd0
rip            0x5555555551a9      0x5555555551a9 <crash_function()+15>
# 7. Check memory contents
(gdb) x/10x $rsp
0x7fffffffddd0: 0xffffddf0      0x00007fff      0x555551c8      0x00005555
# 8. Thread information
(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7ffff7fc1740 (LWP 12345) crash_function () at myapp.cpp:10
  2    Thread 0x7ffff77c0700 (LWP 12346) 0x00007ffff7bc8e2d in poll ()
# 9. Switch to specific thread
(gdb) thread 2
(gdb) bt

Practical Example 1: Null Pointer Dereference

// crash_null.cpp
#include <iostream>
void process_data(int* data) {
    std::cout << "Processing: " << *data << std::endl;  // Crash!
}
int main() {
    int* ptr = nullptr;
    process_data(ptr);
    return 0;
}
# Compile (with debug symbols)
g++ -g -O0 crash_null.cpp -o crash_null
# Run
./crash_null
# Segmentation fault (core dumped)
# GDB analysis
gdb ./crash_null core
(gdb) bt
#0  0x00005555555551b2 in process_data (data=0x0) at crash_null.cpp:4
#1  0x00005555555551e1 in main () at crash_null.cpp:9
(gdb) frame 0
(gdb) print data
$1 = (int *) 0x0
(gdb) list
1       #include <iostream>
2
3       void process_data(int* data) {
4           std::cout << "Processing: " << *data << std::endl;  // Crash here!
5       }
# Cause: data is nullptr
# Solution: Add nullptr check

4. Practical Debugging Scenarios

Scenario 1: Double Free

// crash_double_free.cpp
#include <cstdlib>
#include <iostream>
int main() {
    int* ptr = new int(42);
    
    std::cout << "Value: " << *ptr << std::endl;
    
    delete ptr;
    
    // Double free!
    delete ptr;
    
    return 0;
}
# Compile
g++ -g -O0 crash_double_free.cpp -o crash_double_free
# Run
./crash_double_free
# Value: 42
# free(): double free detected in tcache 2
# Aborted (core dumped)
# GDB analysis
gdb ./crash_double_free core
(gdb) bt
#0  0x00007ffff7a42387 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff7a43a78 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x0000555555555234 in main () at crash_double_free.cpp:12
(gdb) frame 5
(gdb) print ptr
$1 = (int *) 0x555555559eb0  # Already freed address
# Solution: Set ptr = nullptr after delete

Summary

Core Dump Checklist

Development Environment

  • Set ulimit -c unlimited
  • Compile with -g flag
  • Disable optimization (-O0)
  • Use AddressSanitizer (-fsanitize=address)
  • Memory check with Valgrind

Production Environment

  • Set core limit in /etc/security/limits.conf
  • Configure kernel.core_pattern
  • Install and configure systemd-coredump
  • Monitor disk space
  • Set up automatic collection script
  • Integrate notification system

Analysis

  • Install GDB and learn usage
  • Keep debug symbols
  • Auto-extract backtrace
  • Analyze crash patterns
  • Deduplicate and group

GDB Command Cheatsheet

# Basic
gdb <executable> <core>     # Load Core Dump
bt                          # Backtrace
bt full                     # Detailed backtrace
frame <N>                   # Move to frame N
list                        # Source code
# Variables
print <var>                 # Variable value
print *<ptr>                # Value pointed to by pointer
print <array>[0]@10         # 10 array elements
info locals                 # All local variables
info args                   # Function arguments
# Memory
x/10x <addr>                # Memory (hexadecimal)
x/10s <addr>                # Strings
x/10i <addr>                # Instructions (assembly)
# Threads
info threads                # Thread list
thread <N>                  # Switch to thread N
thread apply all bt         # Backtrace all threads
# Registers
info registers              # All registers
print $rip                  # Specific register
# Other
info sharedlibrary          # Loaded libraries
info proc mappings          # Memory map

Common Crash Causes

flowchart TB
    Crash[Program Crash]
    
    Null["Null Pointer\nDereference"]
    Overflow[Buffer Overflow]
    UseAfter[Use After Free]
    DoubleFree[Double Free]
    StackOver[Stack Overflow]
    Race[Race Condition]
    Assert[Assert Failure]
    
    Crash --> Null
    Crash --> Overflow
    Crash --> UseAfter
    Crash --> DoubleFree
    Crash --> StackOver
    Crash --> Race
    Crash --> Assert
    
    Null --> Fix1[nullptr check]
    Overflow --> Fix2[Bounds checking]
    UseAfter --> Fix3[Smart pointers]
    DoubleFree --> Fix4[nullptr after delete]
    StackOver --> Fix5[Recursion limit]
    Race --> Fix6[Mutex]
    Assert --> Fix7[Condition validation]

References