Linux Internals & Operations: CFS Scheduling and Page Cache
Key takeaways
Linux joins user space and the kernel at the syscall boundary, while the fair scheduler shares CPU time, the page cache and swap absorb memory pressure, and VFS unifies filesystems. This article ties those subsystems to limits, observability, and incident patterns you actually see in production.
Introduction
Linux runs on servers, cloud instances and embedded devices. To answer why latency spikes, why memory alarms fire, or why a container is OOM-killed while the node looks fine, you need a coherent picture of a few kernel subsystems. User-space threads enter the kernel through system calls. The scheduler divides CPU time among runnable threads. Virtual memory keeps file data in the page cache and may push anonymous pages to swap. VFS presents a single interface over ext4, XFS, NFS, overlayfs and other filesystems.
This article is not a command cheat sheet. It connects those internals to operations: cgroup limits, pressure signals and debugging workflows.
A few terms are used throughout. User space is where applications and libraries run without direct hardware access; kernel space is the privileged mode that manages CPUs, memory, block devices and networking. A system call is the supported interface for requesting kernel services. The kernel schedules tasks, and in practice a task is a thread; a process is a group of threads sharing an address space.
CPU and multitasking: why the scheduler matters
A CPU core executes one thread at a time, and the kernel uses time slicing so that many threads appear to run concurrently. The scheduler decides which runnable thread runs next and for how long, within the constraints of preemption, priorities and cgroup limits.
CFS and EEVDF
For normal tasks (SCHED_OTHER), Linux used the Completely Fair Scheduler (CFS) from 2.6.23 until 6.6, when it was replaced by EEVDF (Earliest Eligible Virtual Deadline First). Both are built on the same idea of virtual runtime, so the CFS model below still explains most behavior; the differences matter mainly for latency tuning.
Each task accumulates vruntime as it consumes CPU, and the scheduler favors the runnable task that has received the least weighted CPU time. Over time this balances CPU among competing tasks. Weights come from the nice value and, for groups, from cgroup cpu.weight. A task with a lower nice value (higher priority) has a larger weight, so its vruntime grows more slowly and it is picked more often. This is proportional sharing, not a latency guarantee.
Runqueues and time slices
Each CPU has its own runqueue for fair tasks (cfs_rq), with runnable tasks ordered by vruntime in a red-black tree, so the leftmost node is the next candidate. Tasks migrate between CPUs through load balancing.
How long a task runs before it is preempted is not a fixed quantum. Under CFS, a target latency (sched_latency_ns) was divided among runnable tasks, bounded below by a minimum granularity (sched_min_granularity_ns) to avoid switching too often. Since 5.13 these knobs live in debugfs (/sys/kernel/debug/sched/) rather than /proc/sys/kernel/, and EEVDF replaces them with a base slice (base_slice_ns) plus a per-task deadline, which lets latency-sensitive tasks request shorter slices. Guides that tell you to set kernel.sched_min_granularity_ns with sysctl were written for older kernels.
When the number of runnable threads explodes, each thread’s slice shrinks toward the minimum and scheduling overhead grows: context switches rise without a proportional increase in useful work. perf sched and the nr_switches counters help confirm this.
Hierarchical CPU: cgroups and the fair scheduler
Services and containers usually run in CPU cgroups. cpu.weight sets a group’s share relative to its siblings, and cpu.max sets a hard quota per period (for example 50000 100000 for half a CPU). Limits are hierarchical, so a task cannot exceed its parent group’s quota no matter how “fair” its share inside the child group is. When a process looks healthy in top but responds slowly, look at the cgroup tree as well as the process tree.
The most common surprise is quota throttling. A multithreaded service with cpu.max of one CPU can burn its whole 100 ms quota in 25 ms on four cores and then sit throttled for the remaining 75 ms of the period. Average CPU usage looks moderate, but tail latency jumps. cpu.stat in the cgroup shows nr_throttled and throttled_usec; if they keep rising, the quota is too tight for the workload’s burstiness.
Real-time and deadline classes
SCHED_FIFO and SCHED_RR tasks always preempt normal tasks and run by fixed priority; FIFO tasks run until they block or yield, RR tasks round-robin among equal priorities. A real-time task stuck in a loop can therefore starve everything else on that CPU. The kernel’s real-time throttling (sched_rt_runtime_us, by default 950 ms per second) reserves a little time for normal tasks as a safety net. SCHED_DEADLINE schedules periodic tasks by runtime, deadline and period parameters. Typical web and API workloads stay in the normal class; these classes matter when you debug audio, media or industrial software that uses them.
Two scheduling problems show up repeatedly in production. Context-switch storms come from lock contention, far more runnable threads than cores, or tight polling loops; they burn CPU without progress, and perf shows time in the scheduler and in lock code. Throttling from cpu.max inserts forced idle periods, so latency rises even though profiles of the code itself look unchanged.
Memory: page cache, swap, and OOM
The kernel manages memory in pages, usually 4 KiB. Anonymous memory (heap, stack) and file-backed memory are reclaimed differently, and the page cache dominates read-heavy performance.
Page cache
Reads and writes through the normal file APIs go through the page cache, so repeated reads are served from RAM instead of disk. Writes make pages dirty, and background writeback flushes them to storage later. Because Linux deliberately uses otherwise idle RAM for the page cache, free on a healthy server often shows almost no free memory. That cache is reclaimable, which is why MemAvailable in /proc/meminfo (the available column in free) is the number to alert on: it estimates how much memory can be given to new allocations without swapping. Trouble starts when available itself is low and you see swapping, kswapd activity or reclaim-related latency, not when cache fills the remaining RAM.
The reverse effect matters too. After a reboot, failover or migration, the cache is cold, and read latency stays higher until the working set is loaded again; correlate with iostat, vmstat and the application’s access pattern before blaming the new host.
Reclaim and kswapd
The kernel keeps pages on LRU lists (active and inactive lists, or the multi-generational LRU that became available in 6.1) so that recently used pages survive longer. When free memory falls below the low watermark, the kswapd kernel thread reclaims pages in the background. If allocations outrun it and free memory reaches the min watermark, the allocating thread performs direct reclaim itself, synchronously, inside whatever system call or page fault needed memory. Direct reclaim is a common source of “random” latency spikes under memory pressure; /proc/vmstat counters such as allocstall_* and pgscan_direct show whether it is happening.
Dirty limits and writeback stalls
vm.dirty_background_ratio (or _bytes) sets when background writeback starts, and vm.dirty_ratio sets the point at which a process that dirties more pages is made to wait for writeback. Bursty log writers or batch jobs that write faster than the disk can absorb hit the second threshold and stall in write(), which looks like the application freezing for seconds at a time. On machines with a lot of RAM, the percentage defaults allow gigabytes of dirty data, so byte-based limits are often easier to reason about.
Swap and pressure
When memory is short, the kernel reclaims pages. Clean file pages are cheap to drop because they can be re-read from disk; anonymous pages must be written to swap first. Heavy swapping turns memory pressure into disk latency and can degenerate into thrashing, where the system spends its time moving pages instead of running work. vm.swappiness biases reclaim between file cache and anonymous memory; interpret it together with the workload, since databases that manage their own large buffers are especially sensitive to having them swapped out.
Pressure Stall Information (/proc/pressure/memory, and memory.pressure per cgroup) reports the share of time tasks were stalled waiting for memory. It is often a better alerting signal than raw usage, because it measures the effect on work rather than how full memory is.
OOM killer
If reclaim cannot free enough memory, the OOM killer terminates a process. It picks the task with the highest badness score, which is based mainly on memory usage and adjusted by oom_score_adj. With cgroup memory limits, the OOM happens inside the cgroup that hit its limit, so a container can be killed while the host has plenty of free memory. dmesg or journalctl -k records which process was killed and the memory state of the cgroup at that moment.
drop_caches is useful for benchmarks or targeted debugging, but it cools the cache and raises latency, so it is not a fix. vm.min_free_kbytes controls the watermarks; raising it makes kswapd start earlier and keeps more memory in reserve for bursts, while setting it too high wastes memory and can trigger reclaim when there is no real shortage. Change it only with measurements.
Filesystem stack and VFS
open, read and write enter the Virtual File System (VFS) first. VFS sits above concrete filesystems such as ext4, XFS, NFS, tmpfs and overlayfs and gives them a common object model.
Dentry and inode caches
An inode holds a file’s metadata (size, owner, permissions, timestamps) and the mapping to its data on disk. A dentry is a cached path component that links a name in a directory to an inode. Path resolution walks dentries one component at a time; hot paths are resolved entirely from the dentry cache, while cold paths require reading directory blocks from storage. Workloads with millions of small files put pressure on both caches, and on filesystems with a fixed inode table they can also run out of inodes while plenty of disk space remains.
struct file and file_operations
A successful open() creates a struct file and installs it in the process’s file descriptor table. It holds the current offset, open flags (O_APPEND, O_NONBLOCK, …), a reference to the inode and a pointer to the file_operations table (f_op). read, write and mmap dispatch through f_op into the specific implementation, which is why the same system call follows different kernel paths for an ext4 file, a socket, a pipe or a network filesystem.
Path lookup and RCU
Path lookup first tries a lockless walk protected by RCU, which lets many threads resolve paths concurrently without taking locks on shared directories. If it encounters something it cannot handle that way, such as a dentry that is not cached, it falls back to a slower walk that takes references and locks. Metadata-heavy workloads still cost CPU and I/O, but this design is why concurrent stat calls on cached paths scale well.
Page cache and the block layer
Filesystems map file offsets to disk blocks and submit I/O through the block layer to the device drivers. A successful write() usually only means the data is in the page cache. fsync and fdatasync force it to stable storage, which is why databases and distributed systems call them at commit points and why their latency depends heavily on the device’s flush performance.
Network filesystems and containers
NFS, CephFS, FUSE filesystems and similar stacks add network round trips and cache coherence rules beneath VFS, so operations that are cheap locally (stat, open with permission checks) can each cost a round trip. Container images on overlayfs add a lookup across layers for each path and copy a whole file up to the writable layer on its first modification, which explains many cases of “fast on the host, slow in the container” for metadata-heavy builds.
System call interface
System calls are the interface between user space and the kernel. On x86-64, the C library places the syscall number in rax and the arguments in registers according to the ABI, then executes the syscall instruction. The CPU switches to kernel mode and jumps to the kernel’s entry code, which dispatches through the syscall table to the handler. The result comes back in rax; on failure the kernel returns a negative error number, which the C library stores in errno and turns into a -1 return value. The legacy int 0x80 path from 32-bit x86 still exists for compatibility, but 64-bit programs use syscall.
The vDSO is a small shared library that the kernel maps into every process. It implements calls such as clock_gettime and gettimeofday in user space by reading data the kernel keeps updated, so frequent time queries avoid the cost of entering the kernel.
Observability and debugging
strace -c -p PID counts system calls and their errors for a running process, and strace -f -e trace=openat,connect shows individual calls with arguments. Because strace stops the process at every traced call, it can slow a busy process dramatically, so attach briefly. perf trace provides a similar view with much lower overhead, and eBPF tools attached to the sys_enter_*/sys_exit_* tracepoints or to specific calls such as execve, connect and openat aggregate in the kernel and are suitable for continuous telemetry and policy.
seccomp and containers
Container runtimes apply seccomp filters that restrict which system calls a process may make. What happens on a blocked call depends on the filter’s action: Docker’s and containerd’s default profiles return an error (usually EPERM) for calls they do not allow, while a filter with a kill action terminates the process with SIGSYS. Either way, a new library or runtime version that starts using a syscall the profile does not allow can fail in production in a way that never appeared in testing, so custom profiles need to be regenerated and tested alongside binary upgrades.
Production Linux patterns
cgroups and systemd
cgroups v2 organizes CPU, memory, I/O, pids and other controllers in a single hierarchy. systemd places every service in its own cgroup and ties restarts and resource accounting to it, so systemctl status and systemd-cgtop show per-service usage. As a rule of thumb, “alive but slow” often maps to CPU throttling, and “suddenly killed” to memory limits and the OOM killer.
memory.max and memory.high
memory.max is a hard limit: if the cgroup cannot reclaim below it, the OOM killer runs inside the cgroup. memory.high is a throttling threshold: above it, the kernel forces the cgroup’s tasks into reclaim and slows their allocations, but it never invokes the OOM killer. Setting memory.high somewhat below memory.max gives a workload a chance to shed cache and slow down before it is killed. Kubernetes memory limits map to memory.max, which is why pods can be OOM-killed while the node looks healthy.
PID limits
The pids controller (pids.max) caps the number of tasks in a cgroup. A fork bomb or a thread leak hits this limit before memory runs out, and the symptom is fork() or pthread_create() failing with EAGAIN, which applications often report with confusing messages. pids.events records how many times the limit was hit.
Limits operators touch often
ulimit -n (RLIMIT_NOFILE) is the per-process file descriptor limit, and reverse proxies, databases and connection pools hit it first; for services, set it with LimitNOFILE= in the systemd unit, because shell ulimit settings do not apply to them. fs.file-max is the system-wide ceiling and is rarely the limiting factor on modern kernels. Network sysctls should be checked against the documentation for your kernel version; copying tuning values without measuring their effect often does nothing or makes things worse.
Observability signals
For CPU, watch utilization together with saturation: run-queue length, scheduling latency and cgroup throttling. For memory, watch MemAvailable and memory pressure (PSI) rather than “used”. For disks, utilization alone is misleading on SSDs and RAID, so look at latency (await) and queue depth as well. For networks, TCP retransmits and listen-queue overflows (nstat, ss -s) reveal problems that bandwidth graphs hide. journalctl -k and dmesg give the kernel’s side of the timeline, including OOM kills, storage errors and NIC resets, and perf, async-profiler or eBPF-based profilers show combined kernel and user stacks.
Triage order
During an incident, first decide which resource is the bottleneck: CPU, memory, disk or network. Then check the limits that sit between the workload and the hardware, such as cgroup quotas, ulimit, cloud instance credits and Kubernetes requests and limits, because many apparent resource shortages are actually limits. Read the kernel log for OOM kills, I/O errors and driver resets. If the CPU is busy without obvious cause, use perf sched and a short strace to look for scheduling storms or hot syscall loops. Finally, line the timeline up against recent changes: deployments, kernel or image upgrades, and network path changes.
Troubleshooting map
| Symptom | Likely subsystem | Start with |
|---|---|---|
| Broad slowness, high steal time | CPU scheduling or hypervisor contention | vmstat 1 (st column), cloud host metrics |
| Plenty of available RAM but slow reads | Page cache misses, I/O wait | iostat -xz 1, the workload’s I/O pattern |
| Swap usage rising | Reclaim under memory pressure | vmstat 1 (si/so), sar -W, PSI |
Storms of open/stat | Dentry/inode caches, metadata I/O | strace -c, jobs that walk many files |
| OOM only inside a container | cgroup memory.max | journalctl -k, memory.events, Kubernetes limits |
| Latency spikes with moderate CPU | CPU quota throttling | cpu.stat (nr_throttled, throttled_usec) |
Where to go from here
The subsystems above explain why a server misbehaves; the next layer is what you expose to the network in the first place. Linux server security hardening covers SSH keys, sshd_config and forwarding controls, and is worth reading before a new host goes on the internet.