Redis Internals and Usage: The Event Loop, Encodings, RDB vs AOF, Replication and Cluster
Key takeaways
Hands-on Redis (data types, caching, Pub/Sub, Streams) plus how it works inside: the ae event loop, string and zset implementations, RDB snapshots vs AOF logs, PSYNC replication, Sentinel, Cluster, and production-grade patterns.
Core takeaways
Redis is an in-memory data structure store used for caching, sessions, messaging and more. This guide pairs the everyday APIs with the internals that explain its behavior in production: the single-threaded event loop, the physical encodings behind each type, RDB versus AOF persistence, PSYNC replication, and the operational side (Sentinel, Cluster, memory policies, observability).
The one fact that explains most Redis incidents is that commands run one at a time on a single thread. Redis is fast, but KEYS, a large SORT or SUNION, or any other O(N) command on a big key monopolizes that thread and stalls every other client until it finishes. Keep that in mind when reading SLOWLOG or a latency spike.
What is Redis?
Redis (Remote Dictionary Server) is a memory-backed key–value store whose values are typed data structures: strings, hashes, lists, sets, sorted sets, streams, bitmaps, HyperLogLog and more. Because the server understands these types, operations such as “increment this counter” or “add to this sorted set and return the top ten” happen atomically on the server instead of as read-modify-write cycles in the application.
Common uses follow from those types. It caches database and API responses, stores sessions and token deny lists, carries messages through Pub/Sub and Streams, implements rate limits with atomic counters, ranks leaderboards with sorted sets, and provides simple locks with SET key value NX PX ttl (which needs careful design if correctness depends on it).
Benchmarks can reach hundreds of thousands of operations per second on one instance, but the command mix, payload size, persistence settings and replication all change what you will see in practice.
The single-threaded event loop
Why one thread for commands
Redis executes commands on a single main thread. That removes lock contention inside the data structures, keeps the code simple, and makes every single command naturally atomic. Thousands of clients still share the server because the thread never waits on one socket; it multiplexes all of them through one event loop.
The ae loop
Redis uses its own small event library, ae, which sits on epoll on Linux, kqueue on BSD and macOS, or select as a fallback. When a client socket becomes readable, Redis reads and parses the request, runs the command handler, and queues the reply to be written when the socket is writable. Timers for expiring keys, replication heartbeats and similar periodic work run on the same loop.
I/O threads (Redis 6+)
Redis 6 added optional I/O threads (io-threads) that parallelize reading requests and writing replies. Command execution still happens on the main thread. If your bottleneck is network handling for many clients with small commands, I/O threads can help; if the main thread is busy executing expensive commands, they will not.
What blocks the loop
Anything that keeps the main thread busy delays every client. The usual causes are slow commands (KEYS, large SORT, unions or deletes of huge collections), synchronous disk work under appendfsync always, the fork() call itself for BGSAVE or AOF rewrite on a large dataset (copying page tables takes time proportional to memory size), and long Lua scripts or functions, which run to completion without letting other commands in.
Deleting a key with millions of elements is a less obvious example: DEL frees every element synchronously. UNLINK (or lazyfree-lazy-* settings) hands the freeing to a background thread instead.
Operational tools
SLOWLOG GET lists commands whose execution time exceeded slowlog-log-slower-than. Note that it measures execution only, not network or queueing time, so a client can see high latency while the slow log is empty. LATENCY DOCTOR and LATENCY LATEST explain stalls from events like fork or fsync, but only after you enable the latency monitor with CONFIG SET latency-monitor-threshold <ms>.
Data structure implementations (encodings)
Each logical type has more than one physical representation. Small values use compact sequential encodings that save memory; when a value grows past configurable thresholds, Redis converts it to a hash table or skiplist with better complexity for large sizes. The names change between releases (ziplist was replaced by listpack in Redis 7), so it is more useful to remember the idea than the exact names.
SDS (Simple Dynamic String)
String values are stored as SDS rather than NUL-terminated C strings. SDS keeps the length in a header, so length lookups are O(1), appends can reuse preallocated space, and values are binary-safe: a string can contain zero bytes, which is why you can store serialized protobufs or images in Redis.
Hash
Small hashes are stored as a listpack, a single contiguous block of field/value pairs. Lookups in it are linear scans, which is cheap for a few dozen entries. Once a hash exceeds hash-max-listpack-entries (128 by default) or a value exceeds hash-max-listpack-value (64 bytes), it converts to a real hash table. Converting is one-way, so a hash that briefly grew stays in the larger encoding.
List
Lists are quicklists: a doubly linked list whose nodes are listpacks. This keeps pointer overhead low while still allowing cheap pushes and pops at both ends. Index access in the middle of a long list is O(N).
Sorted set
Small sorted sets use a listpack too. Larger ones combine a hash table (member to score, for O(1) ZSCORE) with a skiplist ordered by score (for O(log N) inserts and range queries). Holding both structures costs memory, but it is why ZADD plus ZREVRANGE on a leaderboard stays fast at millions of members.
Set
Sets of small integers use an intset, a sorted array of integers. Since Redis 7.2, small sets of non-integer members can use a listpack. Larger sets become hash tables.
Inspecting encodings
OBJECT ENCODING key shows the representation of a key. Encoding thresholds and defaults change between versions, so check the release notes when upgrading if memory usage matters to you.
Install and connect
docker run -d --name redis -p 6379:6379 redis:7-alpinenpm install redis// lib/redis.ts
import { createClient } from "redis";
export const redis = createClient({
url: process.env.REDIS_URL || "redis://localhost:6379",
socket: {
reconnectStrategy: (retries) => Math.min(retries * 50, 500),
},
});
redis.on("error", (err) => console.error("Redis Client Error", err));
await redis.connect();Register the error listener before connecting. Without it, a dropped connection emits an unhandled error event and crashes the Node.js process.
Core data types (APIs)
String
await redis.set("user:1:name", "John");
const name = await redis.get("user:1:name");
await redis.setEx("session:abc", 3600, "user-data");
await redis.incr("page:views");Hash
await redis.hSet("user:1", {
name: "John",
email: "[email protected]",
age: "30",
});
const user = await redis.hGetAll("user:1");Sorted set (leaderboards)
await redis.zAdd("leaderboard", { score: 100, value: "user1" });
const top = await redis.zRangeWithScores("leaderboard", 0, 2, { REV: true });Caching strategies
Cache-aside
async function getUser(id: number) {
const cacheKey = `user:${id}`;
const cached = await redis.get(cacheKey);
if (cached !== null) return JSON.parse(cached);
const user = await db.user.findUnique({ where: { id } });
await redis.setEx(cacheKey, user ? 3600 : 60, JSON.stringify(user));
return user;
}The application reads the cache first and fills it from the database on a miss. Two details matter. Comparing against null rather than truthiness lets an empty string be a valid cached value. And caching a missing user (null) for a short time stops repeated requests for nonexistent IDs from hitting the database every time; a shorter TTL for negative results keeps a newly created user from being hidden for long.
Combine the TTL with explicit deletion of the key when the underlying row changes. When a popular key expires, many requests miss at once and all hit the database (a cache stampede); a per-key lock, request coalescing in the application, or refreshing slightly before expiry avoids that.
Pub/Sub and Streams
Pub/Sub is fire-and-forget: a message goes to the subscribers connected at that moment, and if none are connected it is gone. A subscriber that disconnects for a second misses whatever was published during that second. When messages must survive consumer restarts or be processed exactly by one worker of a group, use Streams with consumer groups, which store entries, track per-consumer pending messages, and let you acknowledge or reclaim them.
Persistence: RDB and AOF
RDB snapshots
RDB writes a compact binary snapshot of the whole dataset. BGSAVE forks a child process that writes the snapshot while the parent keeps serving clients. The child and parent share memory pages through copy-on-write; every page the parent modifies during the save gets copied. Under heavy writes, RSS can therefore grow substantially during a snapshot, which is why a Redis host needs memory headroom beyond maxmemory.
RDB files restore quickly and are easy to back up, but everything written since the last snapshot is lost on a crash.
AOF (append-only file)
AOF logs every write command. The appendfsync setting controls how often the log is flushed to disk. With always, Redis fsyncs on every write, which gives the strongest durability and the lowest throughput. With everysec, a background thread fsyncs once per second, so a crash typically loses about a second of writes (up to around two seconds if the disk is stalling). With no, Redis leaves flushing to the operating system, which is fastest and has the largest loss window.
The log grows forever, so Redis periodically rewrites it into the minimal set of commands that recreates the current data. The rewrite also uses fork(), with the same memory considerations as RDB, and needs disk space for the new file. Since Redis 7, AOF is stored as a base file plus incremental files described by a manifest in the appendonlydir directory.
Choosing
A pure cache usually runs without persistence or with occasional RDB snapshots for faster warm restarts. If Redis holds data you cannot rebuild, AOF with everysec plus periodic RDB backups copied off the host is the common setup. Whatever you choose, restore a backup into a test instance from time to time; a backup you have never restored is an assumption.
Replication
Full sync and partial resync
A new replica, or one that cannot resume, performs a full sync: the master produces an RDB snapshot, streams it to the replica, and then forwards the write commands that arrived meanwhile. After that, the replica applies the master’s write stream continuously.
Each side tracks a replication offset. The master also keeps a replication backlog, a fixed-size ring buffer of recent writes. When a replica reconnects after a short disconnect, it sends PSYNC with its replication ID and offset; if those bytes are still in the backlog, the master sends only the missing part (a partial resync). Since Redis 4 (PSYNC2), a replica promoted to master keeps the old replication ID as well, so other replicas can partially resync against the new master after a failover.
Replication backlog
If a replica is disconnected longer than the backlog can cover, partial resync fails and a full sync is required, which costs another fork on the master and a full data transfer. The default repl-backlog-size is only 1 MB. Estimate it as peak write throughput in bytes per second times the longest disconnect you want to survive without a full sync.
Monitoring
INFO replication on the master shows each replica’s offset; the difference from master_repl_offset is the lag in bytes. A replica that never catches up usually points to a saturated network link or a replica that cannot apply writes fast enough. Also watch for the replica’s output buffer on the master hitting client-output-buffer-limit replica, which disconnects the replica and can cause a loop of full syncs on large datasets.
Diskless replication
With repl-diskless-sync yes (the default since Redis 7), the master streams the RDB directly to the replica’s socket instead of writing it to disk first. This helps when the master’s disk is slow and the network is fast. Replicas can also load the incoming RDB without writing it to disk via repl-diskless-load.
Production patterns
Sentinel (HA)
Sentinel processes monitor the master and replicas, agree on failure, promote a replica, and tell clients the new master’s address. A single Sentinel marks a master as subjectively down (SDOWN) when it stops responding; when enough Sentinels to reach the configured quorum agree, it becomes objectively down (ODOWN) and a failover starts. Clients must ask Sentinel for the current master instead of hardcoding an address, which is what Sentinel-aware client libraries do. Run at least three Sentinels on separate hosts so a single host failure cannot block the vote.
Replication is asynchronous, so writes acknowledged by the old master but not yet replicated are lost when a replica is promoted. WAIT can make a client wait for replicas to acknowledge a write, but it does not turn Redis into a strongly consistent system.
Cluster (sharding)
Redis Cluster splits the key space into 16,384 hash slots, computed as CRC16 of the key modulo 16384, and assigns ranges of slots to master nodes. Clients that receive a MOVED redirect update their slot map and go to the right node. Hash tags change what gets hashed: only the part inside the first {...} counts, so {user123}:profile and {user123}:cart land in the same slot and can be used together in multi-key commands. Putting too many keys under one tag puts them all on one shard and recreates the hot-node problem that sharding was meant to solve.
Memory policies
Set maxmemory and choose maxmemory-policy according to what Redis holds. The default noeviction rejects writes once memory is full, which is right when Redis stores data you cannot lose. For a cache, allkeys-lru or allkeys-lfu evicts the least recently or least frequently used keys. The volatile-* policies evict only keys that have a TTL, which lets one instance mix cache entries with keys that must stay; if no key has a TTL, they behave like noeviction. Redis approximates LRU and LFU by sampling a few keys (maxmemory-samples) rather than tracking exact order.
Security and operations
Use ACLs (Redis 6+) to give each application its own user with only the commands and key patterns it needs, and enable TLS when traffic crosses untrusted networks. Never expose Redis directly to the internet. Disabling commands with rename-command still works, but ACLs are the more flexible way to block FLUSHALL, KEYS or CONFIG for application users. Alert on memory use against maxmemory, evictions, replication lag and latency, and rehearse failovers so client reconnect behavior is known before a real outage.
Pipelines and transactions
A pipeline sends several commands without waiting for each reply, saving round trips. A transaction (MULTI/EXEC) additionally guarantees the queued commands run back to back without other clients’ commands in between. In node-redis, multi() builds a transaction; to send the same batch as a plain pipeline, call execAsPipeline() instead of exec().
// MULTI/EXEC transaction
await redis.multi().set("key1", "value1").set("key2", "value2").exec();
// Pipeline only: fewer round trips, no MULTI/EXEC wrapper
await redis.multi().set("key1", "value1").set("key2", "value2").execAsPipeline();Commands issued concurrently from the same client (for example with Promise.all) are also pipelined automatically by node-redis.
A Redis transaction does not roll back: if one queued command fails at execution time, the others still apply. For read-then-write logic, combine WATCH with MULTI/EXEC (the transaction aborts if a watched key changed) or move the logic into a Lua script or function, which runs atomically on the server. Keep scripts short, since they block the main thread while they run, and in Cluster all keys they touch must be in one slot.
Redis or Memcached is a common question here. Memcached is a simpler multi-threaded string cache and can still make sense for very large, plain key–value caching. Redis is the more common default because the data types, persistence, replication and scripting cover far more use cases.