MongoDB Internals for Production: WiredTiger, ESR Indexes, Sharding and Oplog Replication

By 0xmote · Sep 10, 2026 · 15 min read · Updated Oct 15, 2026 Advanced tutorial

Key takeaways

A practical guide to MongoDB advanced features plus how the pieces fit together: WiredTiger cache and MVCC, oplog-driven replication and elections, sharded chunks and the balancer, ESR-style index design, and production patterns for connection pooling, backups, and monitoring.

This guide covers the MongoDB features you use daily (indexes, the Aggregation Pipeline, sharding, replication, transactions) and ties each one to the internals behind it: the WiredTiger storage engine, the oplog and elections, chunk routing through mongos, and compound index design. When “MongoDB feels slow”, the cause is usually a combination of a missing or badly ordered index, a working set that no longer fits in the WiredTiger cache, or a shard key that sends most writes to one shard. Knowing how those layers work makes explain output and server metrics much easier to read.

What is MongoDB?

MongoDB is a document-oriented database that stores BSON documents. It runs as a single node for development, as a replica set for availability, and as a sharded cluster when one replica set can no longer hold the data or the load.

Its main strengths follow from that design. Documents in a collection do not have to share a schema, so fields can be added without a migration step. Sharding partitions a collection across replica sets for horizontal scale. The Aggregation Pipeline handles grouping, joins ($lookup), and reshaping on the server. Replica sets fail over automatically. The default WiredTiger engine adds compression and document-level concurrency control.

WiredTiger storage engine internals

WiredTiger has been the default storage engine since MongoDB 3.2, and the older MMAPv1 engine was removed in 4.2, so any current deployment should be reasoned about in WiredTiger terms.

Cache and disk

WiredTiger keeps frequently used data and index pages in an internal cache. By default that cache is the larger of 50% of (RAM − 1 GB) or 256 MB; the rest of memory is left to the operating system’s file cache, which still holds compressed data files. Pages not in either cache are read from disk, so a working set that outgrows the cache shows up as rising read I/O and latency. Watch cache usage, eviction activity, and read throughput through db.serverStatus().wiredTiger, collStats, or your monitoring stack.

Checkpoints and the journal

WiredTiger periodically writes a checkpoint, a consistent snapshot of the data, to the data files (every 60 seconds by default). Writes between checkpoints are protected by the journal, a write-ahead log. A write concern with j: true asks the server to acknowledge only after the write is in the on-disk journal.

MVCC and document-level concurrency

WiredTiger uses multi-version concurrency control (MVCC). Reads see a consistent snapshot, and writes conflict only when they touch the same document. Long-running reads and transactions keep old versions pinned in the cache, which increases cache pressure, so keep transactions short.

Compression

Collection data is compressed with snappy by default (zlib and zstd are also available), and indexes use prefix compression. This trades some CPU for less disk I/O and space.

Operational notes

Change wiredTiger.engineConfig.cacheSizeGB deliberately. Giving almost all RAM to the WiredTiger cache starves the OS file cache and other processes, and in containers you should make sure the cache is sized against the container’s memory limit rather than the host’s.

Very frequent updates to a single document (a global counter, for example) serialize on that document. Spreading the counter over several documents, or bucketing writes, usually works better than trying to make the single document faster.

Indexing strategies (design and behavior)

MongoDB indexes are B-tree structures. In a compound index the order of fields decides which queries can use it efficiently.

The ESR rule

A common heuristic is ESR: Equality, then Sort, then Range.

Equality: fields matched exactly come first, because they narrow the scan to one contiguous part of the index.
Sort: fields used for sorting come next, so the matching entries are already in sort order and no in-memory sort is needed.
Range: range predicates ($gt, $lte, and so on) go last. Placing a range field before the sort field would break the ordering and force a blocking sort.

$in is handled as an equality when there is no sort on that index, but combined with a sort it can behave more like a range. Negations such as $ne and $nin are usually unselective and scan most of the index. In every case, confirm the plan with explain("executionStats") on real queries rather than relying on the rule alone.

Single-field, compound, partial, unique

db.users.createIndex({ email: 1 });
db.orders.createIndex({ userId: 1, createdAt: -1 });
db.users.createIndex({ email: 1 }, { unique: true });
db.users.createIndex(
  { email: 1 },
  { partialFilterExpression: { isActive: true } }
);

A partial index only contains documents that match its filter, which makes it smaller and cheaper to maintain. The planner uses it only when the query’s filter guarantees a subset of that expression; a query on email alone, without isActive: true, cannot use the partial index above.

Multikey indexes

Indexing an array field creates a multikey index with one entry per array element. Very large arrays inflate index size and the cost of every update to that document. A compound index can include at most one array field per document.

Text and geospatial

db.articles.createIndex({ title: 'text', content: 'text' });
db.articles.find({ $text: { $search: 'mongodb tutorial' } });

db.places.createIndex({ location: '2dsphere' });
db.places.find({
  location: {
    $near: {
      $geometry: { type: 'Point', coordinates: [127.0276, 37.4979] },
      $maxDistance: 1000,
    },
  },
});

A collection can have only one text index, although that index can cover several fields, as above. GeoJSON coordinates are written as [longitude, latitude], and $maxDistance with a GeoJSON point is in meters.

Covered queries and projection

If the index contains every field the query filters on and returns, MongoDB can answer from the index without fetching documents. _id is returned by default, so exclude it unless it is part of the index.

db.users.createIndex({ email: 1, name: 1 });
db.users.find(
  { email: '[email protected]' },
  { email: 1, name: 1, _id: 0 }
);

Index count and write amplification

Every insert and delete must update all indexes on the collection, and an update must touch every index containing a modified field. Review index usage periodically with the $indexStats aggregation stage and drop indexes that are not used. Check that the statistics cover a long enough period (they reset on restart) before dropping anything that might serve a monthly job.

Aggregation Pipeline

Stages run in order, so put $match as early as possible. An early $match can use an index and shrinks the number of documents every later stage has to process.

Basic example

db.orders.aggregate([
  { $match: { status: 'completed' } },
  {
    $group: {
      _id: '$userId',
      totalAmount: { $sum: '$amount' },
      orderCount: { $sum: 1 },
    },
  },
  { $sort: { totalAmount: -1 } },
  { $limit: 10 },
]);

Richer example

db.orders.aggregate([
  {
    $match: {
      createdAt: {
        $gte: new Date('2026-01-01'),
        $lt: new Date('2027-01-01'),
      },
    },
  },
  {
    $group: {
      _id: {
        userId: '$userId',
        year: { $year: '$createdAt' },
        month: { $month: '$createdAt' },
      },
      revenue: { $sum: '$amount' },
      orders: { $sum: 1 },
    },
  },
  {
    $lookup: {
      from: 'users',
      localField: '_id.userId',
      foreignField: '_id',
      as: 'user',
    },
  },
  { $unwind: '$user' },
  {
    $project: {
      _id: 0,
      userName: '$user.name',
      year: '$_id.year',
      month: '$_id.month',
      revenue: 1,
      orders: 1,
    },
  },
  { $sort: { revenue: -1 } },
]);

Blocking stages such as $group and $sort are limited to 100 MB of memory each. Beyond that they must spill to temporary files, which requires allowDiskUse: true on older versions; since MongoDB 6.0 the allowDiskUseByDefault parameter makes spilling the default. Note also that $unwind drops documents whose user array is empty, so orders from deleted users disappear from this report unless you add preserveNullAndEmptyArrays: true.

Sharding architecture

Sharding is horizontal partitioning. A sharded collection is split into chunks (contiguous ranges of shard-key values), mongos routes each operation to the shards that own the relevant ranges, and the config servers store that metadata.

Shard keys and distribution

The shard key decides both how writes spread and which queries can be routed to a single shard.

A monotonically increasing key on its own, such as a timestamp or an ObjectId, sends every new insert to the shard holding the highest range, which becomes a hot spot.
A hashed shard key spreads inserts evenly, but range queries and sorts on that field have to be sent to all shards.
A compound shard key that starts with a field queries usually filter on (a tenant or user id) and adds a more distinct second field often gives both even distribution and targeted queries.

sh.shardCollection('mydb.users', { userId: 'hashed' });
// Risky: monotonic time-only key can hot-spot recent data
// sh.shardCollection('mydb.orders', { createdAt: 1 });

Queries that do not include the shard key become scatter-gather operations across every shard, so pick the key from your most frequent queries, not only from write distribution.

Chunks and the balancer

The balancer moves data between shards when the distribution becomes uneven. Before MongoDB 7.0 it balanced by chunk count; from 7.0 it balances by the amount of data per shard. Migrations add I/O and some latency on both shards, so some teams restrict the balancer to an off-peak window.

Zone sharding

Zones pin ranges of the shard key to specific shards, which is useful for keeping a region’s or a tenant’s data on particular hardware.

Minimal dev-style layout (conceptual)

mongod --configsvr --replSet configReplSet --port 27019
mongod --shardsvr --replSet shard1 --port 27018
mongos --configdb configReplSet/localhost:27019 --port 27017

Each replica set still has to be initiated with rs.initiate() before mongos can use it. In production, config servers and every shard run as three-member (or larger) replica sets, not single nodes.

Replication and the oplog

A replica set has one primary and several secondaries. Writes go to the primary, which records each change in the oplog (local.oplog.rs), and secondaries pull and replay those entries.

Role of the oplog

Oplog entries are written in an idempotent form, so replaying an entry twice yields the same result. The oplog is size-bounded, so it covers a limited time window. If a secondary falls behind by more than that window, for example after a long outage, it can no longer catch up from the oplog and needs a full initial sync. Size the oplog for your peak write rate and the longest maintenance window you expect, and alert on replication lag.

Elections

When the primary becomes unreachable, the remaining members hold an election, and a candidate needs votes from a majority of voting members. That majority rule is what prevents two primaries in a network partition, and it is why replica sets use an odd number of voting members. An arbiter can provide the odd vote, but it holds no data, so with w: "majority" writes a primary-secondary-arbiter set cannot acknowledge majority writes while the secondary is down.

Read and write semantics

const client = new MongoClient(uri, {
  readPreference: 'secondaryPreferred',
});

Reading from secondaries can return slightly stale data, because secondaries apply the oplog asynchronously. Read concern and write concern control the trade-off between consistency and latency: w: "majority" waits until a majority of data-bearing members have the write, and readConcern: "majority" returns only data that has been majority-committed and therefore cannot be rolled back.

Replica set setup

rs.initiate({
  _id: 'rs0',
  members: [
    { _id: 0, host: 'localhost:27017' },
    { _id: 1, host: 'localhost:27018' },
    { _id: 2, host: 'localhost:27019' },
  ],
});
rs.status();

Transactions

Multi-document transactions are supported on replica sets since MongoDB 4.0 and on sharded clusters since 4.2.

const session = client.startSession();
try {
  await session.withTransaction(async () => {
    await db.collection('accounts').updateOne(
      { _id: fromAccount },
      { $inc: { balance: -100 } },
      { session }
    );
    await db.collection('accounts').updateOne(
      { _id: toAccount },
      { $inc: { balance: 100 } },
      { session }
    );
  });
} finally {
  await session.endSession();
}

withTransaction retries the callback on transient errors and retries the commit on unknown commit results, so the callback must be safe to run more than once. Transactions are aborted after 60 seconds by default (transactionLifetimeLimitSeconds), and long ones hold old snapshots in the WiredTiger cache, so keep them short. Often a single-document update with embedded data removes the need for a transaction entirely.

Performance optimization

Query shape

// An unanchored regex cannot use index bounds and scans the whole index
db.users.find({ email: { $regex: /.*@example\.com/ } });
// Exact match: uses the email index directly
db.users.find({ email: '[email protected]' });
db.users.find({ email: '[email protected]' }).explain('executionStats');

In the explain output, compare totalKeysExamined and totalDocsExamined with nReturned. When the examined counts are far larger than the number returned, the index is not selective enough for that query, or the wrong index was chosen.

Projection

db.users.find({}, { name: 1, email: 1, _id: 0 });

Returning only the fields you need reduces network transfer and driver deserialization cost, which matters for large documents.

Bulk writes

For bulk inserts, insertMany and bulkWrite batch operations into fewer round trips. With ordered: false, the server continues past individual failures (such as duplicate keys) and can apply the operations in any order, which is usually faster for independent writes.

Production MongoDB patterns

Connection pooling

Create one MongoClient per process and reuse it; it maintains a connection pool internally. Creating a new client per request opens new connections each time and can exhaust file descriptors on the server. When sizing maxPoolSize, remember that the server sees the pool size multiplied by the number of application instances.

Retryable writes

Retryable writes let the driver retry a write once after a network error or a primary failover. Drivers compatible with MongoDB 4.2+ enable them by default (retryWrites=true in the connection string).

Backup and restore

mongodump and mongorestore produce logical backups. They are simple, but slow for large data sets, and on a sharded cluster they do not give a consistent point-in-time snapshot by themselves. Filesystem or cloud volume snapshots are faster; they are consistent only if the journal is on the same volume or writes are paused during the snapshot. Managed services such as Atlas offer continuous backup with point-in-time restore. Whichever method you use, test restores regularly.

TTL and schema validation

db.sessions.createIndex({ expiresAt: 1 }, { expireAfterSeconds: 0 });

With expireAfterSeconds: 0, each document expires at the time stored in its expiresAt field. The TTL monitor runs about every 60 seconds, so documents are not deleted at the exact second they expire.

Document validation ($jsonSchema in the collection’s validator option) lets the server reject documents with missing required fields or wrong types.

Metrics to watch

The useful starting set is operation counters, replication lag, WiredTiger cache usage and eviction, connection count, and queued operations. The database profiler or the slow-query log (slowms) shows which individual queries are responsible when those numbers change.

Security and operations

Enable authentication with least-privilege roles, use TLS, and keep the database off public networks. Read the release notes and compatibility changes before every major-version upgrade, and upgrade one major version at a time.

Hands-on example: log analytics

db.logs.insertMany([
  {
    userId: 1,
    action: 'login',
    ip: '192.168.1.1',
    timestamp: new Date('2026-04-01T10:00:00Z'),
  },
  {
    userId: 1,
    action: 'view_product',
    productId: 123,
    timestamp: new Date('2026-04-01T10:05:00Z'),
  },
  {
    userId: 2,
    action: 'purchase',
    amount: 99.99,
    timestamp: new Date('2026-04-01T10:10:00Z'),
  },
]);
db.logs.createIndex({ userId: 1, timestamp: -1 });
db.logs.createIndex({ timestamp: 1 });
db.logs.aggregate([
  {
    $match: {
      timestamp: {
        $gte: new Date('2026-04-01'),
        $lt: new Date('2026-04-02'),
      },
    },
  },
  {
    $group: {
      _id: { userId: '$userId', action: '$action' },
      count: { $sum: 1 },
    },
  },
  {
    $group: {
      _id: '$_id.userId',
      actions: {
        $push: { action: '$_id.action', count: '$count' },
      },
      totalActions: { $sum: '$count' },
    },
  },
  { $sort: { totalActions: -1 } },
  { $limit: 10 },
]);

The first $match filters only on timestamp, so it needs an index that starts with timestamp; the { userId: 1, timestamp: -1 } index cannot serve a query that has no condition on userId. The two $group stages first count each action per user, then fold those counts into one document per user.

Frequently asked questions

Q. MongoDB vs PostgreSQL—which should I choose?

MongoDB fits data that is naturally shaped as documents and read together, and it has built-in scale-out through sharding. PostgreSQL is stronger for complex relational queries, joins across many tables, and strict SQL semantics. Decide from your access patterns, consistency requirements, and your team’s experience.

Q. Can I rely on multi-document transactions?

Yes, on supported versions, but prefer short transactions and schema designs that keep related data in one document where possible.

Q. Do many indexes hurt performance?

Yes. Each write updates every affected index, and each index uses cache and disk. Balance read speed against write cost.

Q. When is replication not enough?

Replication improves availability and lets you spread reads, but every member still holds the full data set and all writes go through one primary. When data size or write throughput exceeds what one replica set can handle, you need sharding or bigger hardware.