MongoDB Complete Guide | WiredTiger, Replication, Sharding, Indexes & Production Patterns

Core takeaways

This guide covers MongoDB features you use daily—indexes, Aggregation Pipeline, sharding, replication, transactions—and ties them to internals: WiredTiger (cache, checkpoints, journaling, MVCC), oplog and elections, sharded chunks and mongos routing, index design (ESR), and production patterns (connection pooling, retries, backups, TTL, monitoring).

From the field: On a cluster handling on the order of a billion log events per day, we improved query latency with sharding and shard-key review, and reduced write amplification and disk usage by dropping unused indexes and adopting partial indexes. Your numbers will differ—always measure on your workload.

Introduction: “MongoDB feels slow”

Real-world scenarios

Scenario 1: Queries take many seconds

At large scale, missing indexes, a bad shard key, and cache misses stack up. Use explain, index shape, and WiredTiger cache metrics together.

Scenario 2: Data does not fit one node

Replication increases availability but does not expand disk on a single shard. Consider sharding when data or throughput exceeds vertical limits.

Scenario 3: Complex aggregation

Express multi-step transforms in the Aggregation Pipeline; push selective $match stages early and align them with indexes where possible.

---

1. What is MongoDB?

Key characteristics

MongoDB is a document-oriented database storing BSON documents. It scales from a single node to replica sets and sharded clusters.

Main strengths:

• Flexible schema: documents evolve without heavy migrations
• Horizontal scaling: sharding (partitioning)
• Rich analytics: Aggregation Pipeline
• High availability: replica sets with automatic failover
• Storage engine: WiredTiger by default (compression, document-level concurrency)

---

2. WiredTiger storage engine internals

Since MongoDB 4.0 the default storage engine is WiredTiger. MMAPv1 was removed; production thinking should assume WiredTiger.

Cache and disk

WiredTiger keeps frequently used data and index pages in an internal cache (by default a fraction of RAM). Pages not in cache are read from disk—high fault rates increase latency. Watch cache usage, eviction, and read throughput via serverStatus, collStats, or your monitoring stack.

Checkpoints and the journal

WiredTiger periodically checkpoints dirty data to data files on disk. Durability between checkpoints is backed by a write-ahead style journal. writeConcern options such as j: true relate to journal durability (behavior depends on deployment).

MVCC and document-level concurrency

WiredTiger uses multi-version concurrency control (MVCC). Reads typically see a consistent snapshot; writes coordinate at document granularity. Long-running read transactions can keep old snapshots alive and pressure the cache—keep transactions short when possible.

Compression

Data and indexes are stored with compression (e.g., snappy, zlib, zstd—depending on configuration), trading CPU for disk I/O and space.

Operational notes

• Cache size: tune wiredTiger.engineConfig.cacheSizeGB (and related settings). Giving almost all RAM to MongoDB can starve the OS and other processes.
• Hot spots: extremely high-frequency updates to a single document increase contention; counters and monotonic hotspots often need sharding or bucketing strategies.

---

3. Indexing strategies (design and behavior)

Indexes use B-tree–style structures. Field order in compound indexes strongly affects which queries can use them efficiently.

The ESR rule

A common heuristic is ESR: Equality → Sort → Range.

1. Equality: fields matched with = (or equivalent selective predicates) first.
2. Sort: fields used for sorting next.
3. Range: range predicates ($gt, $lte, some $in patterns) often last.

Operators like $ne and $nin frequently prevent efficient index use—validate with explain("executionStats") on real queries.

Single-field, compound, partial, unique

db.users.createIndex({ email: 1 });
db.orders.createIndex({ userId: 1, createdAt: -1 });
db.users.createIndex({ email: 1 }, { unique: true });
db.users.createIndex(
  { email: 1 },
  { partialFilterExpression: { isActive: true } }
);

Partial indexes shrink the index and reduce write cost—must match the query filter.

Multikey indexes

Indexing an array field creates a multikey index (one index entry per array element). Very large arrays inflate index size and update cost.

Text and geospatial

db.articles.createIndex({ title: 'text', content: 'text' });
db.articles.find({ $text: { $search: 'mongodb tutorial' } });

db.places.createIndex({ location: '2dsphere' });
db.places.find({
  location: {
    $near: {
      $geometry: { type: 'Point', coordinates: [127.0276, 37.4979] },
      $maxDistance: 1000,
    },
  },
});

Check MongoDB limits (e.g., one text index per collection).

Covered queries and projection

If the index alone can satisfy the query, a covered query may be possible. Remember default projection of _id unless excluded.

db.users.createIndex({ email: 1, name: 1 });
db.users.find(
  { email: '[email protected]' },
  { email: 1, name: 1, _id: 0 }
);

Index count and write amplification

Every insert/update/delete must update all indexes on the collection. Periodically review unused indexes with $indexStats and drop what is safe.

---

4. Aggregation Pipeline

Stages run in order. Move $match early to shrink working sets and save CPU and memory.

Basic example

db.orders.aggregate([
  { $match: { status: 'completed' } },
  {
    $group: {
      _id: '$userId',
      totalAmount: { $sum: '$amount' },
      orderCount: { $sum: 1 },
    },
  },
  { $sort: { totalAmount: -1 } },
  { $limit: 10 },
]);

Richer example

db.orders.aggregate([
  {
    $match: {
      createdAt: {
        $gte: new Date('2026-01-01'),
        $lt: new Date('2027-01-01'),
      },
    },
  },
  {
    $group: {
      _id: {
        userId: '$userId',
        year: { $year: '$createdAt' },
        month: { $month: '$createdAt' },
      },
      revenue: { $sum: '$amount' },
      orders: { $sum: 1 },
    },
  },
  {
    $lookup: {
      from: 'users',
      localField: '_id.userId',
      foreignField: '_id',
      as: 'user',
    },
  },
  { $unwind: '$user' },
  {
    $project: {
      _id: 0,
      userName: '$user.name',
      year: '$_id.year',
      month: '$_id.month',
      revenue: 1,
      orders: 1,
    },
  },
  { $sort: { revenue: -1 } },
]);

Large sorts and groups may spill to disk—consider allowDiskUse: true. Memory limits depend on version and server parameters.

---

5. Sharding architecture

Sharding is horizontal partitioning. Data is split into chunks; mongos routes operations to the right shards; config servers store chunk metadata and cluster configuration.

Shard keys and distribution

• Monotonic keys alone (e.g., raw timestamp): risk hot spots on the latest chunk.
• Hashed shard keys: spread writes but may not match range queries and sorts.
• Compound shard keys: often combine low- and high-cardinality fields to balance distribution and query locality.

sh.shardCollection('mydb.users', { userId: 'hashed' });
// Risky: monotonic time-only key can hot-spot recent data
// sh.shardCollection('mydb.orders', { createdAt: 1 });

Chunks and the balancer

When chunks are uneven, the balancer migrates chunks between shards. Migrations can add latency and locking considerations—some teams schedule balancing windows.

Zone sharding

Use zones to pin data ranges to specific shards (region, tenant, etc.).

Minimal dev-style layout (conceptual)

mongod --configsvr --replSet configReplSet --port 27019
mongod --shardsvr --replSet shard1 --port 27018
mongos --configdb configReplSet/localhost:27019 --port 27017

In production, run config servers and each shard as replica sets, not single nodes.

---

6. Replication and the oplog

A replica set has one Primary and multiple Secondaries. Writes apply on the Primary; changes are written to the oplog (a capped collection) for Secondaries to replay.

Role of the oplog

• The oplog records idempotent operations for replay.
• If a secondary falls behind, replication lag grows; if it falls outside the oplog window, a full resync may be required.

Elections

When the Primary fails, members hold an election. Network partitions make quorum rules critical to avoid split brain—typically use an odd number of voting members (and arbiters only when you understand the trade-offs).

Read and write semantics

const client = new MongoClient(uri, {
  readPreference: 'secondaryPreferred',
});

Read concern and write concern balance consistency and durability. For example, w: "majority" waits for replication to a majority of nodes.

Replica set setup

rs.initiate({
  _id: 'rs0',
  members: [
    { _id: 0, host: 'localhost:27017' },
    { _id: 1, host: 'localhost:27018' },
    { _id: 2, host: 'localhost:27019' },
  ],
});
rs.status();

---

7. Transactions

MongoDB 4.0+ supports multi-document transactions on replica sets and sharded clusters (when configured).

const session = client.startSession();
try {
  await session.withTransaction(async () => {
    await db.collection('accounts').updateOne(
      { _id: fromAccount },
      { $inc: { balance: -100 } },
      { session }
    );
    await db.collection('accounts').updateOne(
      { _id: toAccount },
      { $inc: { balance: 100 } },
      { session }
    );
  });
} finally {
  await session.endSession();
}

Keep transactions short. Long transactions stress cache and storage layers and increase conflict or retry costs.

---

8. Performance optimization

Query shape

// Leading wildcard regex often cannot use indexes efficiently
db.users.find({ email: { $regex: /.*@example.com/ } });
db.users.find({ email: '[email protected]' });
db.users.find({ email: '[email protected]' }).explain('executionStats');

Projection

db.users.find({}, { name: 1, email: 1, _id: 0 });

Bulk writes

For bulk inserts, tune batch size and consider ordered: false when appropriate to reduce round-trips (driver-specific).

---

9. Production MongoDB patterns

Connection pooling

Use a single MongoClient per process with a shared connection pool. Size pools with instance count and concurrency. Creating a new client per request can exhaust file descriptors and memory.

Retryable writes

Enable retryable writes in the driver to smooth transient errors during failover.

Backup and restore

• mongodump/mongorestore: logical backup—plan for time and consistency.
• Storage snapshots: filesystem or cloud snapshots—understand crash-consistent vs point-in-time procedures for your platform.
• Continuous backup / PITR: common on managed services such as Atlas.

TTL and schema validation

db.sessions.createIndex({ expiresAt: 1 }, { expireAfterSeconds: 0 });

Use document validation to enforce required fields and types at the server.

Metrics to watch

• Opcounters, Replication lag, WiredTiger cache, Page faults, Connections, Queues
• Profiler or slow-query logs for outliers

Security and operations

• Least-privilege RBAC, TLS, network isolation
• Read release notes before upgrades

---

10. Hands-on example: log analytics

db.logs.insertMany([
  {
    userId: 1,
    action: 'login',
    ip: '192.168.1.1',
    timestamp: new Date('2026-04-01T10:00:00Z'),
  },
  {
    userId: 1,
    action: 'view_product',
    productId: 123,
    timestamp: new Date('2026-04-01T10:05:00Z'),
  },
  {
    userId: 2,
    action: 'purchase',
    amount: 99.99,
    timestamp: new Date('2026-04-01T10:10:00Z'),
  },
]);
db.logs.createIndex({ userId: 1, timestamp: -1 });
db.logs.createIndex({ action: 1 });
db.logs.aggregate([
  {
    $match: {
      timestamp: {
        $gte: new Date('2026-04-01'),
        $lt: new Date('2026-04-02'),
      },
    },
  },
  {
    $group: {
      _id: { userId: '$userId', action: '$action' },
      count: { $sum: 1 },
    },
  },
  {
    $group: {
      _id: '$_id.userId',
      actions: {
        $push: { action: '$_id.action', count: '$count' },
      },
      totalActions: { $sum: '$count' },
    },
  },
  { $sort: { totalActions: -1 } },
  { $limit: 10 },
]);

---

Summary and checklist

Key points

• WiredTiger: cache, checkpoints, journal, MVCC, compression—tie metrics to latency.
• Indexes: ESR, partial and multikey indexes, index hygiene.
• Sharding: chunks, mongos, balancer, hot spots, zones.
• Replication: oplog, lag, elections, read/write concerns.
• Production: pooling, retries, backups, TTL, monitoring.

Production checklist

• Validate index usage with explain on critical queries
• Review unused or duplicate indexes
• Alert on replication lag, oplog pressure, and elections
• For sharded clusters, review chunk distribution and balancer behavior
• Drill backup and restore regularly
• Patch, TLS, and least-privilege access

---

Related reading

• Redis Advanced Guide
• Elasticsearch Complete Guide
• Mongoose Complete Guide

---

Keywords covered in this post

MongoDB, WiredTiger, oplog, Replication, Sharding, mongos, Indexing, Aggregation, Production

Frequently asked questions (FAQ)

Q. MongoDB vs PostgreSQL—which should I choose?

A. MongoDB fits flexible document models and scale-out patterns. PostgreSQL excels at complex relational queries and strong SQL semantics. Decide from access patterns, consistency needs, and team skills.

Q. Can I rely on multi-document transactions?

A. Yes on supported versions, but prefer short transactions and designs that minimize cross-document coupling.

Q. Do many indexes hurt performance?

A. Yes—each write updates every index. Balance read optimization with write amplification and disk space.

Q. When is replication not enough?

A. Replication improves availability and read scaling options but does not split a single shard’s data volume or write throughput—then you need sharding or vertical scaling.