Replication and Sharding

Replication and Sharding are two fundamental techniques used to scale databases, but they solve different problems.

1. What Problem Are We Trying to Solve?

As applications grow, databases face two major challenges:

i. Too many read requests

Example:

1 million users reading posts
Product catalog searches
News websites

One database server may become overloaded.

ii. Too much data

Example:

Billions of records
Terabytes of storage
Single machine runs out of memory and disk

Different solutions exist for each problem:

Problem	Solution
Too many reads	Replication
Too much data	Sharding

2. Replication

Replication means keeping multiple copies of the same database on different servers.

          Users
             |
      ----------------
      |              |
   Read Request   Write Request
      |              |
      ↓              ↓
      Replica      Primary
      Replica
      Replica

The primary database receives writes.

Replicas contain copies of the primary database and mainly serve reads.

2.1 Primary-Replica Architecture

                 Write
Client --------------------> Primary DB
                                   |
                                   |
                             Replicate data
                                   |
                --------------------------------
                |              |               |
            Replica 1      Replica 2      Replica 3
                ↑              ↑               ↑
                |              |               |
               Read           Read            Read

Example:

Suppose Instagram has:

10 million users.
Most requests are reading feeds.

Without replication:

Users
   |
Single Database

All traffic hits one server.

With replication:

                Primary
                   |
          -------------------
          |        |        |
      Replica1 Replica2 Replica3

Reads are distributed among replicas.

2.2 Benefits of Replication

Increased read performance:

Instead of: 1000 reads/sec on one DB

you can do:

Primary     : writes
Replica1    : 500 reads/sec
Replica2    : 500 reads/sec
Replica3    : 500 reads/sec

High availability:

If a replica fails, Application still works.
```
Replica1 (fail)
Replica2 
Replica3
```
Backup and disaster recovery

Copies exist on multiple machines.

2.3 Replication Types

1. Synchronous Replication

Primary waits until replicas confirm.
```
Client
  |
Write
  |
Primary
  |
Replica confirms
  |
Success
```
Pros:
- Strong consistency
- No data loss
Cons:
- Slower writes

2. Asynchronous Replication

Primary immediately responds.

Write
  |
Primary
  |
Success returned
  |
Replication happens later
  |
Replica

Pros:

Faster

Cons:

Replica may lag behind.

2.4 Replication Lag

Suppose: User updates profile picture.

Primary: Image = new-image.png

Replica: Image = old-image.png

For a few milliseconds or seconds, replicas may contain stale data.

This is called Replication lag.

3. Sharding

Sharding means splitting data across multiple databases. Instead of copies, each server stores only part of the data.

Example:

Suppose there are 1 billion users.

One machine cannot store everything.

Split them:

Shard 1
User IDs 1 - 1M
 
Shard 2
User IDs 1M - 2M
 
Shard 3
User IDs 2M - 3M

Each shard contains different data.

Diagram

                Router
                   |
      --------------------------------
      |              |               |
   Shard 1        Shard 2         Shard 3
 (1-1M users)   (1M-2M users)   (2M-3M users)

3.1 Benefits of Sharding

Horizontal scaling:

Instead of: One huge server

you have: Small server, Small server, Small server
Faster queries:

Each database handles smaller datasets.
No single machine limit:

Can store billions or trillions of rows.

3.2 Sharding Strategies

1. Range-based Sharding

1-100000 → Shard1
100001-200000 → Shard2

Problem:

If new users mostly fall into one range: Shard2 becomes overloaded.

2. Hash-based Sharding
hash(user_id) % 3
Example:
hash(123) % 3 = 0
User goes to: Shard0

Distribution becomes more balanced.

3. Geographic Sharding

US users → US database
Europe users → EU database
Asia users → Asia database

Useful for reducing latency.

4. Replication vs Sharding

| Feature            | Replication  | Sharding                |
| ------------------ | ------------ | ----------------------- |
| Stores same data?  | Yes          | No                      |
| Solves             | Read scaling | Storage + write scaling |
| Number of copies   | Multiple     | Partitioned             |
| Write location     | Primary      | Each shard              |
| Read performance   | Improved     | Improved                |
| Data size capacity | Same         | Increased               |
| Complexity         | Medium       | High                    |

5. Can We Use Both Together?

Absolutely. Large companies combine both.

                    Router
                       |
      -------------------------------------
      |                  |                |
   Shard 1           Shard 2           Shard 3
      |                  |                |
   --------           --------         --------
   |      |           |      |         |      |
Primary Replica    Primary Replica  Primary Replica

Each shard has its own replicas.

This gives:

Horizontal scaling
High availability
Faster reads
Fault tolerance

6. Mental Model

Think of a library.

Replication: Making copies of the same book and placing them in multiple rooms.

Room A: Harry Potter
Room B: Harry Potter
Room C: Harry Potter

More people can read simultaneously.

Sharding: Splitting the books among rooms.

Room A → A-F
Room B → G-M
Room C → N-Z

No room contains every book, but together they hold the whole library.