Kafka is a distributed event streaming platform.

Event streaming is the practice of capturing real-time data from applications, databases and IoT devices and transporting it to various destinations for immediate processing and storage, or for real-time analysis and analytics reporting.

Imagine you're building an ecommerce website.

Client
   |
   v
API Server
   |
   +--> Save Order
   |
   +--> Send Email
   |
   +--> Update Inventory
   |
   +--> Notify Warehouse
   |
   +--> Generate Invoice

One API request does everything.

• Slow response
• If email fails → API fails
• Hard to scale
• Tight coupling(highly dependent on each other)

Now imagine there are 1000 orders per second.

Your API has to

1000 Database writes
1000 Emails
1000 Inventory updates
1000 Notifications
1000 Invoices

Everything becomes overloaded.

Instead of doing everything immediately:

  Client
    |
    v
   API
    |
    v
  Kafka

The API simply says:

"Someone placed an order."

Kafka stores that event.

Later...

  Kafka
    |
    +------> Email Service
    |
    +------> Inventory Service
    |
    +------> Analytics Service
    |
    +------> Warehouse Service
    |
    +------> Billing Service

Each service works independently.

Think of Kafka as a giant post office.

Instead of calling everyone directly,

you drop a letter into a mailbox.

Anyone interested can collect it later.

• Kafka does not process data.
• It only stores and delivers messages.

When someone uploads a photo:

• generate thumbnails
• notify followers
• update search
• run AI moderation

  Upload API
    |
    ↓
  Kafka

Then

Thumbnail Service
     |
     ↓
Notification Service
     |
     ↓
Search Index
     |
     ↓
AI Moderation
     |
     ↓
Analytics

Each service works separately.

Before writing any code, you only need to know these six terms.

Produces messages.

Example:

producer.send({
  topic: "orders",
  message: ...
})

Reads messages.

Kafka
  |
  ↓
Consumer

Think of a topic as a folder.

orders
 
payments
 
emails
 
notifications

Messages are grouped into topics.

Actual data.

{
  "userId": 10,
  "product": "Laptop"
}

Kafka server.

Broker
 
stores
 
messages

Every message gets a unique position.

Message A
 
Offset 0
 
Message B
 
Offset 1
 
Message C
 
Offset 2

Consumers remember the last offset they processed, allowing them to resume where they left off.

  Node API
      |
      | Producer
      |
      v
+-------------------+
|       Kafka       |
|                   |
| Topic: orders     |
+-------------------+
      |
      |
      +------------------+
      |                  |
      v                  v
Email Consumer     Inventory Consumer
      |                  |
      v                  v
Send Email        Update Stock

Notice that the producer doesn't know who is consuming the message. It only writes to the orders topic.

In a large system, an API may need to communicate with many services such as email, inventory, billing, notifications, and analytics. If it calls all of them synchronously, the response time increases because the client has to wait for every service to finish.

If one service is slow or unavailable, the entire request can fail. This creates tight coupling between services and makes the system difficult to scale and maintain. Instead, we use an event broker like Kafka to decouple these services.

Kafka solves the problem of asynchronous communication between services. Instead of services calling each other directly, producers publish events to Kafka, and consumers process those events independently.

This reduces coupling, improves scalability, increases fault tolerance, and allows multiple services to react to the same event without affecting the producer.

A producer is an application or service that publishes messages or events to a Kafka topic. It doesn't know who will consume the message; it only sends the event to Kafka.

Example:

Order API
↓
Kafka Topic: orders

The Order API is the producer.

A consumer is an application that reads messages from a Kafka topic and processes them. Multiple consumers can read the same events for different purposes.

Example:

Order Created
  ↓
Kafka
  ↓
Email Service
  ↓
Inventory Service
  ↓
Analytics Service

Each service acts as a consumer.

A topic is a logical category or channel where Kafka stores related messages. Producers write messages to a topic, and consumers subscribe to topics to receive those messages.

Examples:

orders
 
payments
 
notifications
 
users

Each topic contains events related to that domain.

An offset is the unique sequential position of a message within a partition. Kafka uses offsets to keep track of which messages a consumer has already processed, allowing consumers to resume reading from where they left off.

Example:

Offset 0 → Order 101
 
Offset 1 → Order 102
 
Offset 2 → Order 103

If a consumer crashes after processing Offset 1, it can restart from Offset 2.

Producers and consumers are loosely coupled because they communicate through Kafka instead of directly with each other. A producer only sends messages to a topic and doesn't know which consumers exist.

Consumers can be added, removed, or updated without changing the producer, making the system more flexible and scalable.

A partition is a subdivision of a Kafka topic. Instead of storing all messages in a single file, Kafka splits a topic into multiple partitions. This allows messages to be processed in parallel by multiple consumers, improving scalability and throughput.

Every message within a partition has a unique offset. By distributing partitions across brokers, Kafka can scale horizontally and allow multiple consumers to process messages in parallel.

Suppose your e-commerce application receives: 3000 orders/sec

One consumer may only process: 1000 orders/sec

That's too slow.

Instead:

Topic: orders
 
Partition 0:
1000 orders → Consumer A
 
Partition 1:
1000 orders → Consumer B
 
Partition 2:
1000 orders → Consumer C

Kafka can choose a partition in several ways:

• Round-robin if no key is provided (messages are distributed evenly).

• Key-based partitioning if a key is provided. Kafka hashes the key so all messages with the same key go to the same partition.

• Custom partitioner if you implement your own partitioning logic.

For example:

{
  key: "user-123",
  value: {
    orderId: 101
  }
}

All messages with the key "user-123" will always be written to the same partition, preserving their order.

In distributed systems, it's common for one service to need to communicate with many other services. Direct communication makes systems tightly coupled and harder to scale.

Kafka solves this by acting as a distributed event streaming platform where producers publish events to topics and consumers process those events asynchronously. This enables scalable, fault-tolerant, and loosely coupled communication between services