joshita / dev

Published

- 2 min read

Apache Kafka

Apache Kafka Streaming platform Big Data logs Pipeline
img of Apache Kafka

Apache kafka

What is apache kafka and its usecases Kafka is an event streaming platform for building real time data pipeline application Usecases : Real time analytics(website activity tracking) : User click activities are sent as logs to kafka server which can be used in analytic, here producer can be spark/apache beam/java application log aggregration and monitoring - processing millions of messages per second Streaming processing application - provides message streaming to kafka topic

Key components in Kafka

  1. topic
  2. partitions
  3. Brokers
  4. Producers
  5. Consumers
  6. Consumer group
  7. zookeeper

How does kafka ensure fault tolerance

  1. Replication
  2. Leader follower
  3. ISR - In sync replica
  4. Acknowlegement - configurable acknowledgement levels acks=0, acks=1, acks=all

How does kafka partitioning work

  1. Topic divided into partitions
  2. producer determine partitions key based partition round robin Partition
  3. Each partition is processed by a single consumer in a consumer group
  4. formula to generate kafka partitions: (Throughput/per second * no of consumers) P = Max(T/M, C, R) T: Required throughput in messages per second or bytes per second. M: Maximum throughput per partition (based on producer/consumer performance). C: Number of consumer instances (or threads) for concurrent processing. R: Replication factor (typically greater than 1 for fault tolerance).

What are kafka delivery semantics

  1. exactly once - messages are delivered to consumers only once, no duplicates, no losses producers and consumers use kafka IDEMPOTENET data producer ensure no duplicate data is written consumers commit the transactions atomically
  2. At most once - messages are delivered to consumers at most once meaning no reties occur. producer sends messages to broker without ack, consumers processes the messages and commits the offset before processing it as successful.
  3. at-least once - messages are delivered to consumers at least once, if that fails retry happens which may lead to duplicates, producers wait for acknowledgement from the consumers and consumers commit the offset only after successful write example : banking transations, applications that can handle data loss