Apache Kafka Training

You can explain the architecture of event streaming platforms and initialize a local Apache Kafka broker instance using Docker Compose.

1. What event streaming is and why it matters90 min
   • event-driven architecture
   • publish-subscribe model
   • stateless request-response vs event streams
   • decoupling producers from consumers
   • asynchronous event propagation
   • event immutability principles
2. Messaging versus streaming90 min
   • message queues vs distributed commit logs
   • message consumption durability
   • replayability of events
   • message retention policies
   • point-to-point vs broadcast semantics
   • consumer offset management basics
3. Where Kafka is used in the real world90 min
   • real-time clickstream tracking
   • financial transaction processing
   • log aggregation pipelines
   • IoT telemetry ingestion
   • activity auditing streams
   • real-time fraud detection systems
4. Kafka's place in the data ecosystem90 min
   • decoupling microservices
   • feeding analytics data stores
   • integration with downstream batch systems
   • change data capture pipelines
   • event sourcing architectures
   • CQRS pattern implementation
5. Setting up Kafka locally120 min
   • docker-compose configuration
   • kafka-topics.sh CLI commands
   • verifying broker connectivity
   • KAFKA_BROKER_ID configuration
   • PLAINTEXT listener settings
   • cluster port binding and exposure
6. Command-line tools for topic administration90 min
   • kafka-topics.sh --create
   • kafka-topics.sh --describe
   • kafka-topics.sh --alter
   • kafka-console-producer.sh invocation
   • kafka-console-consumer.sh subscription
   • topic configuration overrides

Hands-on project: Spin up a single-node Kafka and ZooKeeper container stack via docker-compose.yml and verify broker health using kafka-topics.sh with custom partition configurations.

You can design topics with optimal partition counts and configure cluster replication for absolute durability under node failure.

1. Brokers, clusters and the log120 min
   • broker node roles
   • the append-only commit log
   • segment files and index structures
   • .log and .index file management
   • sequential disk I/O performance
   • zero-copy data transfer mechanics
2. Topics, partitions and offsets120 min
   • partition ordering guarantees
   • monotonic offset assignment
   • partition count sizing strategies
   • physical log layout on disk
   • partition distribution across brokers
   • throughput scaling via partitions
3. Replication and durability120 min
   • Leader and follower replicas
   • in-sync replicas (ISR)
   • unclean leader election configuration
   • replication factor trade-offs
   • follower fetch requests
   • high watermark offset tracking
4. ZooKeeper and KRaft mode120 min
   • Apache ZooKeeper metadata management
   • Kafka Raft metadata mode (KRaft)
   • migrating clusters away from ZooKeeper
   • quorum controller configuration
   • metadata log partition setup
   • cluster initialization via kafka-storage.sh
5. Retention and storage120 min
   • log.retention.bytes and log.retention.hours
   • log compaction policies
   • disk space management
   • cleaner thread configuration
   • segment rolling intervals
   • delete vs compact retention strategies
6. Under-the-hood broker internals90 min
   • request channel and network threads
   • IO threads pool configuration
   • controller node responsibilities
   • broker graceful shutdown process
   • metadata propagation delays
   • disk failure handling on brokers

Hands-on project: Create a multi-broker Kafka cluster with a replication factor of 3 and test data availability by killing a broker container.

You can write high-throughput Java producers featuring custom partitioners and robust acknowledgement configurations.

1. The Producer API90 min
   • ProducerRecord structure
   • KafkaProducer initialization
   • asynchronous send() with Callback handling
   • Future and RecordMetadata parsing
   • producer thread-safety model
   • serializer configuration
2. Keys, partitioning and ordering90 min
   • hashing keys for partition assignment
   • custom Partitioner interface
   • preserving ordering per key
   • DefaultPartitioner murmur2 hashing
   • null key round-robin distribution
   • sticky partitioner optimization
3. Acknowledgements and delivery guarantees90 min
   • acks=0, acks=1, and acks=all
   • min.insync.replicas setting
   • at-least-once delivery behavior
   • retriable exception handling
   • idempotent producer configuration
   • enable.idempotence=true internals
4. Batching, compression and throughput90 min
   • linger.ms and batch.size tuning
   • lz4, snappy, and gzip compression codecs
   • maximizing network throughput
   • buffer.memory configuration
   • max.block.ms timeout handling
   • producer performance profiling
5. Error handling and retries120 min
   • retries and max.in.flight.requests.per.connection
   • retriable vs non-retriable exceptions
   • dead-letter queue patterns
   • SerializationException management
   • BufferExhaustedException handling
   • producer interceptors for telemetry
6. Writing a robust Java producer client90 min
   • ProducerConfig property mapping
   • resource cleanup and close() handling
   • integration with SLF4J logging
   • producer metric monitoring
   • unit testing producers with MockProducer
   • handling broker disconnection gracefully

Hands-on project: Implement a Java producer using KafkaProducer API that publishes 100,000 keyed records with a custom Partitioner class and acks=all.

You can scale reading workloads horizontally across consumer groups while managing offset commits and rebalance listeners.

1. The Consumer API120 min
   • KafkaConsumer poll() loop mechanism
   • subscription vs assignment models
   • poll interval configuration
   • max.poll.interval.ms tuning
   • max.poll.records batch sizing
   • deserializer configuration
2. Consumer groups and scaling120 min
   • group.id coordination
   • partition assignment strategies (Range, RoundRobin, CooperativeSticky)
   • scaling consumers up to partition limits
   • group coordinator election
   • heartbeat thread mechanics
   • session.timeout.ms configuration
3. Offset management and commits120 min
   • auto.commit.enable risks
   • commitSync() vs commitAsync()
   • storing offsets externally
   • __consumer_offsets topic inspection
   • manual offset tracking per record
   • handling OffsetCommitFailedException
4. Rebalancing120 min
   • triggering events for consumer rebalances
   • ConsumerRebalanceListener interface
   • minimizing stop-the-world rebalance lag
   • cooperative rebalancing protocol
   • revoking and assigning partitions
   • state cleanup during rebalance
5. Delivery semantics: at-least-once and exactly-once120 min
   • idempotent consumer design
   • transactions with read-committed isolation level
   • exactly-once processing patterns
   • read-uncommitted vs read-committed modes
   • transaction coordinator interaction
   • consumer offset transaction commit
6. Building a resilient multi-threaded consumer90 min
   • worker thread pool pattern
   • queue-based record handoff
   • shutdown hook implementation
   • consumer exception handling loops
   • lag monitoring integration
   • integration testing with EmbeddedKafka

Hands-on project: Build a multi-threaded consumer application in Java that handles partition rebalances gracefully and implements manual asynchronous offset commits.

You can build real-time stream processing applications that execute aggregations, windowing operations, and stream joins.

1. Stream processing concepts120 min
   • event time vs processing time
   • streams vs tables duality
   • stateless versus stateful topologies
   • stream processing semantics
   • out-of-order event handling
   • watermarking in stream processing
2. The Kafka Streams API120 min
   • StreamsBuilder configuration
   • KStream and KTable abstractions
   • topology DSL fundamentals
   • GlobalKTable for reference data
   • Serdes configuration for types
   • topology optimization settings
3. Stateless and stateful operations120 min
   • filter, map, and flatMap transformations
   • groupBy and aggregate methods
   • RocksDB state store backing
   • branching streams with split()
   • foreach and peek operations
   • custom ValueTransformer implementation
4. Windowing, joins and aggregations120 min
   • Tumbling, Hopping, and Session windows
   • stream-stream and stream-table joins
   • handling late-arriving records
   • grace period configuration
   • foreign-key joins in KTables
   • reducing and aggregating state
5. Building a streaming application120 min
   • KafkaStreams lifecycle management
   • interactive queries over state stores
   • unit testing topologies with TopologyTestDriver
   • state store changelog topics
   • local state directory configuration
   • application ID and state isolation
6. Production management of Kafka Streams90 min
   • scaling stream threads horizontally
   • handling uncaught exception handlers
   • state store re-derivation from changelogs
   • monitoring stream lag and task metrics
   • rolling deployments of stream apps
   • standby replicas for fast failover

Hands-on project: Develop a Kafka Streams topology that reads raw user click events, computes sliding-window aggregates, and writes results to an output topic.

You can configure source and sink connectors to stream data between databases and Kafka without writing custom integration code.

1. Kafka Connect and connectors90 min
   • Connect runtime architecture (standalone vs distributed)
   • connector configuration lifecycle
   • task parallelism
   • connect-configs, connect-offsets, connect-status topics
   • distributed worker pool scaling
   • plugin path configuration
2. Source and sink connectors90 min
   • SourceConnector and SourceTask classes
   • SinkConnector and SinkTask classes
   • offset tracking in connectors
   • poll() and put() lifecycle methods
   • converter configuration (key.converter, value.converter)
   • error tolerance and DLQ settings
3. Moving data to and from databases90 min
   • Change Data Capture (CDC) with Debezium
   • polling vs log-based capture
   • writing records to Postgres via JDBC sink
   • database binlog and WAL parsing
   • auto-create tables in sink databases
   • primary key mapping in JDBC sink
4. Single Message Transforms90 min
   • SMT chaining in connector configs
   • masking fields and adding timestamps
   • extracting record keys
   • RegexReplace and Cast transformations
   • InsertField and TimestampRouter
   • writing custom SMT plugins
5. Building pipelines without custom code120 min
   • REST API administration of connectors
   • monitoring connector task health
   • handling transient connection failures
   • pausing and restarting connectors
   • connector configuration validation API
   • scaling connector tasks across workers
6. Debugging failing Kafka Connect jobs90 min
   • inspecting connect worker logs
   • resolving schema mismatch errors
   • handling database timeout exceptions
   • dead-letter queue message inspection
   • connector offset resetting via CLI
   • performance tuning batch sizes in connectors

Hands-on project: Deploy Debezium MySQL source connector and JDBC sink connector in distributed Kafka Connect mode to stream database table changes in real time.

You can enforce data contracts across distributed producers and consumers using Confluent Schema Registry and Apache Avro.

1. Why schemas matter in streaming90 min
   • schema drift in event pipelines
   • breaking changes in consumer applications
   • decoupling data producers from consumers
   • data contracts in event-driven systems
   • untyped JSON payload risks
   • schema enforcement at the broker boundary
2. The Schema Registry90 min
   • Confluent Schema Registry architecture
   • subject naming strategies (TopicNameStrategy, RecordNameStrategy)
   • REST API schema validation
   • schema ID caching in clients
   • global compatibility level settings
   • storage of schemas in Kafka compacted topics
3. Apache Avro and serialization90 min
   • Avro IDL and schema definition files
   • KafkaAvroSerializer and KafkaAvroDeserializer
   • binary payload efficiency
   • schema fingerprinting and magic bytes
   • codegen vs schema reference approaches
   • handling primitive and complex Avro types
4. Schema evolution and compatibility90 min
   • Backward, Forward, and Full compatibility modes
   • adding and removing optional fields safely
   • compatibility check API calls
   • default values in Avro schemas
   • transitively checking compatibility
   • handling breaking schema changes correctly
5. JSON and Protobuf120 min
   • Protocol Buffers syntax in Kafka
   • JSON Schema validation
   • comparing serialization overheads
   • Protobuf serializer configuration
   • JSON schema registry integration
   • choosing the right format for enterprise pipelines
6. Enforcing data governance in production90 min
   • CI/CD validation of schema files
   • preventing incompatible schema registration
   • schema registry security and ACLs
   • multi-environment schema promotion
   • monitoring registry availability and latency
   • handling schema migration failure modes

Hands-on project: Set up a local Confluent Schema Registry instance and configure Java producers to serialize messages using Avro with Backward compatibility rules.

You can tune Kafka cluster performance parameters, configure Prometheus metrics scraping, and secure brokers with TLS and SASL.

1. Replication and fault tolerance90 min
   • preferred replica election
   • partition reassignment tool execution
   • handling disk failures on broker hosts
   • unclean leader election recovery
   • leader balancing across brokers
   • replica verification commands
2. Throughput and latency tuning90 min
   • JVM garbage collection tuning for Kafka brokers
   • OS page cache and network buffer sizing
   • zero-copy data transfer mechanics
   • socket buffer size configuration
   • num.io.threads and num.network.threads tuning
   • disk I/O scheduler selection
3. Monitoring and metrics90 min
   • JMX metrics exporter configuration
   • key consumer lag metrics (records-lag-max)
   • alerting on under-replicated partitions
   • Prometheus scraping setup for Kafka
   • Grafana dashboard provisioning
   • tracking request queue time and local time
4. Security: authentication and encryption90 min
   • TLS encryption for data in transit
   • SASL/SCRAM and mTLS authentication
   • Access Control Lists (ACLs) for topic authorization
   • keystore and truststore generation via keytool
   • SASL/PLAIN and SASL/SCRAM-SHA-512 setup
   • super-users and cluster authorization rules
5. Operating Kafka in production120 min
   • rolling upgrades of Kafka clusters
   • backup and disaster recovery strategies
   • capacity planning for peak load
   • cluster expansion and broker decommissioning
   • log directory migration
   • incident response for broker partitions offline
6. Disaster recovery and multi-datacenter replication90 min
   • Cluster Linking configuration in Confluent
   • active-passive disaster recovery setups
   • offset synchronization across clusters
   • failover testing procedures
   • latency considerations in WAN replication
   • split-brain prevention strategies

Hands-on project: Configure SASL/SCRAM authentication on a Kafka cluster and set up a Prometheus and Grafana dashboard tracking broker request latency and disk saturation.

You can integrate Kafka with Apache Spark streaming jobs and deploy managed streaming infrastructure on AWS MSK and Confluent Cloud.

1. Feeding real-time data into Apache Spark90 min
   • Spark Structured Streaming Kafka source
   • offset management in Spark queries
   • micro-batch processing of event streams
   • watermarking in Spark streaming
   • fault-tolerant stateful processing
   • writing Spark output back to Kafka topics
2. Connecting Kafka to data stores90 min
   • streaming data into Snowflake and ClickHouse
   • idempotent sink design patterns
   • handling write failures gracefully
   • batch insertion optimization
   • handling duplicate records in analytics stores
   • schema synchronization with data warehouses
3. Managed Kafka: Confluent and MSK90 min
   • AWS Managed Streaming for Apache Kafka (MSK)
   • Confluent Cloud serverless clusters
   • managing VPC peering and security groups
   • IAM authentication with AWS MSK
   • managed connector provisioning in cloud
   • cost optimization for cloud Kafka workloads
4. Event-driven microservices patterns90 min
   • Transactional Outbox pattern
   • CQRS with Kafka event sourcing
   • saga orchestration across services
   • handling eventual consistency
   • compensating transaction design
   • idempotent service consumer implementation
5. Designing a streaming architecture120 min
   • multi-datacenter mirroring with Cluster Linking
   • disaster recovery topologies
   • capacity sizing for enterprise data volumes
   • network bandwidth estimation
   • topic governance and naming standards
   • multi-tenant cluster isolation strategies
6. Serverless streaming and modern event mesh90 min
   • serverless Kafka paradigms
   • event-driven architecture at enterprise scale
   • integrating Kafka with serverless functions (AWS Lambda)
   • schema registry scaling in cloud environments
   • observability mesh across distributed services
   • future trends in event streaming technology

Hands-on project: Connect a Kafka topic to an AWS MSK cluster and run a Spark Structured Streaming job to consume and aggregate events in real time.

You can perform load testing on Kafka pipelines, write robust integration tests with EmbeddedKafka, and execute zero-downtime maintenance operations.

1. Load testing Kafka producers and consumers90 min
   • kafka-producer-perf-test.sh execution
   • kafka-consumer-perf-test.sh benchmarking
   • measuring end-to-end pipeline latency
   • identifying network and disk bottlenecks
   • simulating high throughput traffic spikes
   • interpreting performance test metrics
2. Integration testing event-driven services90 min
   • Spring Kafka Test and EmbeddedKafka
   • Testcontainers for Kafka and Schema Registry
   • mocking external message brokers
   • asserting message consumption in unit tests
   • testing consumer offset commit logic
   • verifying idempotency in tests
3. CI/CD pipelines for streaming applications90 min
   • automated topic creation in CI environments
   • schema validation checks in build pipelines
   • deploying Kafka Streams apps via Docker
   • blue-green deployment of consumer services
   • automated rollback strategies for stream apps
   • managing configuration across environments
4. Failure injection and resilience testing90 min
   • chaos engineering for event-driven systems
   • simulating broker network partitions
   • testing consumer behavior during rebalances
   • verifying dead-letter queue routing under failure
   • recovering from corrupted consumer offsets
   • testing producer retry storms
5. Operational runbooks and troubleshooting90 min
   • writing incident response runbooks
   • investigating consumer lag spikes
   • resolving stuck partition rebalances
   • debugging broker out-of-memory errors
   • analyzing JVM heap dumps of Kafka brokers
   • emergency cluster scaling procedures
6. Cost engineering and resource optimization90 min
   • calculating storage costs for retained logs
   • optimizing network cross-AZ transfer fees
   • right-sizing broker instance types
   • compressing payloads to reduce storage footprint
   • monitoring resource utilization via metrics
   • fintech and enterprise cost governance

Hands-on project: Write comprehensive integration tests for a Kafka consumer service using EmbeddedKafka and simulate peak load using kafka-producer-perf-test.sh.

You can establish enterprise data governance frameworks, enforce security compliance policies, and manage multi-team Kafka ownership models.

1. Multi-team ownership and self-service Kafka90 min
   • decentralized topic creation models
   • GitOps workflows for topic management
   • Enabling self-service via developer portals
   • ownership metadata tagging on topics
   • managing producer and consumer contracts
   • cross-team coordination for breaking changes
2. Data governance and compliance in streaming90 min
   • GDPR and data deletion requirements in Kafka
   • handling right-to-be-forgotten via log compaction
   • PII data masking in event streams
   • audit logging for topic access
   • compliance reporting for financial data
   • data lineage tracking across pipelines
3. Enterprise security frameworks90 min
   • integrating Kafka with enterprise LDAP/ActiveDirectory
   • OAuth2 and OIDC token authentication
   • secrets management with HashiCorp Vault
   • certificate rotation procedures for mTLS
   • fine-grained authorization policies
   • vulnerability scanning of connector plugins
4. Migration strategies for legacy systems90 min
   • strangler fig pattern for event-driven migration
   • dual-writing from legacy databases to Kafka
   • backfilling historical data into Kafka topics
   • zero-downtime cutover strategies
   • validating data parity during migration
   • retiring legacy message queues safely
5. Capacity planning and enterprise scaling90 min
   • forecasting storage and throughput growth
   • hardware sizing formulas for brokers
   • network topology design for global clusters
   • disaster recovery SLA definition
   • multi-region deployment architecture
   • vendor evaluation for managed platforms
6. Architecture review and technical governance90 min
   • architecture decision records (ADRs) for Kafka
   • reviewing topic schema proposals
   • evaluating partition scaling requests
   • cost-benefit analysis of real-time vs batch
   • establishing internal Kafka centers of excellence
   • conducting production readiness reviews

Hands-on project: Design an enterprise Kafka governance document specifying topic naming conventions, ACL access control matrices, and data retention policies for a regulated financial institution.

You can replicate a cluster across regions and state the data loss a failover would cost.

1. What failure are you planning for90 min
   • Broker vs AZ vs region loss
   • RPO and RTO for streams
   • Cost of active-active
2. Replicating between clusters90 min
   • MirrorMaker 2 topology
   • Topic and config sync
   • Offset translation
3. Failing consumers over90 min
   • Consumer group migration
   • At-least-once duplicates
   • Idempotent consumers
4. Backup and rebuild90 min
   • Cluster metadata backup
   • Rebuilding from a compacted topic
   • Verified recovery drill

Hands-on project: Replicate a topic to a second cluster with MirrorMaker 2, document offset translation for consumers, then run a failover of a consuming application to the secondary and measure the duplicate and lost message counts.

You can build, deploy, and monitor an end-to-end real-time event streaming pipeline processing millions of records with fault tolerance.

1. Capstone architecture and system design90 min
   • defining data flow requirements
   • choosing topic partitioning and replication strategy
   • designing Avro schemas for events
   • mapping producer and consumer topologies
   • identifying potential bottleneck points
   • documenting system architecture diagrams
2. Building data producers and ingestion sources90 min
   • implementing high-throughput event generators
   • configuring idempotent producer settings
   • handling connection drops and retries
   • logging metrics and telemetry
   • testing producer throughput locally
   • integrating Debezium CDC for database changes
3. Implementing stream processing logic90 min
   • setting up Kafka Streams application topology
   • writing windowed aggregations and joins
   • managing RocksDB state stores
   • handling late arriving data gracefully
   • testing stream logic with TopologyTestDriver
   • handling uncaught exception handlers
4. Connecting to sinks and downstream stores90 min
   • deploying Kafka Connect clusters
   • configuring JDBC sink connectors
   • handling dead-letter queue routing
   • verifying data consistency in PostgreSQL
   • monitoring connector task health
   • optimizing batch insertion performance
5. Containerization and local orchestration90 min
   • writing multi-stage Dockerfiles for services
   • orchestrating containers with Docker Compose
   • managing environment variables and secrets
   • configuring health checks for services
   • testing end-to-end flow locally
   • troubleshooting container networking issues
6. Monitoring, logging and final presentation90 min
   • setting up Prometheus and Grafana dashboards
   • monitoring consumer lag and broker metrics
   • writing operational runbooks
   • preparing project documentation and README
   • code review and architecture defense
   • demonstrating pipeline failover recovery

Hands-on project: Build and deploy a complete real-time event processing system consisting of a custom Java producer, Kafka broker cluster, Kafka Streams aggregator, and PostgreSQL sink connector.