10 Best Apache Kafka Courses On Udemy (2024)

Apache Kafka Series - Learn Apache Kafka for Beginners v3

This course provides a comprehensive introduction to Apache Kafka, covering both theoretical concepts and hands-on practical applications.

You will start by learning the fundamentals of Kafka, including topics, partitions, offsets, producers, consumers, brokers, and replication.

The course guides you through setting up Kafka on different platforms like Mac, Linux, and Windows, with detailed instructions for each environment.

The command-line interface (CLI) is explored in-depth, teaching you how to work with topics, producers, consumers, consumer groups, and offset management.

You will also learn about the Kafka user interface (UI) through a demo of Conduktor.

Moving on to programming, the course covers Java programming with Kafka, including creating producers and consumers, handling callbacks, working with keys, consumer groups, cooperative rebalancing, and offset management.

You will build a real-world project to solidify your understanding.

The course then dives into advanced producer and consumer configurations, such as acknowledgments, retries, idempotence, compression, batching, and partitioning strategies.

You will build a Wikimedia producer and an OpenSearch consumer, exploring concepts like delivery semantics, commit strategies, batching, replaying data, internal threads, and rack awareness.

Additionally, you will learn about Kafka’s extended APIs, including Kafka Connect for integrating with external systems, Kafka Streams for stream processing, and the Schema Registry for data governance.

Real-world case studies and insights into big data and fast data architectures are also covered.

For administrators, the course covers topics like cluster setup, monitoring, operations, security, multi-cluster setups, and communication protocols.

Advanced topics like topic configurations, log cleanup policies, log compaction, and handling large messages are also explored.

Apache Kafka Series - Kafka Streams for Data Processing

You will start by getting a first look at Kafka Streams and running your very first Kafka Streams application - a WordCount example.

This hands-on approach allows you to quickly grasp the core concepts.

The course then dives deep into building an end-to-end Kafka Streams application for WordCount from scratch.

You’ll learn how to set up the development environment with Java 8, Maven, and IntelliJ IDEA.

You’ll write code to define the application topology, handle graceful shutdown, debug from the IDE, and package the application as a fat JAR file.

The course also covers scaling your application and understanding internal topics.

Next, you’ll explore simple (stateless) operations on KStreams and KTables, such as map, filter, flatMap, and branch.

You’ll learn about stateless vs. stateful operations, reading/writing from/to Kafka, and the duality between KStreams and KTables.

A hands-on practice exercise on building a “FavouriteColour” application reinforces these concepts.

The course then moves on to advanced (stateful) operations like groupBy, count, aggregate, and reduce.

You’ll also learn about the peek operation, transform/transformValues, and integrating with external systems using Kafka Connect.

A crucial topic covered is Exactly Once Semantics - the theory behind it, how Kafka 0.11 solves it, and how to achieve it in Kafka Streams.

You’ll practice this through a “BankBalance” exercise, writing a producer for random bank transactions and a Streams application to maintain account balances.

Joins are explored in-depth, covering inner, left, and outer joins, join constraints, and using GlobalKTables.

You’ll build a “UserEnrich” application to join a KStream with a KTable.

Finally, the course touches on testing your Kafka Streams applications, setting you up for production-ready development.

Apache Kafka for absolute beginners

This course provides a comprehensive introduction to Apache Kafka, covering the core concepts, architecture, and practical applications.

You’ll start by understanding the Kafka ecosystem and its various components like Kafka Connect, Streams, and SQL.

This big-picture overview will help you determine when to use each tool.

Next, you’ll dive into setting up Kafka clusters, both single-node and multi-node environments.

You’ll learn how to use command-line producers and consumers, as well as consumer groups for scalability.

The course also covers the storage architecture of Kafka, including topics, partitions, replication, log segments, and message offsets.

The cluster architecture section explains the role of Zookeeper, the cluster controller, partition allocation, fault tolerance, and the in-sync replica (ISR) list.

You’ll understand the differences between partition leaders, followers, committed, and uncommitted records.

Producers are covered in-depth, from creating your first producer to understanding producer records, serializers, partitioners, and message timestamps.

You’ll learn about producer message buffers, I/O threads, and retries.

Advanced topics include horizontal and vertical scalability, multi-threading, at-least-once and at-most-once semantics, idempotence, and transactions.

The course explores data types and serialization, with hands-on examples using JSON and Avro schemas.

You’ll build a micro-project simulating a point-of-sale system using JSON and Avro-serialized invoices.

Consumers are introduced, covering the consume-transform-produce pipeline, consumer groups, and current vs. committed offsets.

The course concludes by discussing challenges with consumers and suggesting next steps for further learning.

Throughout the course, you’ll gain practical experience by configuring your development IDE and working with code examples.

The lectures are structured to provide a natural flow, guiding you from the fundamentals to advanced topics while maintaining a conversational tone.

Apache Kafka Series - Kafka Connect Hands-on Learning

The course provides a comprehensive introduction to Apache Kafka Connect, covering both theoretical concepts and hands-on practical exercises.

You’ll start by learning the prerequisites and objectives, ensuring you have the necessary foundation for the material ahead.

The initial chapters dive into the core concepts of Kafka Connect, including its architecture, components like connectors, configurations, tasks, and workers, as well as the differences between standalone and distributed modes.

This theoretical groundwork prepares you for the hands-on sections that follow.

Setting up your development environment is a crucial step, and the course guides you through installing Docker and launching a Kafka Connect cluster tailored for your operating system, be it Mac, Linux, or Windows.

Troubleshooting techniques are also covered, equipping you with the skills to diagnose and resolve issues that may arise.

With your environment ready, you’ll embark on a practical journey, starting with source connectors.

The course walks you through deploying the FileStream Source Connector in both standalone and distributed modes, providing a solid foundation for understanding source connectors.

You’ll also explore the Twitter Source Connector, learning how to configure and analyze its output in a distributed setup.

Sink connectors are equally important, and the course covers the ElasticSearch Sink Connector and the JDBC Sink Connector, teaching you how to push data into ElasticSearch and PostgreSQL databases, respectively.

The Kafka Connect REST API is introduced, enabling you to automate and streamline connector deployments and monitoring.

One of the highlights is learning to develop your own Kafka Connector from scratch.

Using the GitHub Source Connector as an example, you’ll go through the entire development process, from setting up the project to writing configurations, schemas, data models, HTTP clients, and source tasks.

You’ll even learn how to build and deploy your connector on a Kafka Connect cluster.

The course concludes with advanced concepts, including guidance on setting up Kafka Connect in a production environment and recommendations for further learning resources.

Throughout the journey, you’ll gain practical experience with various source and sink connectors, develop a deep understanding of Kafka Connect’s architecture and components, and acquire the skills to build and deploy your own custom connectors.

Apache Kafka for Developers using Spring Boot[LatestEdition]

The course starts by introducing Kafka, its terminologies, and how to set it up locally.

You’ll learn about the core Kafka components like topics, partitions, brokers, and how they work under the hood.

The real hands-on part begins with building a Spring Boot Kafka producer application.

You’ll create endpoints to publish library event messages to a Kafka topic using the KafkaTemplate.

The course covers different approaches to achieve this, including synchronous and asynchronous messaging.

You’ll write integration and unit tests using tools like Embedded Kafka and MockMVC.

Moving on, you’ll build a Spring Boot Kafka consumer application to read messages from the topic.

The course dives deep into consumer groups, offset management (manual and automatic), and scaling message consumption using concurrent consumers.

You’ll integrate with a database (H2) to persist the library events.

Error handling is crucial, and the course dedicates sections to implementing retry mechanisms, exponential backoff, and recovery techniques like dead-letter topics for both producers and consumers.

You’ll also learn about important producer configurations for reliable message delivery.

The course wraps up by covering Kafka security using SSL encryption.

You’ll set up SSL locally and access the secured Kafka cluster from your Spring Boot applications.

Apache Kafka Series - Kafka Cluster Setup & Administration

This course provides a comprehensive guide to setting up and administering an Apache Kafka cluster.

You will start by learning about Kafka’s architecture and the role of Zookeeper in a Kafka cluster.

The course then dives into setting up a Zookeeper quorum on AWS, including configuring security groups, EC2 instances, and installing Zookeeper as a service.

You will gain hands-on experience using the Zookeeper Command Line Interface (CLI) and learn about Zookeeper’s internal file system and performance factors.

The course also covers setting up a web-based management tool, Zoonavigator, to visualize the Zookeeper cluster.

Moving on to Kafka, you will learn about Kafka basics, cluster sizing, and configuration options.

Through step-by-step instructions, you will set up a single Kafka broker and then expand it to a multi-broker cluster on AWS.

You will test the cluster using various Kafka commands and learn about networking considerations, including the crucial advertised.listeners setting.

The course introduces you to Kafka Manager, a web-based tool for cluster management, and demonstrates Kafka’s resiliency by stopping brokers and observing the cluster’s health.

You will also explore factors affecting Kafka’s performance, such as I/O, network, RAM, CPU, and the operating system.

Additionally, you will learn about running Kafka in production on AWS, changing broker configurations, and advanced configuration parameters.

The course covers setting up Landoop Kafka Topics UI, Confluent REST Proxy, and Confluent Schema Registry.

Finally, you will receive guidance on next steps for further learning and special discounts on the instructor’s other courses.

Apache Kafka Series - Confluent Schema Registry & REST Proxy

You’ll start by learning the importance of schemas in Kafka and the target architecture involving these components.

The first major section dives into Avro schemas, covering primitive and complex types, logical types, and schema evolution rules.

You’ll practice creating schemas for scenarios like customers and addresses.

The course then shifts to working with Avro in Java, using GenericRecord, SpecificRecord, ReflectedRecord, and performing schema evolution hands-on.

Next, you’ll set up a Kafka cluster with Docker and Docker Compose, including the Schema Registry and REST Proxy.

This hands-on section teaches you to use the Kafka Avro console producer/consumer and write Avro producers/consumers in Java while handling schema evolution.

The course extensively covers the Confluent Schema Registry, exploring how it works behind the scenes and efficiently managing schemas.

You’ll also learn to use the Confluent REST Proxy with a REST client like Insomnia, performing topic operations, producing, and consuming binary, JSON, and Avro data.

Throughout the course, you’ll complete quizzes to reinforce your learning and receive guidance on setting up the required tools like Docker and Insomnia.

The course wraps up with annexes covering end-to-end Avro examples and scaling the REST Proxy for production use.

Apache Kafka - Real-time Stream Processing (Master Class)

This course provides a comprehensive introduction to Apache Kafka and real-time stream processing.

You will start by understanding the emergence of big data and the need for real-time stream processing.

The course then dives into the core concepts of Apache Kafka, including its architecture, storage mechanisms, and work distribution strategies.

One of the key focuses is on creating real-time data streams using Kafka producers.

You will learn how to quickly get started with producers, understand their internals, and scale them to handle high throughput.

Advanced topics like achieving exactly-once semantics and implementing transactions are also covered.

The course then shifts to stream processing using Kafka consumers, the Kafka Streams API, and creating stream processing topologies.

You will learn how to implement complex processing logic, handle state management, and leverage key abstractions like KTables and GlobalKTables.

Crucial aspects of stream processing, such as real-time aggregations, windowing, and joining streams and tables, are explored in-depth with practical examples.

The course also covers testing strategies for stream processing applications and implementing interactive query microservices using Kafka’s remote state stores.

Additionally, you will learn about data serialization and deserialization using JSON and Avro schemas, which is essential for working with real-world data formats.

Throughout the course, you will work on hands-on projects and coding exercises to reinforce the concepts learned.

The course strikes a balance between theoretical foundations and practical implementations, ensuring you gain a solid understanding of Apache Kafka and real-time stream processing.

Apache Kafka Series - KSQL on ksqlDB for Stream Processing !

This course provides a comprehensive introduction to ksqlDB and KSQL, the powerful streaming SQL engine for Apache Kafka.

You will start by setting up ksqlDB and learning the basics of creating streams and tables from Kafka topics.

The course dives into intermediate topics like joins, pull queries, and integrating with Kafka Connect for data ingestion.

You will learn how to work with different data formats such as JSON, Avro, and CSV.

The course covers important concepts like schema evolution for Avro data, nested data handling, rekeying tables, repartitioning streams, and merging multiple streams.

Windowing and geospatial operations are also covered, enabling you to perform complex event processing and location-based analytics.

The course teaches you how to extend ksqlDB’s functionality using user-defined functions (UDFs) and user-defined aggregate functions (UDAFs).

As you progress, you will learn production-ready practices like running ksqlDB in headless mode, load balancing, configuring state stores, and testing ksqlDB applications.

The course also covers the “explain plan” feature for optimizing query performance.

Throughout the course, you will work with hands-on examples and code exercises, ensuring you gain practical experience with ksqlDB and KSQL.

By the end, you will be equipped to build robust stream processing applications using ksqlDB and leverage its integration with Apache Kafka.

Apache Kafka Series - Kafka Security | SSL SASL Kerberos ACL

In this course you’ll dive deep into securing your Kafka clusters using various techniques like SSL encryption, authentication with SASL and Kerberos, and authorization with ACLs.

The course starts by setting up a Kafka cluster on an EC2 instance, ensuring you have a hands-on environment to follow along.

From there, you’ll learn about SSL encryption in Kafka - why it’s needed, how it works, and how to configure it for brokers and clients.

You’ll even explore the performance impact of enabling SSL.

Next up is authentication.

You’ll understand SSL authentication and then move on to the powerful SASL authentication mechanism using Kerberos/GSSAPI.

Expect detailed walkthroughs on setting up Kerberos principals, keytabs, and configuring Kafka and clients for Kerberos authentication.

Authorization is equally crucial, and you’ll master ACLs (Access Control Lists) in Kafka.

Through hands-on demos, you’ll learn how to define ACLs for different operations and resources.

Securing Zookeeper, which manages Kafka’s metadata, is also covered extensively.

From Kerberos setup to authorization configurations and the security migration tool, you’ll gain comprehensive knowledge.

Throughout the course, you’ll work with real-world examples and hands-on activities, ensuring you grasp the concepts thoroughly.

The instructors are experienced professionals who explain complex topics clearly.