Kafka

Hello 👋, In this article we’re going to build a data pipeline that connects Kafka to MongoDB. In short, we’re going to add a MongoDB Sink connector to a Kafka Connect cluster and run a MongoDB instance in Docker to test the connector. By reading this article I hope that you will learn How to install the MongoDB connector in Kafka Connect How to configure the MongoDB connector How to create topics in Kafka using Confluent Tools How to monitor Kafka Connect using JConsole. Let’s get started! ...

Hello 👋, In this article I will give you some tips on how to improve the throughput of a message producer. I had to write a Golang based application which would consume messages from Apache Kafka and send them into a sink using HTTP JSON / HTTP Protocol Buffers. To see if my idea works, I started using a naïve approach in which I polled Kafka for messages and then send each message into the sink, one at a time. This worked, but it was slow. ...

Hello! In this article we will discuss how to quickly get started with Kafka and Kafka Connect to grab all the commits from a Github repository. This is a practical tutorial which saves you some time browsing the Kafka’s documentation. Environment Kafka is bit difficult to setup, you will need Kafka, Zookeper and Kafka Connect at least. To simplify the setup we’re going to use Docker and docker-compose. Once you have Docker and docker-compose installed on your system, you can run a single command and get a Kafka environment for developing. ...