Write your first Kafka producer and Kafka consumer in Python
When you have Kafka up and running it's a good idea to know how to produce data to the stream and how to consume the data again, else you do get not much value out of using Kafka streams.
So in this article, I will help you to understand the basics of your Kafka cluster. if you already have a Kafka cluster running you can connect to else you can follow my previews Kafka article on how you set Kafka up with docker on your local machine. Remember to create the topic before using it.
Remember to install the Kafka package with pip to Python before starting this sample.
Producing the first message to a Kafka Topic
You need to send your data into Kafka before how can get the data like you know from a queue system eg. SQS on AWS or RabbitMQ if you like it that way, the benefits of Kafka because its a data stream are the data can be re-run on the consumer side it's very powerful to know that and you need to know data streams do not replace queue systems it just a other tool to different workloads, :)
Let's write our producer script so you can see how simple it will look to get up and running with Kafka and send messages into a Kafka Topic.
Remember the limit pr Kafka message to a topic is 1MB so if you have a huge messages bigger then 1MB its recommend to point to a second location where its possible to read the data, but this is a side note to you :)
We are simulating a hotel booking event to our local Kafka cluster there are running with Docker.
Consuming your first Kafka Message(s) as a Kafka Consumer
Now when you have learned how to produce your messages into a Kafka Topic it will be nice if you can read it again right? so let's learn how to consume from a Kafka Topic and do something with this message, that's why we are using Kafka.
When you are running this code the output will show you the partition, the offset, and the message value, and the message value is your JSON from our producer you can use and do whatever you want with your consuming event.
GitHub Repository
You can visit my GitHub repository download the source code and use it including lots of other code examples with Python.
