Prepare your local Kafka environment to use Kafka Connect
When you start learning about Kafka you will quickly hit the word Kafka Connect, and think what is it? it's normal when you learn new technology to think about what everything is but you should think of Kafka connect as a “plugin folder” where you can place plugins, a Kafka connect can be used to produce messages or consume messages.
It’s imported to understand Kafka Connect can’t run without Kafka but you can run a Kafka cluster whiteout Kafka Connect.
The idea is you can have a source eg. database, queue system, or file system you want automatically be produced to a Kafka topic, here you can use a Kafka Connect to it, and it will probley be spoken about as “Kafka Connect Source” and then it's hitting Kafka. If you want to do something with it, it can think you want to store a backend of all incoming messages and then later transform the data, here you will place a Kafka Connect between your Kafka cluster and the target system, which will be called a “Kafka Connect Sink” or “Kafka Sink” a child got many names.
Now you have the basic knowledge about what Kafka Connect is I think it's important for you to have the option to test it locally and not in the cloud, why did you think it because if you are using AWS MSK as an example it will cost you money for each Kafka Connect you are installing so based on that you will save a lot of cash to testing Kafka Connect plugins local before you deploy it to your staging or production environment.
So let's write our docker-compose.yaml file like a boss! :)
When it's done you should not start up yet, you need to create a file in the folder you docker-compose file with some default connect-distributed.properties first and save the file into this file-path: kafka_connect_config/connect-distributed.properties
when your files are saved you are ready to spin your docker containers up by writing the command into your terminal.
If you will try to produce or consume from your Kafka Cluster from Python you can read my article about it.
