DevOps Team

April 03

Get Started with the Built-In Image of Kafka Azure MarketPlace Offering

About the Kafka VM Image:

In this VM Image we have packed Apache Kafka which is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Added Kafka server and Zookeeper so that you can customize it and use it according to your requirements. Also added an example python script so you can check out the working of Apache Kafka.

Benefits of using the VMI:

1. No Setup Required: The setup of Kafka is pre-configured by us which takes a lot of your time.

2. Less Complexity: The Kafka setup can be complex and sometimes you can face errors while installing Kafka in Linux which can give you a headache sometimes.

3. Example Provided: There is an example python script provided in the documentation which can be useful to you while checking connection with Kafka. Otherwise, you have to search for it online which will be a time taking process.

4. High Throughput: Deliver messages at network limited throughput using machine with latencies as low as 2ms.

5. Connect to Almost Anything: Kafka’s out-of-the-box Connect interface integrates with hundreds of event sources and event sinks including Postgres, JMS, Elasticsearch, AWS S3, and more.

Follow the below steps to know how to use the Kafka Image:

1. Log in to your Azure VM using SSH, go to /usr/local/ directory

2. After logging into your VM, type the “ls” command to get a list of files.

3. Get Inside the Kafka Directory

   

4. Go Inside the “cd /usr/local/kafka_2.12-2.5.0/config” Directory and edit the server properties.

   

5. Add your Server IP to the advertised.listeners

   

   Save the file using ctrl + S and exit the file using ctrl + X.

   Now, get back to your Kafka directory.

   

6. Use the below command to start the zookeeper

   bin/zookeeper-server-start.sh config/zookeeper.properties

   

   Your terminal will get locked after running Zookeeper. Keep the current terminal running and open a new terminal and get back inside the Kafka directory.

7. Edit your bashrc using the command “sudo nano ~/.bashrc”

   Add these below-mentioned lines to your .bashrc

   export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

   export PATH=/usr/local/kafka_2.12-2.5.0/bin:$PATH

   

   Save the file using ctrl + S and exit the file using ctrl + X.

   Save the .bashrc using the command “source ~/.bashrc”

   

   After sourcing the file bashrc, get back inside Kafka Directory.

   

8. Use the below command to start the Kafka Server.

   kafka-server-start.sh config/server.properties

   

9. To Send Messages to your Kafka server, use the below script.