Step By Step : Build and Run Kafka in Eclipse IDE + [ Scala || Java ] + Gradle

As a coder, we are more comfortable with the editor tools (specially Eclipse IDE) for rapid development , build & continuous integration. When first time I was trying to develop some Kafka producer and consumer using Scala, I was wondering if I could setup the same through eclipse to make life easier, however after a lot of hit and trial finally able to setup and successfully executed my first program.

Likewise, some other friends out there might struggling on the same. Hence thought to share the simple steps to get the Kafka setup and run program from eclipse IDE done.

Important : Before proceed, please ensure your zookeeper & Kafka servers are already running. To get it handy please refer Step By Step : Install Kafka in OSX / macOS Sierra for setting up and running ZK & Kafka server

Step 1: Install Scala IDE (Eclipse for Scala)

Download Scala IDE from the official site http://scala-ide.org

[Please ensure the IDE version you are downloading is meeting the system requirement]

Step 2: Install Gradle in the downloaded Scala IDE

Open Scala IDE -> Create your own workspace —> Help -> Eclipse Marketplace -> search for “Gradle IDE pack 3.8.x + 1.0.x” (note: my IDE version is Neon, hence this Gradle version is compatible)

Step 3: Importing Kafka libraries into eclipse workspace

• Download the Kafka latest build from git https://github.com/sreev/kafka/tree/0.8.2
• Download the zip into Desktop (to local system)
• Extract the downloaded zip
• Go to eclipse IDE -> File -> Import -> Gradle (STS) project -> Next -> Brouse the extracted Kafka directory from you local disk -> click ‘Build Model’ -> next -> Finish

The following are the step by step screenshots for importing Kafka project

Step 4: Run application

Now expand the project “example” -> run the “KafkaConsumerProducerDemo” class as java program

Now you should able to see the producer and consumer running in your eclipse console and printing the data in consumer console those are published by the producer.

Kafka properties for the above, where we have set all the required configuration params

Now you can also observe the topic name (you have provided in your program) from your terminal where the kafka server is already running.

error when reading RDS pgsql table in pyspark glue Zeppelin

Conclusion : 

           And you all set for writing more complex Kafka code and executing from eclipse IDE (can be Scala or Java as per the need)

Enjoy ….