Step By Step : Build and Run Kafka in Eclipse IDE + [ Scala || Java ] + Gradle
As a coder, we are more comfortable with the editor tools (specially Eclipse IDE) for rapid development , build & continuous integration. When first time I was trying to develop some Kafka producer and consumer using Scala, I was wondering if I could setup the same through eclipse to make life easier, however after a lot of hit and trial finally able to setup and successfully executed my first program.
Likewise, some other friends out there might struggling on the same. Hence thought to share the simple steps to get the Kafka setup and run program from eclipse IDE done.
Important : Before proceed, please ensure your zookeeper & Kafka servers are already running. To get it handy please refer Step By Step : Install Kafka in OSX / macOS Sierra for setting up and running ZK & Kafka server
Step 1: Install Scala IDE (Eclipse for Scala)
Download Scala IDE from the official site http://scala-ide.org
[Please ensure the IDE version you are downloading is meeting the system requirement]
Step 2: Install Gradle in the downloaded Scala IDE
Open Scala IDE -> Create your own workspace —> Help -> Eclipse Marketplace -> search for “Gradle IDE pack 3.8.x + 1.0.x” (note: my IDE version is Neon, hence this Gradle version is compatible)
Step 3: Importing Kafka libraries into eclipse workspace
- Download the Kafka latest build from git https://github.com/sreev/kafka/tree/0.8.2
- Download the zip into Desktop (to local system)
- Extract the downloaded zip
- Go to eclipse IDE -> File -> Import -> Gradle (STS) project -> Next -> Brouse the extracted Kafka directory from you local disk -> click ‘Build Model’ -> next -> Finish
The following are the step by step screenshots for importing Kafka project
Step 4: Run application
Now expand the project “example” -> run the “KafkaConsumerProducerDemo” class as java program
Now you should able to see the producer and consumer running in your eclipse console and printing the data in consumer console those are published by the producer.
Kafka properties for the above, where we have set all the required configuration params
Now you can also observe the topic name (you have provided in your program) from your terminal where the kafka server is already running.
error when reading RDS pgsql table in pyspark glue Zeppelin
Conclusion :
And you all set for writing more complex Kafka code and executing from eclipse IDE (can be Scala or Java as per the need)
Enjoy ….
Don’t miss the chance to participate in the upcoming Internship Program which will be done using Microsoft Dot Net Web Development Full Stack Technology. The new batch will be starting from May 20, 2024. We will have most experienced trainers for you to successfully complete the internship with live project experience.
Why to choose Our Internship Program?
Industry-Relevant Projects
Tailored Assignments: We offer projects that align with your academic background and career aspirations.
Real-World Challenges: Tackle industry-specific problems and contribute to meaningful projects that make a difference.
Professional Mentorship
Guidance from Experts: Benefit from one-on-one mentorship from seasoned professionals in your field.
Career Development Workshops: Participate in workshops that focus on resume building, interview skills, and career planning.
Networking Opportunities
Connect with Industry Leaders: Build relationships with professionals and expand your professional network.
Peer Interaction: Collaborate with fellow interns and exchange ideas, fostering a supportive and collaborative environment.
Skill Enhancement
Hands-On Experience: Gain practical skills and learn new technologies through project-based learning.
Soft Skills Development: Enhance communication, teamwork, and problem-solving skills essential for career success.
Free Demo Class Available
