Apache Kafka is a distributed streaming platform widely used for real-time data pipelines and streaming applications. It allows services to publish and subscribe to streams of records, store them reliably and process them as they arrive. When combined with Spring Boot, Kafka becomes a powerful solution for microservices communication, enabling scalability, decoupling and fault tolerance.
Why Communication Matters in Microservices
In a microservice architecture, services need to communicate with each other to exchange data and orchestrate workflows. There are two primary communication patterns:
- Synchronous Communication: Direct calls using HTTP/REST or gRPC.
- Asynchronous Communication: Message-based communication using systems like Apache Kafka.
Asynchronous communication is often preferred because it provides:
- Decoupling: Services operate independently, making the system more scalable.
- Resilience: Messages are stored and retried if a service is temporarily unavailable.
- Scalability: Kafka efficiently handles large throughput, allowing independent service scaling.
Kafka in the Microservice Architecture
In the microservice architecture, Kafka can be used to:
- Publish events: Services can publish the events or messages to the Kafka topics of the application.
- Subscribe to events: Services can subscribe to the relevant Kafka topics to receive and process events.
Implementation of Microservices Communication with Apache Kafka in Spring Boot
Now, let's move to the main steps to the Implementation of Microservices Communication with Apache Kafka:
Step 1: Set up the Kafka
Please make sure Kafka is installed and running on your local system as a Kafka server application.â
Step 2: Create the Spring Boot Project
Create a Spring Boot project using the Spring Initializr and add the required dependencies.
- Spring Web
- Spring For Apache Kafka
- Lombok
- Spring DevTools
After Creating, the project folder structure will be like the below.
Step 3: Configure the application properties
Open the application.properties and add the below code for the configurations of the Apache Kafka of the Spring project.
spring.application.name=kafka-subscribe
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=my-group
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
Step 4: Consumer Kafka Configuration
- We can create the KafkaConsumerConfig class that can configure the configuration of the Consumer service of the Spring application.
- Go to src -> main -> java > org.example.kafkasubscribe -> config -> KafkaConsumerConfig and put the below code.
package org.example.kafkasubscribe.config;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.EnableKafka;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import java.util.HashMap;
import java.util.Map;
@Configuration
@EnableKafka
public class KafkaConsumerConfig {
@Bean
public ConsumerFactory<String, String> consumerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
return new DefaultKafkaConsumerFactory<>(props);
}
@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
return factory;
}
}
Step 5: Producer Kafka Configuration
- We will create the KafkaProducerConfig class that can configure the configuration of the producer service of the Spring application.
- Go to src -> main -> java -> org.example.kafkasubscribe -> config -> KafkaProducerConfig and put the below code.
package org.example.kafkasubscribe.config;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.core.ProducerFactory;
import java.util.HashMap;
import java.util.Map;
@Configuration
public class KafkaProducerConfig {
@Bean
public ProducerFactory<String, String> producerFactory() {
Map<String, Object> configProps = new HashMap<>();
configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
return new DefaultKafkaProducerFactory<>(configProps);
}
@Bean
public KafkaTemplate<String, String> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
}
Step 6: Create the KafkaConsumerService
- We will create the KafkaConsumerService that provides the service of the Consumer of the application.
- Go to src -> main -> java > org.example.kafkasubscribe -> service -> KafkaConsumerService and put the below code.
package org.example.kafkasubscribe.service;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;
@Service
public class KafkaConsumerService {
@KafkaListener(topics = "my-topic", groupId = "my-group")
public void listen(String message) {
System.out.println("Received Message: " + message);
}
}
Step 7: Create the KafkaProducerService
- We will create the KafkaProducerService that provides the service of the Producer of the application.
- Go to src -> main -> java -> org.example.kafkasubscribe -> service -> KafkaProducerService and put the below code.
package org.example.kafkasubscribe.service;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
@Service
public class KafkaProducerService {
private final KafkaTemplate<String, String> kafkaTemplate;
@Autowired
public KafkaProducerService(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void sendMessage(String topic, String message)
{
kafkaTemplate.send(topic, message);
}
}
Step 8: Create the KafkaController class
- We will create the KafkaController class that can create the endpoint of the publish message of the application.
- Go to src -> main -> java -> org.example.kafkasubscribe -> controller -> KafkaController and put the below code.
package org.example.kafkasubscribe.controller;
import org.example.kafkasubscribe.service.KafkaProducerService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
@RestController
public class KafkaController {
private final KafkaProducerService producerService;
@Autowired
public KafkaController(KafkaProducerService producerService) {
this.producerService = producerService;
}
@PostMapping("/publish")
public ResponseEntity<String> publishMessage(@RequestParam("message") String message) {
producerService.sendMessage("my-topic", message);
return ResponseEntity.ok("Message published to Kafka topic");
}
}
Step 9: Main Class
Open the main class. No changes are required to be done in main class.
package org.example.kafkasubscribe;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import java.time.Duration;
import java.util.Collections;
@SpringBootApplication
public class KafkaSubscribeApplication {
public static void main(String[] args) {
SpringApplication.run(KafkaSubscribeApplication.class, args);
}
}
pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.2.5</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>org.example</groupId>
<artifactId>kafka-subscribe</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>kafka-subscribe</name>
<description>kafka-subscribe</description>
<properties>
<java.version>17</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
</project>
Step 10: Run the Application
Once complete the application then it will start the application at port 8080.
Step 11: Endpoint Testing
Publish the message API:
POST http://localhost:8080/publish?message=HelloKafka
Output:
Application Log to print the Kafka Message:
By following these steps, we can set up the basic microservices communication system using Apache Kafka and Spring Boot. This setup ensures that the microservices can communicate effectively and it can handle high throughput and provide scalability and fault tolerance.
