Event-Driven Microservices with Spring Boot 3.4 and Apache Kafka

Introduction

In a traditional REST-based microservices architecture, services talk to each other directly: Service A calls Service B's API and waits for a response. This works, but it creates tight coupling — if Service B is slow or down, Service A is affected. It also makes it hard to add new services: if a third service needs to react to the same events, Service A must now know about it too.

Event-driven architecture inverts this relationship. Instead of calling each other, services publish events to a message broker. Other services subscribe to those events and react independently. The publisher doesn't know or care who is listening. You can add a new service without touching the publisher.

Apache Kafka is the most widely used event streaming platform in production today. It's designed for high throughput (millions of messages per second), durable storage, and replay — consumers can re-read past events at any time. Spring Boot's spring-kafka library makes integrating with Kafka idiomatic and familiar.

In this tutorial we'll build a real-world order processing system with three services:

  • Order Service — receives REST requests and publishes OrderPlaced events to Kafka
  • Inventory Service — consumes OrderPlaced events and deducts stock
  • Notification Service — consumes OrderPlaced events and sends confirmation emails

Both consumer services react to the same event independently. We'll also implement Dead Letter Topic (DLT) error handling so failed messages don't block the pipeline.

Prerequisites

  • Java 21
  • Maven 3.9+
  • Docker (for running Kafka locally)
  • Basic understanding of Spring Boot

Running Kafka Locally with Docker

The easiest way to run Kafka for development is with the official apache/kafka image (KRaft mode — no ZooKeeper needed since Kafka 3.3):

docker run -d --name kafka \
  -p 9092:9092 \
  -e KAFKA_NODE_ID=1 \
  -e KAFKA_PROCESS_ROLES=broker,controller \
  -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 \
  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
  -e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
  -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \
  -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 \
  -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
  apache/kafka:3.9.0

Verify it's running:

docker exec kafka /opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Project Setup

For simplicity we'll build all three services in a single Maven multi-module project, but each module could equally be a standalone deployable.

Root pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
             https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.vimleshpandey.demo</groupId>
  <artifactId>kafka-microservices</artifactId>
  <version>1.0.0</version>
  <packaging>pom</packaging>

  <modules>
    <module>order-service</module>
    <module>inventory-service</module>
    <module>notification-service</module>
    <module>shared-events</module>
  </modules>

  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>3.4.5</version>
    <relativePath/>
  </parent>

  <properties>
    <java.version>21</java.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.springframework.kafka</groupId>
      <artifactId>spring-kafka</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.springframework.kafka</groupId>
      <artifactId>spring-kafka-test</artifactId>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>

Spring Kafka version: Spring Boot 3.4.5 resolves spring-kafka to version 3.3.x automatically — no need to specify the version explicitly.

Shared Events Module

Event classes are shared between producer and consumers. This is the contract between services.

package com.vimleshpandey.demo.events;

import java.math.BigDecimal;
import java.time.Instant;
import java.util.UUID;

public record OrderPlacedEvent(
    String orderId,
    String customerId,
    String customerEmail,
    String productId,
    int quantity,
    BigDecimal totalAmount,
    Instant placedAt
) {
    public static OrderPlacedEvent of(String customerId, String customerEmail,
                                      String productId, int quantity,
                                      BigDecimal totalAmount) {
        return new OrderPlacedEvent(
            UUID.randomUUID().toString(),
            customerId,
            customerEmail,
            productId,
            quantity,
            totalAmount,
            Instant.now()
        );
    }
}

Using Java records for events is ideal: they're immutable, serialise cleanly to JSON, and act as self-documenting contracts.

Order Service (Producer)

application.yml

spring:
  application:
    name: order-service
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      properties:
        spring.json.add.type.headers: false   # don't embed Java type in headers

app:
  kafka:
    topic:
      orders: orders.placed

server:
  port: 8081

Kafka Topic Configuration

package com.vimleshpandey.demo.order.config;

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.TopicBuilder;

@Configuration
public class KafkaTopicConfig {

    @Value("${app.kafka.topic.orders}")
    private String ordersTopic;

    @Bean
    public NewTopic ordersTopic() {
        return TopicBuilder.name(ordersTopic)
            .partitions(3)
            .replicas(1)
            .build();
    }

    @Bean
    public NewTopic ordersDeadLetterTopic() {
        return TopicBuilder.name(ordersTopic + ".DLT")
            .partitions(1)
            .replicas(1)
            .build();
    }
}

Spring's KafkaAdmin bean (auto-configured) picks up NewTopic beans and creates the topics on startup if they don't exist.

Order Producer Service

package com.vimleshpandey.demo.order.service;

import com.vimleshpandey.demo.events.OrderPlacedEvent;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.SendResult;
import org.springframework.stereotype.Service;

import java.util.concurrent.CompletableFuture;

@Service
public class OrderProducerService {

    private static final Logger log = LoggerFactory.getLogger(OrderProducerService.class);

    private final KafkaTemplate<String, OrderPlacedEvent> kafkaTemplate;

    @Value("${app.kafka.topic.orders}")
    private String ordersTopic;

    public OrderProducerService(KafkaTemplate<String, OrderPlacedEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publishOrderPlaced(OrderPlacedEvent event) {
        CompletableFuture<SendResult<String, OrderPlacedEvent>> future =
            kafkaTemplate.send(ordersTopic, event.orderId(), event);

        future.whenComplete((result, ex) -> {
            if (ex != null) {
                log.error("Failed to publish OrderPlaced event for order {}: {}",
                    event.orderId(), ex.getMessage());
            } else {
                log.info("Published OrderPlaced event: orderId={}, partition={}, offset={}",
                    event.orderId(),
                    result.getRecordMetadata().partition(),
                    result.getRecordMetadata().offset());
            }
        });
    }
}

Key design choices:

  • The message key is orderId — Kafka guarantees ordering within a partition, so using orderId as the key ensures all events for the same order go to the same partition.

  • send() returns a CompletableFuture — we handle success and failure asynchronously without blocking the REST thread.

REST Controller

package com.vimleshpandey.demo.order.controller;

import com.vimleshpandey.demo.events.OrderPlacedEvent;
import com.vimleshpandey.demo.order.service.OrderProducerService;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.*;

import java.math.BigDecimal;

@RestController
@RequestMapping("/api/orders")
public class OrderController {

    private final OrderProducerService producerService;

    public OrderController(OrderProducerService producerService) {
        this.producerService = producerService;
    }

    @PostMapping
    @ResponseStatus(HttpStatus.ACCEPTED)
    public OrderAcceptedResponse placeOrder(@RequestBody PlaceOrderRequest request) {
        OrderPlacedEvent event = OrderPlacedEvent.of(
            request.customerId(),
            request.customerEmail(),
            request.productId(),
            request.quantity(),
            request.totalAmount()
        );
        producerService.publishOrderPlaced(event);
        return new OrderAcceptedResponse(event.orderId(), "Order accepted and queued");
    }

    record PlaceOrderRequest(
        String customerId,
        String customerEmail,
        String productId,
        int quantity,
        BigDecimal totalAmount
    ) {}

    record OrderAcceptedResponse(String orderId, String message) {}
}

Note the @ResponseStatus(HttpStatus.ACCEPTED) — the REST endpoint returns 202 Accepted because the order is queued for async processing, not yet fulfilled.

Inventory Service (Consumer 1)

application.yml

spring:
  application:
    name: inventory-service
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: inventory-service-group
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: "com.vimleshpandey.demo.events"
        spring.json.value.default.type: "com.vimleshpandey.demo.events.OrderPlacedEvent"
    listener:
      ack-mode: MANUAL_IMMEDIATE

server:
  port: 8082

Inventory Consumer

package com.vimleshpandey.demo.inventory.consumer;

import com.vimleshpandey.demo.events.OrderPlacedEvent;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.RetryableTopic;
import org.springframework.kafka.retrytopic.TopicSuffixingStrategy;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.messaging.handler.annotation.Payload;
import org.springframework.retry.annotation.Backoff;
import org.springframework.stereotype.Component;

@Component
public class InventoryEventConsumer {

    private static final Logger log = LoggerFactory.getLogger(InventoryEventConsumer.class);

    @RetryableTopic(
        attempts = "3",
        backoff = @Backoff(delay = 2000, multiplier = 2.0),
        topicSuffixingStrategy = TopicSuffixingStrategy.SUFFIX_WITH_DELAY_VALUE,
        dltTopicSuffix = ".DLT"
    )
    @KafkaListener(
        topics = "orders.placed",
        groupId = "inventory-service-group"
    )
    public void handleOrderPlaced(@Payload OrderPlacedEvent event, Acknowledgment ack) {
        log.info("[Inventory] Processing order: {} — product: {}, qty: {}",
            event.orderId(), event.productId(), event.quantity());

        try {
            deductStock(event.productId(), event.quantity());
            ack.acknowledge();
            log.info("[Inventory] Stock deducted for order {}", event.orderId());
        } catch (Exception e) {
            log.error("[Inventory] Failed to process order {}: {}",
                event.orderId(), e.getMessage());
            throw e; // triggers @RetryableTopic retry
        }
    }

    private void deductStock(String productId, int quantity) {
        // In a real service: query DB, check stock, update row
        log.info("[Inventory] Deducting {} units of product {}", quantity, productId);
        // Simulate occasional failures for demo purposes
        if (productId.startsWith("ERR")) {
            throw new RuntimeException("Product " + productId + " not found in inventory");
        }
    }
}

@RetryableTopic — the modern approach to retries:

  • Spring Kafka 2.7+ introduced @RetryableTopic which creates retry topics (e.g., orders.placed-retry-2000, orders.placed-retry-4000) automatically

  • Failed messages are forwarded to the retry topic with an exponential backoff delay

  • After all retries exhausted, the message lands in the Dead Letter Topic (DLT): orders.placed.DLT

  • This is non-blocking — the consumer doesn't sit and sleep waiting for retries; it moves on and processes other messages

Notification Service (Consumer 2)

The notification service listens to the same topic with a different consumer group — it receives every event independently of the inventory service.

application.yml

spring:
  application:
    name: notification-service
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: notification-service-group
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: "com.vimleshpandey.demo.events"
        spring.json.value.default.type: "com.vimleshpandey.demo.events.OrderPlacedEvent"

server:
  port: 8083

Notification Consumer

package com.vimleshpandey.demo.notification.consumer;

import com.vimleshpandey.demo.events.OrderPlacedEvent;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.RetryableTopic;
import org.springframework.retry.annotation.Backoff;
import org.springframework.stereotype.Component;

@Component
public class NotificationEventConsumer {

    private static final Logger log = LoggerFactory.getLogger(NotificationEventConsumer.class);

    @RetryableTopic(
        attempts = "3",
        backoff = @Backoff(delay = 1000, multiplier = 2.0),
        dltTopicSuffix = ".notification.DLT"
    )
    @KafkaListener(
        topics = "orders.placed",
        groupId = "notification-service-group"
    )
    public void handleOrderPlaced(OrderPlacedEvent event) {
        log.info("[Notification] Sending confirmation to {} for order {}",
            event.customerEmail(), event.orderId());

        sendEmail(
            event.customerEmail(),
            "Order Confirmed — " + event.orderId(),
            buildEmailBody(event)
        );
    }

    private void sendEmail(String to, String subject, String body) {
        // In production: Spring Mail / SendGrid / SES
        log.info("[Notification] EMAIL → {} | Subject: {}", to, subject);
    }

    private String buildEmailBody(OrderPlacedEvent event) {
        return String.format(
            "Hi, your order %s for %d unit(s) of product %s totalling %s has been received.",
            event.orderId(), event.quantity(), event.productId(), event.totalAmount()
        );
    }
}

Dead Letter Topic (DLT) Handler

Messages that fail all retry attempts land in the DLT. You need a dedicated handler to monitor and alert on these:

package com.vimleshpandey.demo.inventory.consumer;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Component
public class DltConsumer {

    private static final Logger log = LoggerFactory.getLogger(DltConsumer.class);

    @KafkaListener(
        topics = "orders.placed.DLT",
        groupId = "inventory-dlt-group"
    )
    public void handleDeadLetter(ConsumerRecord<String, byte[]> record) {
        log.error("[DLT] Unprocessable message landed in DLT. " +
            "Key: {}, Topic: {}, Partition: {}, Offset: {}",
            record.key(), record.topic(),
            record.partition(), record.offset());

        // In production:
        // 1. Alert via PagerDuty / Slack
        // 2. Persist to a poison-message store for manual review
        // 3. Optionally replay after the underlying bug is fixed
    }
}

Testing

Spring Kafka ships with @EmbeddedKafka — you can run real Kafka broker in-process for integration tests without Docker:

package com.vimleshpandey.demo.order;

import com.vimleshpandey.demo.events.OrderPlacedEvent;
import com.vimleshpandey.demo.order.service.OrderProducerService;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.test.context.EmbeddedKafka;
import org.springframework.test.context.ActiveProfiles;

import java.math.BigDecimal;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;

import static org.assertj.core.api.Assertions.assertThat;

@SpringBootTest
@EmbeddedKafka(
    partitions = 1,
    topics = {"orders.placed"}
)
@ActiveProfiles("test")
class OrderProducerServiceTest {

    @Autowired
    private OrderProducerService producerService;

    private final BlockingQueue<OrderPlacedEvent> receivedEvents =
        new LinkedBlockingQueue<>();

    @KafkaListener(topics = "orders.placed", groupId = "test-consumer")
    public void consume(OrderPlacedEvent event) {
        receivedEvents.add(event);
    }

    @Test
    void shouldPublishOrderPlacedEvent() throws InterruptedException {
        OrderPlacedEvent event = OrderPlacedEvent.of(
            "cust-123", "test@example.com", "PROD-456",
            2, new BigDecimal("49.98")
        );

        producerService.publishOrderPlaced(event);

        OrderPlacedEvent received = receivedEvents.poll(10, TimeUnit.SECONDS);

        assertThat(received).isNotNull();
        assertThat(received.customerId()).isEqualTo("cust-123");
        assertThat(received.quantity()).isEqualTo(2);
    }
}

Running the System

# Terminal 1 — Order Service (producer)
cd order-service && mvn spring-boot:run

# Terminal 2 — Inventory Service (consumer 1)
cd inventory-service && mvn spring-boot:run

# Terminal 3 — Notification Service (consumer 2)
cd notification-service && mvn spring-boot:run

# Place an order
curl -s -X POST http://localhost:8081/api/orders \
  -H 'Content-Type: application/json' \
  -d '{
    "customerId": "cust-001",
    "customerEmail": "alice@example.com",
    "productId": "PROD-100",
    "quantity": 3,
    "totalAmount": 149.97
  }' | jq .

You'll immediately see both consumer services log their reactions:

[Inventory]     Processing order: abc-123 — product: PROD-100, qty: 3
[Notification]  Sending confirmation to alice@example.com for order abc-123

Test error handling with a product ID prefixed ERR:

curl -X POST http://localhost:8081/api/orders \
  -H 'Content-Type: application/json' \
  -d '{"customerId":"c1","customerEmail":"e@e.com","productId":"ERR-999","quantity":1,"totalAmount":9.99}'

You'll see the inventory service retry 3 times with exponential backoff, then the DLT handler log the dead letter.

How the Consumer Group Model Works

Understanding Kafka's consumer group model is essential:

  • Each consumer group has its own offset pointer in each partition
  • inventory-service-group and notification-service-group are independent — they both read every message from the topic, independently
  • Within a consumer group, partitions are distributed across instances — if you start 3 instances of the inventory service and the topic has 3 partitions, each instance handles one partition in parallel (horizontal scaling)

This means adding the Notification Service required zero changes to the Order Service. The architecture is genuinely extensible.

Common Pitfalls and Tips

1. spring.json.trusted.packages is mandatory
Without it, the JsonDeserializer refuses to deserialise messages and throws ClassNotFound or a trust exception. Always set it to the package containing your event classes.

2. Don't use AUTO_COMMIT
The default enable-auto-commit=true can cause messages to be marked as consumed before your handler finishes — if the handler crashes mid-way, the message is lost. Set ack-mode: MANUAL_IMMEDIATE and call ack.acknowledge() only after successful processing.

3. @RetryableTopic vs DefaultErrorHandler
Spring Kafka provides two retry mechanisms. @RetryableTopic (introduced in Spring Kafka 2.7) is the modern, non-blocking approach that forwards to retry topics. DefaultErrorHandler with FixedBackOff is the older approach that blocks the thread. Use @RetryableTopic for production.

4. Partition count and consumer scaling
The number of partitions is the maximum parallelism ceiling. If your topic has 3 partitions and you start 4 consumer instances in the same group, one instance will be idle. Choose partition count upfront based on your expected scale — you can increase partitions but you cannot decrease them.

5. Message ordering
Kafka guarantees ordering within a partition. If ordering matters (e.g., all events for the same order must be processed in sequence), use the same key for all related events — they'll land on the same partition.

6. Schema evolution
JSON is flexible but fragile — adding a required field breaks old consumers. Consider Apache Avro with a Schema Registry for production systems where event schemas evolve over time.

Conclusion

You've built a production-pattern event-driven system where:

  • The Order Service publishes events without knowing who consumes them
  • Inventory and Notification services react independently to the same events
  • @RetryableTopic handles transient failures without blocking the consumer
  • Dead Letter Topics catch poison messages for manual review and alerting
  • The system scales horizontally — add more partitions and more consumer instances

Event-driven architecture with Kafka is the backbone of large-scale systems at LinkedIn, Netflix, Uber, and Airbnb. Spring Boot's spring-kafka makes it accessible with the same declarative, annotation-driven model you already know from Spring — but with the resilience and throughput of one of the most battle-tested messaging systems in the industry.