Kafka CommonErrorHandler fatal exceptions

Apache Kafka has been around for over a decade, and anyone who has worked with it knows how intricate it can be. Its inner workings are complex, and the overwhelming number of configuration options doesn’t make things any easier.

While the Spring Kafka documentation is undoubtedly helpful, it mirrors Kafka’s complexity—dense and packed with information. If you're anything like me, you might find yourself missing key details simply because it's a lot to process.

When it comes to error management in Kafka, the options are plentiful. Should you discard failures? Set up retries? Use fixed intervals, backoff strategies, or a dead-letter queue?

The golden rule is to tailor your error-handling approach to the specific needs of your topic. If an event can be safely skipped or replayed later, minimal handling might suffice. However, for critical processes—like a state machine where every event is essential—you’ll likely want to configure retries to continue indefinitely, ensuring processing resumes as soon as the issue is resolved.

Spring provides a CommonErrorHandler (DefaultErrorHandler) right out of the box. By default, it comes configured with a FixedBackOff policy that retries a failed record up to 10 times before skipping it and logging an error.

Now, let’s consider a scenario involving a critical event that cannot be missed and requires infinite retries. You can easily achieve this by defining a custom error handler as a @Bean. Spring Boot will automatically discover and wire it into your application.

To prevent aggressive retries, a practical approach is to use an ExponentialBackOff strategy, which gradually increases the retry interval over time. This provides a more controlled and resource-friendly solution.

                @Bean
public CommonErrorHandler defaultErrorHandler() {
	
	var exponentialBackOff = new ExponentialBackOff();
	exponentialBackOff.setMaxInterval(Duration.ofHours(1).toMillis());

	return new DefaultErrorHandler(exponentialBackOff);
}
                

You might think that any exception thrown during event consumption would be caught by the error handler, right? Well... not quite. By default, the DefaultErrorHandler classifies certain exceptions as fatal .

These exceptions completely bypass the defined error-handling policy—they’re acknowledged, and the record is skipped without retries. This means that issues like DeserializationException, ClassCastException, or even runtime dependency errors such as NoSuchMethodException can slip through, effectively sidestepping the handler you’ve configured.

Let’s finalize the setup to handle all events as critical, ensuring infinite retries regardless of the exception. By overriding the default exception classifications, you can ensure that all exceptions are treated uniformly and handled consistently by your error handler.

                @Bean
public CommonErrorHandler defaultErrorHandler() {

	var exponentialBackOff = new ExponentialBackOff();
	exponentialBackOff.setMaxInterval(Duration.ofMinutes(2).toMillis());
	exponentialBackOff.setInitialInterval(Duration.ofSeconds(2).toMillis());

	var handler = new DefaultErrorHandler(exponentialBackOff);

	// do not specify any fatal exceptions
	handler.setClassifications(Map.of(), true);

	return handler;
}