Testcontainers are a powerful library that allows you to reproduce the production environment on every developer’s computer with Docker containers. It reduces the time needed for setting up the environment and guarantees that everyone uses the same configuration, reducing the number of potential points of failure.
Integrating Testcontainers into existing application can be painful due to past technical decisions, that binds you with some framework. After reading this article you will know how to manage Testcontainers manually and how to integrate any available container with your application. This article will focus on Spring-based application with Apache Kafka container, although presented general solution can be also used with other frameworks and containers.
Before you start
I assumed, that you are familiar with Spring, have installed Docker and do know, what Testcontainers are, but you are looking how to integrate them into you application. All application code in this article is written in Java, tests are written in Spock using Groovy, but you should easily translate them into Java. To start, add Testcontainers dependency to your project. In here, we are using Apache Maven, so we will add following dependency to pom.xml.
<dependency> <groupId>org.testcontainers</groupId> <artifactId>kafka</artifactId> <version>1.12.5</version> <scope>test</scope> </dependency>
What are we working on?
Let’s take a quick look into application that will be used within this sample.
One of functionalities provided by system, is exposing details of customer. All customers are registered by Customer Support and, through Customer Management service, distributed by Message Bus to other parts of the system. What we are focusing on, is Customer Details service, that will read information from Message Bus, and expose it through REST API.
Going into details, an endpoint for User Details is exposed using Spring Web. It contains pre-formatted information processed by Service.
Controller retrieves user data from repository and maps it into desired format.
All user’s related information is sent through Message Bus, specifically Apache Kafka, inside dedicated event. All those events are mapped into entity and stored in repository for further usage by mentioned
We are using Spring Kafka for listening on topic.
Of course, we want to test our application, and it would be the best if we could do it in black-box manner. Our expected behaviour would be that
UserDetails are available through REST API by identifier. Processing of user data will be invoked by an event, sent to Apache Kafka topic. Using this approach in test, we are defining contract and clearly describe the purpose of this application, hiding implementation details. Internals of application can be unit-tested, but we will not cover this topic.
End to End tests allows us to define contract.
To perform this test, we are missing one block — Apache Kafka itself. As we are using spring-kafka and its auto-configuration, it expects that
spring.kafka.bootstrap-serversproperty contains an actual Kafka broker address. Default value points to 127.0.0.1:9092, which means that it expects we have installed and run a Kafka instance on our workstation. We clearly do not want to do this, as it requires a lot of work and maintenance on every developer’s workstation that will work on our codebase. Another way is to use
EmbeddedKafka, but it is not meant to be used with this type of tests.
Testcontainers to the rescue!
Testcontainers are able to run Docker images in its ecosystem directly from your code. They are providing multiple, ready to use modules, including Kafka. However, the default mechanism is integrating with Spring very well. Started Kafka container is exposing random port for its broker, so we will need to find a way to obtain and provide it to application with
spring.kafka.bootstrap-servers property. To achieve this, we could use managed container.
Manual managing container can be performed thanks to one interface. Basically, there are just two methods that we are interested in, provided by
stop(). While invoking
start() method, container is fetching all its dependencies and start them (this is a recurring behaviour, so dependencies of dependencies are also started). After dependencies are set up, container is configured, startup is attempted, and you are ready to go! There are multiple ways to stop the container. Let’s go briefly through them:
Startableinterface, you can invoke
stop()method. It will perform cleanup and will stop container.
- All Testcontainers are also implementing
AutoCloseable, so they will be stopped automatically if you set up them within try-with-resources block.
close()method defined in this interface is invoking
Startable. However, this will not work in our case, as we want only one instance of container to be used for all tests within application.
- Third option, most automated one, causes that we do not need to care about stopping it at all. It does not need any configuration, Testcontainers are using it by default. During startup, there is one more container started, called ryuk. Ryuk is responsible for stopping and removing all created containers after JVM process is stopped, so basically it will happen after tests will be finished. On the end of cleanup, ryuk shuts down itself. It will require the least effort from us to do it, so we will use this approach.
Starting a container
We will manually create and start container, basing on Testcontainers Kafka module. To ensure usage of only one instance of container per JVM, we will use Singleton Pattern implemented as wrapper. You can use almost any implementation, but for our sample we will use simplified check.
KafkaContainerWrapper is used only in tests, so it is written in Groovy.
KafkaContainer randomises exposed port that is mapped to the broker. It gives a lot of benefits (like possibility to easily run multiple containers at once), but somehow we need to find broker address. Good thing is that authors of Testcontainers exposed an
getBootstrapServers() method that gives us exactly what we need. Now we only need to provide this address to spring-kafka.
Spring Boot allows to provide application properties in multiple ways. One of the possibilities is to use Java System property, which will be used in our case. It benefit is overriding application properties from file, while it still can be overridden by other configuration approaches. It might be worth to take a look into
PropertySource interface, and provide your own implementation of it, but we will not use it. If you are interested in, check the framework documentation. Choose your approach and provide bootstrap servers address to the application.
If you are working with legacy system that uses different property name for broker address, you can also expose it here.
Putting everything together, we only need to invoke
setupSpringProperties(). Only thing to remember is that container address is not available until it is started, so we have to invoke setup after container startup.
You can also setup Spring properties in overridden start method.
Usage and configuration
Very last piece is to invoke the startup of our cluster and create required Kafka topics. Using spring-kafka we can create a @Configuration annotated class in a test that will use
KafkaAdmin to create a new topic. Spring auto-configuration mechanism will automatically detect beans of
NewTopic type, and create topics on Kafka instance. Prior to that, we have only to start one. Using our Wrapper and
static final field, we will have a guarantee that container will be started before
KafkaAdmin will attempt to create topics. In case if you do not use Spring, you can obtain wrapper instance in a base test class and extend this base class in all integration tests.
If you need another topics, provide another bean definition.
Thats all! Right now, running tests should create a container and topics, allowing you to create and test application faster and with confidence, that it will work the same way on your workstation and production.
Testcontainers for Spring Boot
There is an official Testcontainers support for Spring Boot applications, that supports multiple modules. Its Kafka support provides some nice features, like changing Apache Kafka image version with property, or configure topic names with comma-separated property value. However, I decided not to use it, as it seems not to be further developed, can be tricky to integrate it with already existing application, which is using different property names than Spring defaults, and it is not using provided
GenericContainer allows you to run any Docker image, but it requires more configuration, while
KafkaContainer is already preconfigured and maintained.
By now, you should be able to manually create a container instance with Wrapper approach, configure Spring application with additional properties and create topics that will be used during tests. Wrapper approach is pretty flexible while using available modules, but can also provide different containers available in Docker Hub by extending
GenericContainer. If you need to take a deeper look into the sample, you can find presented code on my GitHub. Read also about optimising Testcontainers startup time.