adjoe Engineers’ Blog
 /  Backend  /  Running Apache Kafka® on Spot Instances
abstract design with kafka logo embedded
Backend

Running Apache Kafka® on Spot Instances

Apache Kafka is an open-source distributed event-streaming platform. At adjoe we deploy our Kafka cluster on Kubernetes and use it for event streaming – but also in some cases as an event bus. 

Some of the applications that process requests publish messages to Kafka topics. This means that the Kafka brokers should be reliable. Running reliable Kafka deployment can be costly. The high costs come from the way Kafka achieves the resiliency; in order to avoid unplanned downtimes, the data should be replicated across brokers. 

Here at adjoe we always consider the financial impact of our solutions without sacrificing the reliability of our product. Our solutions need to be scalable, reliable, and cost-effective. An easy way of decreasing the costs is using AWS spot instances instead of on-demand instances. Spot instances can be up to 90 percent cheaper than on demand. 

In this article, I will showcase how we managed to run self-managed Apache Kafka on AWS spot instances to cut costs by around 60 percent.

The Setup before the Switch

  • We usually use a replication factor of 3 for our topics with minimum in-sync replicas set to 2. 
  • We use segmentio/kafka-go as our Go Kafka client indirectly by using justtrackio/gosoline. This is a framework for creating Go applications developed by our sister company justtrack.
  • We publish the messages in async mode.
  • Our Kafka and Zookeeper run on Kubernetes.

Which Problems Did We Try to Solve?

When switching from an on-demand deployment to a spot instance deployment, you should expect the nodes to go down at any time. When a node that runs a Kafka broker goes down, all the partitions for which this broker was leader for will become unavailable, a new leader will need to be elected – but this process can sometimes be a bit slow. Some in-flight requests may also exceed the timeout, and some of the error responses are not retryable. 

There are use cases when it would be acceptable for the request to return an error and then be retried. But in some cases, we don’t want to propagate the error back to the user, so we have to guarantee that the messages will eventually be produced. In theory that would mean having to keep the messages in memory until we can write them to Kafka, but if the leader election takes too long, we risk losing those messages due to OOM kill.

The Idea

When a broker goes down, all the partitions for which the broker is a leader will become unavailable until a new leader is elected. Kafka uses a key to partition the messages. There can be multiple strategies, but usually the default partitioner is used. The default partitioner guarantees that all the messages with the same partition key will be assigned to the same partition. 

In our use case, this guarantee is not important, so we asked ourselves: “What would happen if we were to change that behavior, so that when we detect a partition is offline, we try to send the message to a different partition?” And that is what we implemented as an experiment.

Without Active Partition Balancer

diagram showing adjoe running Apache Kafka cluster without active partition balancer

With Active Partition Balancer

diagram showing adjoe running Apache Kafka cluster with active partition balancer

How Does It Work?

First we had to get rid of the async writing because we want to be able to detect if the message we try to write failed or not. This async writing functionality was provided by the Kafka-Go client.

Next we had to implement our own partitioner, which would be aware of errors when we publish a message. Kafka-Go calls this partitioner a Balancer and provides an interface.

type Balancer interface {
   Balance(msg Message, partitions ...int) (partition int)
}

As you can see, this interface takes the message to be produced and a slice of partitions. For example, if your topic has five partitions, the call would look like this:

p := Balance(msg, 0, 1, 2, 3, 4)

If we want to introduce a mechanism that can react when a write request fails, the Balancer should be aware of that. We created a new interface to do this.

type KafkaBalancer interface {
   kafka.Balancer


   OnSuccess(kafka.Message)
   OnError(kafka.Message, error)
}

Now we can notify the Balancer when an error happens. 

Next we created a new Balancer that we call activePartitionBalancer that implements the KafkaBalancer interface. This new Balancer maintains a list of circuit breakers per topic and partition.

How Does activePartitionBalancer Work?

When a new message is about to be balanced, this is how it works.

diagram showing how activePartitionBalancer works
  • When the write message operation fails, the error is passed to the onError function of the Balancer, where it registers the failed attempt.
  • When the write message operation succeeds, the message is passed to the OnSuccess function of the Balancer, where it will reset the partition circuit breaker.

You can find all the implementation details here.

Things We Consider When We Write Cost-Effective Code

  • Try to take advantage of the spot instances whenever possible. 
  • If you doubt that a service can run in spot instances, you can always perform an experiment and evaluate your ideas.
  • Do not settle down – re-evaluate your solutions.
  • Design the code in a way that can withstand unexpected disruptions. See chaos engineering.

BI Analyst (f/m/d)

  • adjoe
  • Programmatic Supply
  • Full-time
adjoe is a leading mobile ad platform developing cutting-edge advertising and monetization solutions that take its app partners’ business to the next level. Part of the applike group ecosystem, adjoe is home to an advanced tech stack, powerful financial backing from Bertelsmann, and a highly motivated workforce to be reckoned with.

 Meet Your Team: Programmatic Supply

Did you know that in-app ads are sold in a real-time auction before they get rendered in thousands of mobile apps? Dozens of ad networks compete for every single view, choosing their best ad and deciding what price to bid in just a couple hundred milliseconds. We at adjoe have developed our own ad platform WAVE, that connects ad networks and takes part in this competition itself. Our backend system can simultaneously handle a few billion auctions in real-time every day. Behind the auction scenes, we have an impressive prediction model working to figure out if an ad will lead to installing an app or not.

Want to be a part of this industry-first adtech solution? Join our discussions, explore implementation, and put your problem-solving skills to the test in our cross-functional Programmatic team!
What You Will Do
  • Analyze data from multiple sources (AWS Athena, AWS QuickSight, Kibana,  internal dashboards, spreadsheets and more) on a regular basis and extract insights from it.
  • Investigate user behavior and product performance by analyzing data, reading logs and talking to engineers.
  • Document your findings and communicate them to the team.
  • Prepare calculations for A/B tests launches and analyze experiments results.
  • Challenge and contribute to product-related decisions based on your insights.
  • Who You Are
  • You have a degree in information technology, economics, analytics, or a similar field.
  • You have 2+ years’ of relevant experience, preferably in product companies.
  • You’re proficient in SQL.
  • You have a strong knowledge of Python or R.
  • You have an excellent understanding of data visualization and have experience building dashboards (e.g. QuickSight, MS PowerBI, Tableau, R Shiny, Streamlit) to showcase analysis results.
  • You know how to perform statistical analyses (e.g. use hypothesis tests to evaluate A/B tests).
  • You’re excited to work with data and have experience in extracting business insights from it.
  • You worked closely with engineers (e.g. investigating how the system works) and are comfortable reading tech documentation.
  • You have experience in working with task-tracking tools (Jira, GitLab, etc).
  • Plus: Experience in AdTech or Mobile App analytics
  • Heard of Our Perks?
  • Work-Life Package: 2 remote days per week, 30 vacation days, 3 weeks per year of remote work, flexible working hours, dog-friendly kick-ass office in the center of the city.
  • Relocation Package: Visa & legal support, relocation bonus, reimbursement of German Classes costs and more.
  • Happy Belly Package: Monthly company lunch, tons of free snacks and drinks, free breakfast & fresh delicious pastries every Monday
  • Physical & Mental Health Package: In-house gym with personal trainer, various classes like Yoga with expert teachers.
  • Activity Package: Regular team and company events, hackathons.
  • Education Package: Opportunities to boost your professional development with courses and trainings directly connected to your career goals 
  • Wealth building: virtual stock options for all our regular employees.
  • Free of charge access to our EAP (Employee Assistance Program) which is a counseling service designed to support your mental health and well-being.
  • Skip writing cover letters. Tell us about your most passionate personal project, your desired salary and your earliest possible start date. We are looking forward to your application!

    We welcome applications from people who will contribute to the diversity of our company.

    We’re programmed to succeed

    See vacancies