adjoe Engineers’ Blog
 /  Backend  /  Running Apache Kafka® on Spot Instances
abstract design with kafka logo embedded
Backend

Running Apache Kafka® on Spot Instances

Apache Kafka is an open-source distributed event-streaming platform. At adjoe we deploy our Kafka cluster on Kubernetes and use it for event streaming – but also in some cases as an event bus. 

Some of the applications that process requests publish messages to Kafka topics. This means that the Kafka brokers should be reliable. Running reliable Kafka deployment can be costly. The high costs come from the way Kafka achieves the resiliency; in order to avoid unplanned downtimes, the data should be replicated across brokers. 

Here at adjoe we always consider the financial impact of our solutions without sacrificing the reliability of our product. Our solutions need to be scalable, reliable, and cost-effective. An easy way of decreasing the costs is using AWS spot instances instead of on-demand instances. Spot instances can be up to 90 percent cheaper than on demand. 

In this article, I will showcase how we managed to run self-managed Apache Kafka on AWS spot instances to cut costs by around 60 percent.

The Setup before the Switch

  • We usually use a replication factor of 3 for our topics with minimum in-sync replicas set to 2. 
  • We use segmentio/kafka-go as our Go Kafka client indirectly by using justtrackio/gosoline. This is a framework for creating Go applications developed by our sister company justtrack.
  • We publish the messages in async mode.
  • Our Kafka and Zookeeper run on Kubernetes.

Which Problems Did We Try to Solve?

When switching from an on-demand deployment to a spot instance deployment, you should expect the nodes to go down at any time. When a node that runs a Kafka broker goes down, all the partitions for which this broker was leader for will become unavailable, a new leader will need to be elected – but this process can sometimes be a bit slow. Some in-flight requests may also exceed the timeout, and some of the error responses are not retryable. 

There are use cases when it would be acceptable for the request to return an error and then be retried. But in some cases, we don’t want to propagate the error back to the user, so we have to guarantee that the messages will eventually be produced. In theory that would mean having to keep the messages in memory until we can write them to Kafka, but if the leader election takes too long, we risk losing those messages due to OOM kill.

The Idea

When a broker goes down, all the partitions for which the broker is a leader will become unavailable until a new leader is elected. Kafka uses a key to partition the messages. There can be multiple strategies, but usually the default partitioner is used. The default partitioner guarantees that all the messages with the same partition key will be assigned to the same partition. 

In our use case, this guarantee is not important, so we asked ourselves: “What would happen if we were to change that behavior, so that when we detect a partition is offline, we try to send the message to a different partition?” And that is what we implemented as an experiment.

Without Active Partition Balancer

diagram showing adjoe running Apache Kafka cluster without active partition balancer

With Active Partition Balancer

diagram showing adjoe running Apache Kafka cluster with active partition balancer

How Does It Work?

First we had to get rid of the async writing because we want to be able to detect if the message we try to write failed or not. This async writing functionality was provided by the Kafka-Go client.

Next we had to implement our own partitioner, which would be aware of errors when we publish a message. Kafka-Go calls this partitioner a Balancer and provides an interface.

type Balancer interface {
   Balance(msg Message, partitions ...int) (partition int)
}

As you can see, this interface takes the message to be produced and a slice of partitions. For example, if your topic has five partitions, the call would look like this:

p := Balance(msg, 0, 1, 2, 3, 4)

If we want to introduce a mechanism that can react when a write request fails, the Balancer should be aware of that. We created a new interface to do this.

type KafkaBalancer interface {
   kafka.Balancer


   OnSuccess(kafka.Message)
   OnError(kafka.Message, error)
}

Now we can notify the Balancer when an error happens. 

Next we created a new Balancer that we call activePartitionBalancer that implements the KafkaBalancer interface. This new Balancer maintains a list of circuit breakers per topic and partition.

How Does activePartitionBalancer Work?

When a new message is about to be balanced, this is how it works.

diagram showing how activePartitionBalancer works
  • When the write message operation fails, the error is passed to the onError function of the Balancer, where it registers the failed attempt.
  • When the write message operation succeeds, the message is passed to the OnSuccess function of the Balancer, where it will reset the partition circuit breaker.

You can find all the implementation details here.

Things We Consider When We Write Cost-Effective Code

  • Try to take advantage of the spot instances whenever possible. 
  • If you doubt that a service can run in spot instances, you can always perform an experiment and evaluate your ideas.
  • Do not settle down – re-evaluate your solutions.
  • Design the code in a way that can withstand unexpected disruptions. See chaos engineering.

Product Lead (f/m/d)

  • adjoe
  • Programmatic Supply
  • Full-time
adjoe is a leading mobile ad platform developing cutting-edge advertising and monetization solutions that take its app partners’ business to the next level. Part of the applike group ecosystem, adjoe is home to an advanced tech stack, powerful financial backing from Bertelsmann, and a highly motivated workforce to be reckoned with.

Meet Your Team: WAVE Supply Services
In this competitive adtech market, adjoe stands for greater transparency and fairness for app publishers and advertisers – and a more relevant and enjoyable experience for users.
It’s exactly for this that adjoe has built its own programmatic mobile ad platform WAVE which connects app publishers with advertisers. We are working with dozens of advertising networks, measurement providers and other external services with whom we exchange millions of data points every minute. The WAVE Supply Services team is responsible for developing tools through which app publishers can manage their WAVE integration, analyze their ad monetisation performance and assess the ads’ UX through dashboards and APIs.

Join our discussions, explore implementation, and put your problem-solving skills to the test in our cross-functional Programmatic team!

As a part of the WAVE Supply Services team, you’ll be responsible for developing the face of the product: services to set up an SDK, analyze ad revenue and apps’ UX – from gathering the necessary data from our SDK to visualizing it on the dashboard or providing it via APIs.
What You Will Do
  • Build and manage a cross-functional team together with a Tech Lead: do hiring, feedback exchange, organize routines in the team.
  • Conduct competitors research, market research, customer research to come up with business requirements for the product.
  • Work with designers to prepare mockups for the UI of the product.
  • Prepare and communicate requirements to the engineers.
  • Take care of critical product documentation about features and business logic (both internal and external).
  • Work with statistics from different sources (internal system dashboards, BI tools, spreadsheets) on a regular basis to understand product use cases and identify issues.
  • Align with the company’s overall strategic goals. Work on long-term product roadmap and quarterly OKRs.
  • Align with other tech teams on common guidelines for development, design, etc.
  • Who You Are
  • You have a degree in information technology, economics, analytics, or a similar field or 5 years working experience in product or analyst positions.
  • You have 2+ years’ of experience working as Product Lead / Manager / Owner – preferably in B2B SaaS products, developing web applications.
  • You have experience hiring people, doing regular 1-1s, creating career plans.
  • You can speak both tech & business languages: discuss feature implementation with engineers and business needs with business development colleagues.
  • You have experience in basic data analysis and extracting valuable insights from data.
  • You have experience in working with task-tracking tools (Jira, GitLab, etc).
  • Plus: You have experience in using BI tools (QuickSight, Tableau, MS PowerBI).
  • Plus: You know how to work with SQL, Python, or R.
  • Plus: You experience managing mobile app/SDK development.
  • Heard of Our Perks?
  • Tech Package: Create game-changing technologies and work with the newest technologies out there.
  • Wealth building: virtual stock options for all our regular employees.
  • Work-Life Package: 2 remote days per week, 30 vacation days, 3 weeks per year of remote work, flexible working hours, dog-friendly kick-ass office in the center of the city.
  • Relocation Package: Visa & legal support, relocation bonus, reimbursement of German Classes costs and more.
  • Happy Belly Package: Monthly company lunch, tons of free snacks and drinks, free breakfast & fresh delicious pastries every Monday
  • Physical & Mental Health Package: In-house gym with personal trainer, various classes like Yoga with expert teachers.
  • Activity Package: Regular team and company events, hackathons.
  • Education Package: Opportunities to boost your professional development with courses and trainings directly connected to your career goals 
  • Free of charge access to our EAP (Employee Assistance Program) which is a counseling service designed to support your mental health and well-being.
  • Skip writing cover letters. Tell us about your most passionate personal project, your desired salary and your earliest possible start date. We are looking forward to your application!We welcome applications from people who will contribute to the diversity of our company.

    We’re programmed to succeed

    See vacancies