adjoe Engineers’ Blog
 /  Infrastructure  /  Migrate to Kubernetes
decorative purple image with kubernetes logo
Infrastructure

How to Build Your Own Cloud Architecture & Migrate to Kubernetes

Our Cloud Engineering team at adjoe is committed to choosing when to use self-hosted or managed services to reduce USD 300,000 of monthly hosting costs – as well as still ensuring convenience and data security for our developers.

In this series of short-but-sweet articles, I – as a DevOps engineer – will walk you through the process of migrating from a traditional Elasticsearch cluster on spot EC2 instances to a cluster running on Kubernetes.

What Is Kubernetes Used for at adjoe? 

First things first: Kubernetes automates the networking, scaling, deployment, and availability of containerized workloads and services. Elasticsearch is a real-time, scalable search engine that supports full-text and structured search as well as in-depth analytics. It is useful for storing and searching through large amounts of textual data, such as logs. It can also search through a wide variety of documents.

Previously, we were running Elasticsearch on spot EC2 instances. This required a lot of scripting and configuration or using third-party services. Now, we use Kubernetes and Elasticsearch operator to deploy Elasticsearch clusters, which improves the availability, scalability, and maintainability of the overall cluster. This ultimately increases the reliability of the system.

The first thing we need to understand is the following: 

  • How was the cluster initially running? 
  • Which problems arose and thus demanded the migration over to Kubernetes?

After clarifying these, I’ll describe which list of tools you need to migrate to Kubernetes.

Why Migrate to Kubernetes?

Initially, the Elasticsearch cluster was running on AWS EC2 instances. The cluster was scalable; however, the costs of running on-demand instances were quite high. We therefore needed instances with a spot lifecycle, which AWS provides at a significantly lower cost. 

So, we chose instances with a spot lifecycle. On one hand, this reduced costs considerably; on the other hand, it also introduced some issues, which required manual intervention or further automation. In order to have a reliable option where we could use an EC2 spot lifecycle as well as reduce manual intervention, we chose to migrate to Kubernetes with an operator that manages the Elasticsearch cluster with little to no human intervention.

In order to handle databases such as Elasticsearch, you can use Kubernetes resources, including StatefulSets and PersistentVolumeClaims.

This is what you’ll need resource-wise:

  • A Kubernetes cluster that is up and running (we use AWS EKS)
  • The ECK operator from Elasticsearch
  • An infrastructure as code platform (we use Terraform)
  • Helm charts
  • A single sign-on provider for authentication

Below, I’m giving you an overview of the architecture of the Elasticsearch cluster running on Kubernetes.

Elasticsearch architecture on Kubernetes

What happens next? Stay tuned for my next article, in which I’ll talk about the following:

  • The essential configurations that would allow us to create an Elasticsearch cluster
  • Automating various manual tasks, so that when we deploy the cluster, all the post-deploy ES and Kibana configurations are in place
  • Kibana configurations, such as index lifecycle management, index templates, Kibana data views, etc.
  • Using Kubernetes ConfigMaps as a resource

Senior DevOps Engineer (f/m/d)

  • adjoe
  • Cloud Engineering
  • Full-time
adjoe is a leading mobile ad platform developing cutting-edge advertising and monetization solutions that take its app partners’ business to the next level. Part of the applike group ecosystem, adjoe is home to an advanced tech stack, powerful financial backing from Bertelsmann, and a highly motivated workforce to be reckoned with.  We are looking for a Senior DevOps engineer to strengthen the Cloud Engineering team.

Meet Your Team: Cloud Engineering

The Cloud Engineering team is the core of adjoe’s tech department. It is responsible for the underlying infrastructure that helps adjoe’s developers to run their software – and the company to grow its business. 

From various AWS services to Kubernetes and Apache foundation open source projects, the team continuously validates new cloud and architecture services to efficiently handle a huge amount of data. Cloud Engineering tackles the challenge of choosing when to use self-managed or managed services to reduce $300K of monthly hosting costs, while still ensuring convenience and data security for adjoe’s developers. 

Because adjoe needs to ensure high-quality service and minimal downtime to grow its business, Cloud Engineering invests heavily in monitoring and alerting technologies for insights into system health (networking, application logs, cloud service information, hardware, etc.). The cloud engineers also provide working solutions, knowledge, and documentation to the entire community of adjoe developers, giving them the autonomy to work on the infrastructure themselves and ensure the smooth sailing of adjoe’s systems. 
What You Will Do
  • You will work together in a team of experienced DevOps engineers to reinvent our cloud infrastructure by introducing new technologies and improving the existing environment.
  • You will help transfer our current managed AWS cloud infrastructure to self-hosted and open source technologies: We believe a hybrid combination between managed and self-hosted offers the best cost/efficiency ratio.
  • You will support our developers in building a high-performance backend with Go (based on our existing backend structures separated over several globally located data centers).
  • You will collaborate with experts from different technological backgrounds and countries, learn from highly experienced colleagues, and share your knowledge.
  • You will work with our current tech stack: Go, DruidDB, Kafka, DynamoDB, ScyllaDB, RDS, Kubernetes, Terraform, Gitlab, ECS, EMR, Lambda, complex CI/CD pipelines, Prometheus, Data pipelines, and many more.
  • You will introduce new technologies, including migrating part of the architecture to our new Kubernetes and Kafka clusters and introducing Apache Spark/Flink and our own hosted object storage.
  • You will troubleshoot issues in complex systems, conduct root cause analysis, and implement appropriate solutions.
  • You will collaborate with junior team members and foster their professional growth.
  • Who You Are
  • You are a skilled DevOps engineer/SRE/Platform engineer with a strong passion for improving scalability and cloud infrastructure and a keen interest in coding.
  • You have an extensive experience with Kubernetes and ECS.
  • You have a profound understanding of the AWS ecosystem and infrastructure as code with Terraform.
  • You have good knowledge of microservice architecture and c communication between microservices (e.g. Topics, Queues, Object storage, etc.).
  • You have a deep understanding of CI/CD tools and experience with building and maintaining pipelines.
  • You have strong problem-solving skills and ability to tackle complex technical challenges.
  • You are self-motivated and eager to learn new technologies and tools.
  • You are open to relocating to Hamburg, Germany
  • Heard of Our Perks?
  • Work-Life Package: 2 remote days per week, 30 vacation days, 3 weeks per year of remote work, flexible working hours, dog-friendly kick-ass office in the center of the city.
  • Relocation Package: Visa & legal support, relocation bonus, reimbursement of German Classes costs, and more.
  • Happy Belly Package: Monthly company lunch, tons of free snacks and drinks, free breakfast & fresh delicious pastries every Monday
  • Physical & Mental Health Package: In-house gym with a personal trainer, various classes like Yoga with expert teachers & free of charge access to our EAP (Employee Assistance Program) to support your mental health and well-being
  • Activity Package: Regular team and company events, and hackathons.
  • Education Package: Opportunities to boost your professional development with courses and training directly connected to your career goals 
  • Wealth building: virtual stock options for all our regular employees
  • Skip writing cover letters. Tell us about your most passionate personal project, your desired salary and your earliest possible start date. We are looking forward to your application!

    We welcome applications from people who will contribute to the diversity of our company.

    Conquer cloud technologies at adjoe

    See vacancies