How to Build Your Own Cloud Architecture & Migrate to Kubernetes

Our Cloud Engineering team at adjoe is committed to choosing when to use self-hosted or managed services to reduce USD 300,000 of monthly hosting costs – as well as still ensuring convenience and data security for our developers.

In this series of short-but-sweet articles, I – as a DevOps engineer – will walk you through the process of migrating from a traditional Elasticsearch cluster on spot EC2 instances to a cluster running on Kubernetes.

What Is Kubernetes Used for at adjoe? 

First things first: Kubernetes automates the networking, scaling, deployment, and availability of containerized workloads and services. Elasticsearch is a real-time, scalable search engine that supports full-text and structured search as well as in-depth analytics. It is useful for storing and searching through large amounts of textual data, such as logs. It can also search through a wide variety of documents.

Previously, we were running Elasticsearch on spot EC2 instances. This required a lot of scripting and configuration or using third-party services. Now, we use Kubernetes and Elasticsearch operator to deploy Elasticsearch clusters, which improves the availability, scalability, and maintainability of the overall cluster. This ultimately increases the reliability of the system.

The first thing we need to understand is the following: 

• How was the cluster initially running? 
• Which problems arose and thus demanded the migration over to Kubernetes?

After clarifying these, I’ll describe which list of tools you need to migrate to Kubernetes.

Why Migrate to Kubernetes?

Initially, the Elasticsearch cluster was running on AWS EC2 instances. The cluster was scalable; however, the costs of running on-demand instances were quite high. We therefore needed instances with a spot lifecycle, which AWS provides at a significantly lower cost. 

So, we chose instances with a spot lifecycle. On one hand, this reduced costs considerably; on the other hand, it also introduced some issues, which required manual intervention or further automation. In order to have a reliable option where we could use an EC2 spot lifecycle as well as reduce manual intervention, we chose to migrate to Kubernetes with an operator that manages the Elasticsearch cluster with little to no human intervention.

In order to handle databases such as Elasticsearch, you can use Kubernetes resources, including StatefulSets and PersistentVolumeClaims.

This is what you’ll need resource-wise:

• A Kubernetes cluster that is up and running (we use AWS EKS)
• The ECK operator from Elasticsearch
• An infrastructure as code platform (we use Terraform)
• Helm charts
• A single sign-on provider for authentication

Below, I’m giving you an overview of the architecture of the Elasticsearch cluster running on Kubernetes.

What happens next? Stay tuned for my next article, in which I’ll talk about the following:

• The essential configurations that would allow us to create an Elasticsearch cluster
• Automating various manual tasks, so that when we deploy the cluster, all the post-deploy ES and Kibana configurations are in place
• Kibana configurations, such as index lifecycle management, index templates, Kibana data views, etc.
• Using Kubernetes ConfigMaps as a resource