DevOps - High Availability Platform Resource Deployment Charts
- Francisco Javier López Acevedo
Available since version 3.1.0-KickOff
Introduction
This version offers several Charts to deploy different Platform components in high availability, both in Kubenetes and Openshift.
These charts are detailed below.
MongoDb Chart in Replica Set
The real-time database could be integrated into the Platform both in containerized, single-node mode, and external to the Kubernetes cluster both in single-node mode and in Replica Set.
With this chart it can be deployed within the Kubernetes cluster in high availability/Replica Set using a Kubernetes StatefulSet.
Each Mongo node or pod, thanks to the StatefulSet object, will have its own persistence assigned to it in the form of a Persistent Volume Claim:
Chart de Kafka/Zookeeper and HA
Historically, kafka and zookeeper deployments in Kubernetes consisted of a pod/container deployed on a single node. For infrastructures with more than one worker, this architecture is not sufficient to guarantee the high availability of these services.
To solve this problem, two independent charts have been created that deploy 3 pods/containers on separate nodes, thus guaranteeing high availability service.
For Zookeeper, each deployment will have its own associated resources in kubernetes: pvc, pv and service.
In the case of Kafka, the picture is the same, each deployment has its own kubernetes resources associated with it.
Chart de OpenDistro/Elasticsearch and HA
In previous releases, the OpenDistro project was included for use as an audit database, in addition to the possibility of using it as a RealTimeDB. When this service needs to be deployed in multi-node environments, the need for HA arises.
OpenDistro needs several unique resources for each pod, such as certificates and configuration files. Therefore, a chart has been created that deploys 3 deployments, each with its associated resources.
In addition to the service associated with each pod, an additional service is deployed in Kubernetes that acts as a balancer between the 3 nodes.
Chart of historical DB deployment (Presto-MinIO) in single-instance and HA mode
In addition to being released in this quarter, two very important functionalities, Presto and MinIO, are offered for deployment with Helm in clusters of k8s.
MinIO is an S3-compliant object storage system that enables secure and replicated storage of large volumes of information.
PrestoDB is a Big Data type SQL query engine, which works in distributed mode and is very efficient.
Charts for these two components are provided as follows:
Chart for MinIO deployment in development environments.
Chart for PrestoDB and Hive deployment in development environments.
Chart for MinIO High Availability deployment in production environments.
Chart for High Availability deployment of PrestoDB and Hive in production environments.
Once MinIO is deployed in development mode, two subcomponents can be distinguished: on the one hand the administrative console and on the other hand the storage server.
In the case of Presto in "minimal" environments without HA, two subcomponents are also distinguished: the Hive metadata server with MariaDB destination and the worker/coordinator.
