Solr and Kubernetes - Automated Horizontal Scalability Test

08/07/2024

Auteur(s) :

Benjamin Dauvissat

Temps de lecture : 4 minute(s)

Tests for implementing automated horizontal scalability on a Solr 9 cluster in a Kubernetes environment

Solr and Kubernetes - Automated Horizontal Scalability Test

Introduction

A Solr cluster consists of multiple nodes.

Thanks to the horizontal scalability offered by Kubernetes, it is possible to adjust the size of the Solr cluster based on the resources consumed.

Here is an application of this capability, based on this article

Prerequisites

To facilitate various installations, we decided to use Helm.

Install Helm

The implementation of automatic scalability is based on metrics.

In this purpose, the installation of Metrics Server is very useful if there is no existing way to access Kube cluster metrics.

Install Metrics Server

The version of the Solr operator used is 0.8.1

The version of SolrCloud used for this test is 9.6.1

Installing Zookeeper

Thanks to Helm, installing Zookeeper is done with a simple command that will include all the important elements.

helm install bitnami-zookeeper oci://registry-1.docker.io/bitnamicharts/zookeeper \
--set image.tag=3.8 \
--set fourlwCommandsWhitelist="mntr\,conf\,ruok" \
--set autopurge.purgeInterval=1 \
--set heapSize=512 \
--set replicaCount=3

Important points to note:

image.tag specifies which version of Zookeeper will be installed.

replicaCount indicates the number of Zookeeper nodes to be deployed. 3 is the ideal number to start with for High Availability.

Installing Solr

Installing CRDs (Custom Resources Definitions)

The first step is to install the specific CRDs for SolrCloud.

kubectl create -f https://dlcdn.apache.org/solr/solr-operator/v0.8.1/crds/solrclouds.yaml

Installing the operator

Next, the Solr operator can be installed on our Kube cluster.

helm install solr-operator apache-solr/solr-operator --version 0.8.1 \
--set zookeeper-operator.install=false

Deploying the cluster

helm install example-solr apache-solr/solr --version 0.8.1 \
--set image.tag=9.6.1 \
--set solrOptions.javaMemory="-Xms500m -Xmx500m" \
--set zk.address="bitnami-zookeeper-headless:2181" \
--set podOptions.resources.requests.cpu=200m \
--set addressability.external.method=Ingress \
--set addressability.external.domainName="ing.local.domain" \
--set addressability.external.useExternalAddress="true" \
--set ingressOptions.ingressClassName="nginx"

Two important points to note:

The lines starting with addressability and ingress are to be used only if an Ingress service is used in the Kube cluster. In this case, configuration for the new Solr service will need to be created.
In some cases, the cluster may behave illogically. This may be due to a lack of allocated memory. By default, each pod is allocated 500 MB of memory. This can be modified by adding a line:

helm install example-solr apache-solr/solr --version 0.8.1 \
 --set image.tag=9.6.1 \
 --set solrOptions.javaMemory="-Xms500m -Xmx500m" \
 --set zk.address="bitnami-zookeeper-headless:2181" \
 --set podOptions.resources.requests.cpu=200m \
 --set podOptions.resources.requests.memory=1Gi \
 --set addressability.external.method=Ingress \
 --set addressability.external.domainName="ing.local.domain" \
 --set addressability.external.useExternalAddress="true" \
 --set ingressOptions.ingressClassName="nginx"

Initial situation

In our example, a collection addresses was created explicitly specifying that it should have 5 shards and one replica.

Then, 10,000 documents were indexed.

The visualization in the Solr UI shows the following result:

Cluster overview with 3 nodes. Node 0 contains one replica, the other two contain two each

Treev view of the shards. Shards 1 and 3 are on node 1. Shard 2 is on node 0 and shards 4 and 5 are on node 2

Implementing autoscale

Implementing autoscale is simple: specify the limit that will trigger an upscaling, as well as the minimum and maximum number of pods allowed. For example:

kubectl autoscale solrcloud example --cpu-percent=2 --min=3 --max=6

Here, the limit is specified as above 2% CPU consumption (the value is deliberately low to force autoscaling without overloading the Solr cluster).

Once the command is executed, we quickly end up with 6 pods, which can be easily verified with a few simple commands.

benjamin@kube01:~$ kubectl get pods
NAME                             READY   STATUS    RESTARTS       AGE
bitnami-zookeeper-0              1/1     Running   3 (11m ago)    46h
bitnami-zookeeper-1              1/1     Running   3 (11m ago)    46h
bitnami-zookeeper-2              1/1     Running   3 (11m ago)    46h
example-solrcloud-0              1/1     Running   0              47s
example-solrcloud-1              1/1     Running   0              70s
example-solrcloud-2              1/1     Running   0              98s
example-solrcloud-3              1/1     Running   0              32s
example-solrcloud-4              1/1     Running   0              32s
example-solrcloud-5              1/1     Running   0              32s
solr-operator-575d5866f4-8wrrv   1/1     Running   11 (11m ago)   46h

As well as the horizontal scaling verification command (HPA)

kubectl get hpa

Which gives

NAME      REFERENCE           TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
example   SolrCloud/example   3%/2%     3         6         6          163m

Cluster status after autoscale

After this update, we end up with a cluster composed of 6 nodes. How was the distribution of the different shards managed?

Cluster overview with 6 nodes. Nodes 0 to 5 contain a replica. Node 5 does not

Tree view of the shards. Shards 1 to 5 are on nodes 3, 4, 0, 1 and 2 respectively

Conclusion

This small test shows that in the case where a Solr cluster is managed in Kubernetes via the dedicated operator and the number of nodes increases, data distribution is automatically handled to have the shards spread across a maximum number of nodes.