OpenShift 4: Pushing my cluster to the limit

When I involved in the design of Red Hat OpenShift 4 platform for customer deployment, the common question coming from the customer was how much my cluster can take the load, how many object my cluster can take and ultimately how I can push my cluster to the limit.

There is no magic number to this, each cluster in each specific environment will give different behaviour and result of the performance due to multiple factors (such as network performance, cpu capacity and capability, etc).

There are many ways of load testing (or soak testing) the cluster. Here we are going to talk about the cluster loader .

Cluster loader works by creating object defined by the user and the same time measuring performance metrics (from the OpenShift monitoring Grafana dashboard).

The cluster loader can be executed by running a origin-tests container image:

#> podman pull quay.io/openshift/origin-tests:4.9

There are built-in test suite, as an example:

#> podman run -v ${LOCAL_KUBECONFIG}:/root/.kube/config:z -i quay.io/openshift/origin-tests:4.9 /bin/bash -c 'export KUBECONFIG=/root/.kube/config && openshift-tests run-test "[sig-scalability][Feature:Performance] Load cluster should populate the cluster [Slow][Serial] [Suite:openshift]"'

We also can set cluster loader configuration YAML file to suite our need, as an example to set the tuning set (stepping, delay etc):

tuningsets: 
    - name: default
      pods:
        stepping: 
          stepsize: 5
          pause: 0 s
        rate_limit: 
          delay: 0 ms

Once we have the test result, we now can have a better view of how much our cluster can take. This also give us some hint if we need to give more resource to the cluster.

It is recommended to extend this with application load testing as well. It is a peace of mind to know what is the maximum load the cluster can take with the given resource with the maximum object deployed while application is running at maximum capacity.

Therefore, we can do a well informed capacity planning and expansion when required.