Provisioning and Scaling Bare-metal Slurm Clusters via Cluster API and Virtual Kubelets

With the increasing adoption of AI applications, users request High Performance Computing (HPC) infrastructures for model training and classic computational applications. In contrast, cloud environments, in particular Kubernetes-based ones, tend to be preferred for big data processing and AI model inference tasks. Usually, HPC and cloud clusters operate on separated infrastructures and utilize distinct management tools. This segregation increases admin overhead and limits efficiency by hindering the movement of resources between clusters. In this paper, we propose a new method that converges the life-cycle management of HPC and cloud environments. In particular, we present an extension of the Kubernetes Cluster API (CAPI) that leverages Virtual Kubelets to enable the provisioning and bootstrapping of the Slurm workload manager. Our prototype implementation shows the benefits of adopting our extended Cluster API as the unifying interface to both setup and scale bare-metal Kubernetes/Slurm clusters.