Toward Efficient Orchestration for GenAI Workloads in Modern Container Clouds

As Artificial Intelligence (AI) continues to drive innovation across multiple industries and verticals, the efficient orchestration of AI workloads in modern containerized cloud environments is becoming increasingly crucial. In contrast to traditional cloud applications, modern Generative Artificial Intelligence (GenAI) workloads involving real-time inference and distributed training have unique and highly demanding computing, storage, and network requirements. This paper presents our vision for efficient orchestration of GenAI workloads in container clouds, including a set of placement strategies designed to meet the dynamic and resource-intensive needs of GenAI applications while ensuring their performance and scalability. We discuss the open challenges in orchestrating AI workloads with diverse communication patterns, such as the need for low-latency communication, high-throughput data pipelines, and efficient resource allocation across highly distributed systems. This work also highlights the importance of integrating AI-specific optimizations into popular container orchestration platforms such as the Kubernetes (K8s) platform, leveraging modern technologies such as GPU scheduling mechanisms, and distributed training and inference strategies. We envision a future where container clouds not only scale seamlessly to accommodate the growing demands of GenAI workloads but also incorporate AI-driven orchestration mechanisms that intelligently adapt to workload fluctuations, predict resource requirements, and mitigate bottlenecks. This paper aims to provide a foundational framework for efficient life-cycle management of GenAI workloads in modern cloud infrastructures, paving the way for future research and innovation in this rapidly evolving field.