Benchmarking and Scaling Earth Observation Foundation Models with vLLM and Kubernetes Orchestration

Earth Observation (EO) foundation models are increasingly transitioning from research environments into production to address global environmental challenges such as floods, wildfires, and other rapidly evolving natural hazards. Their impact depends on the ability to process large scale satellite data with consistent performance. To achieve this, models benefit from being served in a standardised manner, deployed within scalable cloud infrastructures and orchestrated to maintain service quality even in the case of sudden spikes in demand.

The additional requirements deriving from running EO models at scale introduce several operational challenges. Models need to be exposed via inference services allowing clients to request the processing of satellite images. Inference servers hosting running models must be easy to deploy, update, and scale. Moreover, to handle a large volume of requests and ensure efficient use of high-value resources like GPUs, inference server replicas should scale based on incoming traffic and pre-determined serving capacity.

As a step toward exploring and addressing these operational challenges, we integrated TerraTorch-based foundation models, like Prithvi and TerraMind, into the vLLM inference server. vLLM, now the de facto open-source standard for serving LLMs, exposes models through a consistent REST API and simplifies running EO models in cloud environments for development and testing. Using Prithvi-EO-2.0-300M-TL-Sen1Floods11 as the target model, we leveraged the vLLM benchmarking tool vllm bench to evaluate inference performance metrics at different levels of traffic load. Our results show that vLLM’s advanced scheduling mechanisms deliver a 4.4× throughput improvement over serial execution when processing 512×512 satellite tiles, demonstrating its suitability for high-concurrency Earth Observation use-cases.

Building on these results, we investigated how cloud-native technologies can support the deployment and scaling of EO inference services. We selected Kubernetes as our workload orchestration platform due to its large opensource community and its support for a wide range of infrastructure providers. Specifically, we adopted KServe and Knative, two projects that are part of the Kubernetes eco-system, to enable automated lifecycle management and autoscaling of vLLM replicas based on request concurrency. Our experiments show that this stack can scale vLLM instances running Prithvi-EO-2.0-300M-TL-Sen1Floods11 in under two minutes, adapting the deployment in response to changing demand.

In this presentation, we describe our end-to-end journey integrating TerraTorch EO models into the vLLM inference engine and benchmarking their performance under various conditions. Finally, we present a practical blueprint that practitioners can follow to operationalise and scale EO model inference in Kubernetes-based cloud environments using open-source technologies.