Platform Monkey ctx

Ray on Kubernetes: Distributed Python and AI Workloads Without the YAML Pain

· By
Sources

Running distributed Python at scale usually involves a choice between "too much manual effort" (hand-rolling SSH scripts and static IP lists) and "too much abstraction" (managed platforms that hide the cost and performance knobs). Ray on Kubernetes, managed via the KubeRay operator, is the professional middle ground.

But deploying a `RayCluster` CR and calling it "production-ready" is a recipe for an OOM-kill loop and network saturation. To run Ray at the 100+ node scale, you have to solve for three things: the shared memory trap, the "One Pod Per Node" rule, and GCS state persistence.

The Architectural Blueprint

Ray transforms a cluster of machines into a single shared-memory pool. Unlike Spark, which is primarily a data-flow engine, Ray is a distributed execution engine for actors and tasks.

The Control Plane Flow

KubeRay implements the operator pattern to bridge the gap between Ray's cluster-centric view and Kubernetes' pod-centric view.

User/CI -> Create RayCluster CR -> KubeRay Operator -> [Provision Head Pod + Head Service] -> [Provision Worker Groups (Pods)] -> Ray Head (GCS) registers Worker Nodes

The Global Control Store (GCS) is the brain of the cluster. It lives on the head pod and maintains the cluster's metadata: which actors are on which nodes, the status of pending tasks, and the resource map. If the head pod dies, the GCS dies, and the cluster typically loses its state—unless you have configured external state stores.

The Data Plane: Plasma and Zero-Copy

The secret to Ray's performance is Plasma, a shared-memory object store. Instead of serializing and deserializing data between processes on the same node, Ray uses shared memory (`/dev/shm`) to allow multiple workers to read the same object without copying it.



[ Ray Head Pod ] <---------------------> [ Ray Worker Pod A ]


      |                                        |


      | (GCS / Scheduler)                      | (Local Plasma Store)


      |                                        |


      +-----------------------------------------+


                           |


                [ Kubernetes Network / CNI ]


                           |


[ Ray Worker Pod B ] <-----+------> [ Ray Worker Pod C ]


      |                                        |


      | (Local Plasma Store)                   | (Local Plasma Store)

Production Implementation Guide

Step 1: Deploy the KubeRay Operator

The operator is the only way to manage the lifecycle of Ray clusters without manually tracking pod IPs.

# Add the KubeRay Helm repository

helm repo add kuberay https://ray-project.github.io/kuberay-helm/


helm repo update



Install the operator and CRDs


helm install kuberay-operator kuberay/kuberay-operator --version 1.2.2

Step 2: The Hardened RayCluster Manifest

A basic `RayCluster` manifest will fail in production because Kubernetes defaults `/dev/shm` to 64MiB, while Ray's object store defaults to 30% of node memory. This mismatch causes immediate `ObjectStoreFullError` crashes.

Below is a production-grade manifest for a GPU-accelerated cluster.

apiVersion: ray.io/v1alpha1

kind: RayCluster


metadata:


  name: ray-prod-cluster


spec:


  rayVersion: "2.30.0"


  enableInTreeAutoscaling: true


  headGroupSpec:


    rayStartParams:


      dashboard-host: "0.0.0.0"


    template:


      spec:


        containers:


    
  
  • name: ray-head
    • 



          image: rayproject/ray-ml:2.30.0


          resources:


            limits:


              cpu: "4"


              memory: "16Gi"


            requests:


              cpu: "4"


              memory: "16Gi"


          volumeMounts:


    
  
  • mountPath: /dev/shm
    • 



            name: dshm


        volumes:


    
  
  • name: dshm
    • 



          emptyDir:


            medium: Memory


            sizeLimit: "4Gi" # ~25% of head memory


  workerGroupSpecs:


    
  
  • groupName: gpu-workers
    • 



    replicas: 2


    minReplicas: 1


    maxReplicas: 10


    rayStartParams: {}


    template:


      spec:


        containers:


    
  
  • name: ray-worker
    • 



          image: rayproject/ray-ml:2.30.0


          resources:


            limits:


              cpu: "16"


              memory: "64Gi"


              nvidia.com/gpu: "1"


            requests:


              cpu: "16"


              memory: "64Gi"


              nvidia.com/gpu: "1"


          volumeMounts:


    
  
  • mountPath: /dev/shm
    • 



            name: dshm


        volumes:


    
  
  • name: dshm
    • 



          emptyDir:


            medium: Memory


            sizeLimit: "20Gi" # Match object-store-memory setting

Step 3: Verifying the Object Store

Once the pods are running, verify that `/dev/shm` is actually backed by memory and not the default 64MiB limit.

# Identify the worker pod

WORKER_POD=$(kubectl get pods -l ray.io/node-type=worker -o jsonpath='{.items[0].metadata.name}')



Check the mount size


kubectl exec -it $WORKER_POD -- df -h /dev/shm

Production Realities: Where it Breaks

The /dev/shm Memory Paradox

The most common failure in Ray-on-K8s is the "Invisible OOM". Because `/dev/shm` is a separate mount, the memory it consumes is often not tracked by the standard Kubernetes pod memory limit. You can have a pod that appears to be using 10GiB of its 64GiB limit, but the kernel OOM-kills it because the `emptyDir` memory limit was hit.

Fix: Always explicitly set `sizeLimit` on the `emptyDir` volume and ensure it matches the `object-store-memory` parameter in `ray start`.

The "One Pod Per Node" Rule

Ray is designed for high-throughput, low-latency communication. Running multiple small Ray pods on a single large node introduces massive overhead:

1. Redundant Raylets: Every pod runs its own Raylet (node manager), wasting CPU.

2. Fragmented Object Store: Instead of one large 200GB Plasma store, you have ten 20GB stores, forcing Ray to move data across the network (TCP) instead of using local shared memory.

Fix: Size your Pod requests to match the node's capacity. Use `Taints` and `Tolerations` to ensure Ray pods have exclusive access to their nodes.

Network Saturation during "Shuffle"

In large-scale training or data processing, Ray performs "all-to-all" communication. If you are using a standard 10Gbps CNI, a burst of object transfers between 50 nodes can saturate the node's network interface, causing the GCS heartbeats to time out and triggering a "node failure" cascade.

Fix: Use high-bandwidth instances (e.g., AWS `p4d` or `g5` with EFA) and monitor `node_network_transmit_bytes_total`.

Who Should Skip This

Do NOT use Ray on Kubernetes if:

References & Further Reading