Scale Boole AI on Kubernetes: GPU Pods, PVCs, and Probes

Deploy Boole AI on Kubernetes: schedule GPU pods, persist model weights with a PVC, and add liveness and readiness probes for production use.

Boole AI runs on Kubernetes using standard GPU workload patterns. Use a Deployment with a GPU resource limit to schedule pods onto GPU nodes, and a PersistentVolumeClaim to cache model weights across pod restarts so you avoid re-downloading weights every time a pod is rescheduled.

Prerequisites

Before applying the manifests below, make sure your cluster has:

• Kubernetes 1.24+
• NVIDIA GPU Operator installed on the cluster (installation guide)
• At least one GPU node available and schedulable

Basic Deployment Manifest

The manifest below creates a single-replica Deployment and a ClusterIP Service. Adjust the model slug, replica count, and namespace to match your environment.

boole-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: boole-inference
  labels:
    app: boole-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: boole-inference
  template:
    metadata:
      labels:
        app: boole-inference
    spec:
      containers:
        - name: boole
          image: ghcr.io/boole-ai/boole:latest
          args: ["serve", "--model", "llama-3.3-70b-instruct"]
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - name: weights
              mountPath: /weights
      volumes:
        - name: weights
          persistentVolumeClaim:
            claimName: boole-weights-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: boole-inference
spec:
  selector:
    app: boole-inference
  ports:
    - port: 80
      targetPort: 8000
  type: ClusterIP

Apply it with:

kubectl apply -f boole-deployment.yaml

PersistentVolumeClaim

Create a PVC to store downloaded model weights. This prevents the pod from re-downloading weights every time it restarts or is rescheduled.

boole-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: boole-weights-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

Apply the PVC before the Deployment:

kubectl apply -f boole-pvc.yaml

50 Gi is sufficient for a single 70B-class model. Increase storage if you plan to cache multiple models on the same volume.

Scaling

Each running pod requires a dedicated GPU. To increase inference concurrency, scale the number of replicas — the Kubernetes scheduler places each new pod on a node that has a free GPU.

kubectl scale deployment boole-inference --replicas=4

Keep one model per pod for isolation. Sharing a GPU across multiple pods degrades throughput and makes resource accounting unpredictable.

GPU node pools on managed Kubernetes services (GKE, EKS, AKS) typically require specific node selectors or tolerations to schedule onto GPU nodes. Add a nodeSelector or tolerations block to the pod spec to match your cloud provider’s GPU node labels.

Health Checks

Add liveness and readiness probes to the container spec so Kubernetes can detect and recover from failed inference processes without manual intervention.

boole-deployment.yaml

livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /ready
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 5

• /health — returns 200 when the process is alive. If this probe fails, Kubernetes restarts the container.
• /ready — returns 200 when the model is loaded and the server can accept requests. Traffic is only routed to the pod after this probe succeeds.

Set initialDelaySeconds: 30 on both probes to give the server enough time to load the model before Kubernetes starts polling. The cold start time is under 400 ms, but the 30-second buffer accounts for image pull time and volume mount latency on first launch.