Part 6 — Run ML Workloads via NVIDIA GPU on Tanzu

In my last ML lab post, I wrote how A100 GPUs can be carved into MIG profiles for consumption so in this follow‑up, we’ll assign a vGPU profile to a Kubernetes namespace and deploy a small AI training environment on top of it.

The goal is to show the full lifecycle of GPU‑accelerated workloads on Tanzu:

• Deploying a vGPU‑enabled TKG cluster
• Installing the NVIDIA GPU Operator
• Scheduling GPU‑aware ML workloads (TensorFlow, PyTorch, RLlib, Jupyter, VS Code)
• Verifying GPU allocation and MIG slice usage
• Cleaning up the environment cleanly and predictably

ML Lab Topology

A raw A100 GPU can be chopped into MIG profiles as shown for seperate workloads.

Create the VM Class

Create VM Class as gpu-a100-40c and GPU Assignment: 1 × A100 MIG 40c

vGPU-enabled TKG nodes must run Ubuntu. PhotonOS cannot load the NVIDIA vGPU VIB and will fail to boot once a vGPU profile is attached.

Deploy the TKG Workload Cluster

Apply the cluster manifest:

kubectl apply: kubectl apply -f aiml-tkc.yaml apiVersion: run.tanzu.vmware.com/v1alpha3
kind: TanzuKubernetesCluster
metadata:
  name: aiml-tkg-01
  namespace: aiml-gpu-lab
  # vCenter: vcsa9.ash.local  |  Cluster: cluster01  |  DC: datacenter
spec:
  topology:
    controlPlane:
      replicas: 1
      vmClass: best-effort-medium
      storageClass: ash-vsan-policy
      tkr:
        reference:
          name: ubuntu-2204
    workers:
      nodePool:
      - name: gpuworkers
        replicas: 7
        vmClass: gpu-a100-40c
        storageClass: ash-vsan-policy
        tkr:
          reference:
            name: ubuntu-2204

kubectl apply: kubectl apply -f aiml-tkc.yaml apiVersion: run.tanzu.vmware.com/v1alpha3
kind: TanzuKubernetesCluster
metadata:
  name: aiml-tkg-01
  namespace: aiml-gpu-lab
  # vCenter: vcsa9.ash.local  |  Cluster: cluster01  |  DC: datacenter
spec:
  topology:
    controlPlane:
      replicas: 1
      vmClass: best-effort-medium
      storageClass: ash-vsan-policy
      tkr:
        reference:
          name: ubuntu-2204
    workers:
      nodePool:
      - name: gpuworkers
        replicas: 7
        vmClass: gpu-a100-40c
        storageClass: ash-vsan-policy
        tkr:
          reference:
            name: ubuntu-2204

Verify Nodes Are Ready by running kubectl get nodes

Apply Pod Security Policies by running kubectl apply -f pod-policies.yaml 

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: gpu-operator-psp
  namespace: aiml-gpu-lab
spec:
  privileged: true
  allowPrivilegeEscalation: true
  volumes:
  - '*'
  hostNetwork: true
  hostPID: true
  hostIPC: true
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: gpu-operator-psp
  namespace: aiml-gpu-lab
spec:
  privileged: true
  allowPrivilegeEscalation: true
  volumes:
  - '*'
  hostNetwork: true
  hostPID: true
  hostIPC: true
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny

Install the NVIDIA GPU Operator

This installs:

• vGPU drivers
• DCGM
• MIG manager
• Device plugin
• Toolkit runtime

kubectl get pods -n gpu-operator

Deploy a simple TensorFlow GPU pod: (Slice 5 / Ubuntu VM 5)

kubectl apply: kubectl apply -f tensorflow-deployment.yaml

apiVersion: v1
kind: Pod
metadata:
  name: tensorflow-gpu
  namespace: aiml-gpu-lab
  labels:
    app: tensorflow-gpu
spec:
  containers:
  - name: tf
    image: tensorflow/tensorflow:latest-gpu
    resources:
      limits:
        nvidia.com/gpu: "1"
    command: ["python", "train.py"]
  restartPolicy: Never

apiVersion: v1
kind: Pod
metadata:
  name: tensorflow-gpu
  namespace: aiml-gpu-lab
  labels:
    app: tensorflow-gpu
spec:
  containers:
  - name: tf
    image: tensorflow/tensorflow:latest-gpu
    resources:
      limits:
        nvidia.com/gpu: "1"
    command: ["python", "train.py"]
  restartPolicy: Never

Verify pod running: kubectl get all -n aiml-gpu-lab

Confirm All Pods + Services Running

Accesss Services by running kubectl get svc -n aiml-gpu-lab

Ray Dashboard

Used to monitor and manage distributed ML workloads running on Ray and it shows cluster resources, task execution, actors, logs, and performance metrics in real time.

Jupyter Dashboard

Provides a browser‑based interactive notebook environment for data exploration, model development, and GPU‑accelerated experimentation inside Kubernetes.

Olama Dashboard

A lightweight UI for managing local LLMs via Ollama and lets you view running models, load/unload them, inspect logs etc

TensorBoard Dashboard

Visualizes ML training metrics such as loss curves, accuracy, histograms, and model graphs which is good for understanding training behavior.

Teardown the nodes

Remove app deployments

kubectl delete -f jupyter-deployment.yaml
kubectl delete -f vscode-deployment.yaml
kubectl delete -f pytorch-deployment.yaml
kubectl delete -f rllib-deployment.yaml

kubectl delete -f jupyter-deployment.yaml
kubectl delete -f vscode-deployment.yaml
kubectl delete -f pytorch-deployment.yaml
kubectl delete -f rllib-deployment.yaml

Remove GPU Operator

helm uninstall gpu-operator -n gpu-operator

helm uninstall gpu-operator -n gpu-operator

Delete Cluster

kubectl delete -f aiml-tkc.yaml

kubectl delete -f aiml-tkc.yaml

This walkthrough uses a simulated VCF + TKG environment to demonstrate the workflow end‑to‑end. The topology, YAML, and operational flow match a real deployment, making this a safe way to understand the pattern before running it on physical hardware.

References

Here’s the full responsibility matrix for the VCF Admin role