Part 3 – VCF & NVIDIA AI-Ready Enterprise Platform PAIF-N

In the last blog, I wrote a bit about MIG & DirectPath I/O so here we will just understand where this is being applied in VCF 9..

PAIF-N also known as Private AI Foundation with NVIDIA is built on top of the VCF platform. Broadcom has partnered with Nvidia allowing large language models to run inference workloads in private data centers. Here NVIDIA offers the intuitive automation tool which includes its CUDA, cuDNN, PyTorch , deep learning VM images, vector database, and GPU monitoring capabilities offering us the Gen AI models.

An automated wizard in VCF Automation that deploys an entire AI-ready workload domain configures ESXi hosts with NVIDIA drivers, sets up NSX networking, creates the Supervisor cluster, and publishes Deep Learning VM images. What used to take a week of manual work is now a guided workflow.

Source: Broadcom

Running ML workloads on vSphere offers several advantages over cloud deployments.Some of the advantages are here..

Cheaper – Training models cloud-based could lead to unintentional leak of proprietary data in the LLM training data so with Private AI that risk is avoided. Its also super expensive to run LLM models on the cloud for long.

Inference & multi-tenancy – Cloud had been the choice for LLM Training which needs expensive GPU’s and compute but once the initial training is over the models can be moved online rather than staying on Cloud. When you need to retrain your model, you can shift the model back to cloud so there is huge cost saving here..

Sovereign AI – This is probably something you now hearing often, every wants total control of their data so If you’ve got a rack of Dell or HPE servers with NVIDIA GPUs sitting there, it makes no sense to pay hourly cloud rates when you can virtualise those GPUs and share them across teams at fraction of price of cloud and offers better privacy.

Data stays on-prem. If you’re in finance, healthcare, training data to a public cloud is often a non-starter. VCF 9 lets you run the exact same GPU-accelerated ML workloads behind inside our own datacenter…

Cost at scale. Broadcom claims significant lower TCO compared to public cloud AI infrastructure.. This is true because there is no use of running the MI model in the cloud once the initial training is over and this is where the cloud had the advantage..

One team to manage it all. Your existing VMware team can manage AI workloads with the same tools

VCF 9 gives you two ways to give VMs access to NVIDIA GPUs

DirectPath I/O – GPU Passthrough
NVIDIA vGPU – A Slice and Share approach

DirectPath I/O – GPU Passthrough – Passthrough means one VM gets the entire physical GPU. All the memory, all the compute cores, all the NVLink bandwidth. This is great for pre-training LLMs where where maximum GPU is required.

NVIDIA MIG vGPU – vGPU carves a single physical GPU into slices and and hands each slice to a different VM. VCF 9 also introduces NVLink Device Groups — the platform can bundle 2, 4, or 8 GPUs connected via NVLink/NVSwitch and allocate them as a unit to a single VM. This is best for Inference services, multi-tenant GPU sharing, dev/test environments etc..

	DirectPath I/O	NVIDIA vGPU (MIG)
Performance	Full bare-metal	Near-native (slight overhead)
GPU sharing	1 GPU = 1 VM	1 GPU = multiple VMs
Multi-GPU	Full NVLink access	NVLink device groups (2/4/8)
Best for	LLM Training	Inference & multi-tenancy

Responsibilities of a VCF Administrator in a Gen-AI world

Install A40 Driver

[root@sfo-m01-esx01.ash.local:~] cd /vmfs/volumes/sfo-m01-esx01-vmfs01/
[root@sfo-m01-esx01.ash.local:/vmfs/volumes/sfo-m01-esx01-vmfs01] ls
NVIDIA-A40-Driver_525.105.14-1OEM.800.1.0.20613240.zip
NVIDIA-A40-gpu-mgmt-daemon_525.105.14-1OEM.800.1.0.21514245.zip
install_nvidia_vib_8.0

Install the main A40 driver VIB

[root@sfo-m01-esx01.ash.local:/vmfs/volumes/sfo-m01-esx01-vmfs01] esxcli software vib install -v /vmfs/volumes/sfo-m01-esx01-vmfs01/NVIDIA-A40-Driver_525.105.14-1OEM.800.1.0.20613240.zip

Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: NVIDIA_bootbank_NVD-AIE_ESXi_8.0.0_Driver_525.105.14-1OEM.800.1.0.20612430
   VIBs Removed:
   VIBs Skipped:

Install the GPU management daemon

[root@sfo-m01-esx01.ash.local:/vmfs/volumes/sfo-m01-esx01-vmfs01] esxcli software vib install -v /vmfs/volumes/sfo-m01-esx01-vmfs01/NVIDIA-A40-gpu-mgmt-daemon_525.105.14-1OEM.800.1.0.21514245.zip

Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: NVIDIA_bootbank_nvdgpuswitch_daemon_525.105.14-1OEM.700.1.0.15834007
VIBs Removed:
VIBs Skipped:
[root@sfo-m01-esx01.ash.local:/vmfs/volumes/sfo-m01-esx01-vmfs01] reboot

Verify passthrough status