Running AI On-Prem: How to Choose Between Bare Metal, Virtualization, and Kubernetes

The real question

Enterprises are bringing AI in-house for practical reasons: to keep proprietary and regulated data on their own infrastructure, to control latency by putting compute next to the data, and, for sustained heavy GPU use, to avoid the running cost of renting cloud accelerators.

Once that decision is made, a second one follows that gets far less attention: where exactly do these workloads run inside the data center. There are three honest answers, and no universal winner. You can run AI directly on bare-metal servers, inside virtual machines, or on a container platform like Kubernetes. In plain terms: a virtual machine emulates a whole server, a container packages just an application with its dependencies, and Kubernetes is the system that schedules many containers across a cluster.

This article is written for the people who make that call: infrastructure architects, virtualization and platform engineers, and the IT leaders evaluating an on-prem AI platform. It puts all three models on the table with their real strengths and weaknesses, grounded in what the vendors actually document, so you can match the choice to your own situation rather than to a trend. Acronyms are tagged like this the first time they appear, for example CUDA [G7], and explained in a short glossary at the end.

What makes AI workloads different

Before comparing the three models, it helps to see why AI stresses infrastructure differently from an ordinary application. It comes down to the GPU, and to the shape of the work.

GPUs are expensive and often sit idle. Utilization is the hidden economics of any AI platform: a GPU doing nothing still costs money and power, so how well a model lets you share and pack GPUs can matter as much as raw speed.

The work itself comes in a few distinct shapes, and each one pushes toward a different model:

• Training and fine-tuning: large jobs that may span many GPUs and several servers. They lean on high-bandwidth GPU-to-GPU links (NVIDIA NVLink [G1] inside a server, InfiniBand [G2] between servers) and libraries like NCCL [G3] to keep the GPUs in step. These jobs run long, want uncontended access to the hardware, and are usually scheduled as a block.
• Inference and LLM serving: many independent copies of a model answering requests. This work needs to scale out and back with demand, stay highly available, and share GPUs to keep cost down.
• Retrieval-augmented generation (RAG) [G4]: less a training problem than a serving-plus-data one. A RAG system combines an embedding model, a vector database [G5], a retrieval step, and an LLM endpoint. It fits a set of cooperating services, and it is a major reason to run AI on-prem at all, because it lets an organization ground a model in proprietary data that never leaves its own systems.
• Experimentation: notebooks and small, bursty jobs from data scientists who need a slice of a GPU now and then, not a whole one.

Sitting across all of these is MLOps [G6]: the lifecycle of preparing data, training, deploying, monitoring, and retraining models. How much of that lifecycle you need to run, and for how many teams, is itself one of the biggest factors in the decision.

The GPU layer

Whichever model you choose, the same GPU questions run underneath it, and the answers shape everything above.

The default stack is NVIDIA on CUDA [G7]. Most on-prem AI is built on NVIDIA data-center GPUs and the CUDA software ecosystem, because that is what the frameworks, drivers, and tools target first. The current range splits roughly into training-class parts (Hopper H100 and H200, Blackwell B200 and GB200) and inference-class parts (Ada Lovelace L40S and L4).

The single most important question is how you share a GPU, and NVIDIA offers three approaches:

• MIG (Multi-Instance GPU) [G8] hard-partitions a physical GPU into as many as seven instances, each with its own dedicated memory, cache, and compute cores, and with fault isolation between them. MIG is available only on Ampere-generation and newer data-center GPUs: A100, A30, H100, H200, B200, GB200. The inference-class L40S and L4 do not support it.
• vGPU [G9], delivered through NVIDIA AI Enterprise and licensed per physical GPU, lets virtual machines share a GPU, either time-sliced or backed by MIG for hardware isolation.
• Time-slicing [G10] simply lets several workloads take turns on a GPU. It packs the most users onto a card but, unlike MIG, provides no memory or fault isolation between them.

That one distinction, whether a GPU can be hard-partitioned or only time-shared, quietly drives much of the decision that follows.

NVIDIA is the default, not the only option. AMD Instinct GPUs (the MI300X and MI350 series) lead on memory capacity, which lets larger models fit on a single card, and run on the open ROCm stack; AMD provides its own compute partitioning and its own Kubernetes operator. That memory headroom makes AMD increasingly attractive for inference and memory-intensive models, where fitting the whole model on one card is the limiting factor. Intel Gaudi accelerators pitch on price-performance and open Ethernet networking. Both are real, but the moat is software, not silicon. The mature CUDA ecosystem is why NVIDIA stays the default, and moving to AMD or Intel means swapping the whole toolchain of driver, operator, and libraries. AMD's own documentation is candid that porting CUDA code is not a drop-in replacement and expects manual work.

The OS reality: AI compute is Linux-first

One fact cuts across all three models and still surprises people: production AI is a Linux-first world.

CUDA, the NVIDIA data-center drivers, MIG, the GPU operators, and essentially the entire MLOps and serving ecosystem are built for Linux. Bare-metal AI runs on Linux. The GPU-accelerated VM guest is Linux. Kubernetes GPU nodes are Linux.

Windows Server has a real but narrower role. Hyper-V can pass a whole GPU to a VM (Discrete Device Assignment, or DDA [G11]) and, from Windows Server 2025, partition one across VMs (GPU Partitioning, GPU-P [G12], using SR-IOV). But Microsoft positions these mainly for virtual desktops and edge inference, not large-scale training. Two limits matter for AI teams: GPU acceleration in Windows containers is restricted to DirectX and DirectML, not CUDA, and MIG, the hardware partitioning used to slice data-center GPUs, is a Linux capability. Even the common way to run CUDA on a Windows machine, WSL2, works by running a Linux environment underneath.

So Windows Server earns its place as a hypervisor host running Linux GPU VMs, and for GPU-accelerated virtual desktops in a mixed estate, but the AI compute itself almost always lands on Linux.

Model 1: bare metal

Running AI directly on the server, with no hypervisor and no container platform, is the simplest model and still the fastest.

What it does well:

• Maximum performance. There is no virtualization layer between the workload and the hardware, so NVLink, NVSwitch, and GPUDirect RDMA [G13] all work at full speed. For large multi-GPU training, where GPUs exchange data constantly, that uncontended interconnect matters.
• A simple stack. Fewer moving parts to install, patch, and reason about.
• Full control of the hardware and drivers.

Where it falls short:

• Poor utilization. A server tends to belong to one workload or team at a time, so expensive GPUs sit idle when that workload is quiet. There is no built-in way to safely share a box across many users.
• No isolation or live migration. If you need to move a running workload off a host for maintenance, or wall tenants off from each other, bare metal gives you little help.
• Manual operations. Drivers, CUDA, scheduling, and multi-tenancy are all yours to manage by hand.

Bare metal is the natural home for a big, sustained training or HPC job that wants every ounce of performance and effectively owns the hardware. It is a poor fit when many people or workloads must share a limited pool of GPUs.

Model 2: virtualization (VMware and Hyper-V)

Virtualization puts AI workloads inside virtual machines, on VMware vSphere or Microsoft Hyper-V. This is where most enterprises already run everything else, which is much of its appeal.

There are two ways to give a VM a GPU, and the difference is important:

• Passthrough (VMware's DirectPath I/O, Hyper-V's Discrete Device Assignment) dedicates a whole physical GPU to one VM. Performance is near-native; in VMware's own testing the virtualization layer added overhead in the low single digits, around 4 percent for the workload it measured, though the exact figure varies by workload and GPU generation. The cost is flexibility: a passed-through GPU cannot be shared, and, critically, the VM can no longer be live-migrated. VMware documents that vMotion, DRS, and snapshots are not available for DirectPath I/O VMs, and Hyper-V likewise does not support live migration of DDA VMs.
• vGPU sharing lets several VMs share one physical GPU. On vSphere, vGPU through NVIDIA AI Enterprise does support live migration: vMotion of vGPU VMs has worked since vSphere 6.7 and is on by default in 8.0 Update 3, at the cost of a brief stun pause that grows with GPU memory size, so VMware recommends 10 GbE networking for it. Hyper-V gained GPU partitioning with live migration in Windows Server 2025.

What virtualization does well:

• Strong isolation and mature operations: snapshots, high availability, and the management tooling your team already knows.
• It reuses existing skills and infrastructure, and suits a mixed estate where AI shares hosts with everything else.
• Multi-GPU training is possible inside one VM. vSphere 8 can present up to eight GPUs to a single VM over NVSwitch.

Where it falls short:

• Licensing. A virtualized GPU stack means a vSphere or VMware Cloud Foundation license plus NVIDIA AI Enterprise, which is licensed per physical GPU. VMware's Private AI Foundation with NVIDIA, which bundles NIM inference services, vector databases, and RAG tooling, is a further paid add-on.
• A small but real performance overhead, and the hard trade-off that the fastest option, passthrough, is also the one that gives up live migration.

Virtualization fits enterprises that want AI to live under the same operational model, isolation, and skills as the rest of the estate, and are willing to pay the licensing for that consistency.

Model 3: containers and Kubernetes (OpenShift)

The third model runs AI in containers orchestrated by Kubernetes, most often through a distribution like Red Hat OpenShift. It is the most flexible and feature-rich model, and also the most demanding to operate.

How GPUs work here: the NVIDIA GPU Operator [G14] automates the whole GPU stack on the cluster, the driver, container toolkit, device plugin, DCGM monitoring, and the MIG manager, while Node Feature Discovery labels which nodes actually have GPUs. Sharing uses the same three options as everywhere else: MIG, time-slicing, and, newer on OpenShift, an on-demand slicer that creates a MIG slice for each pod.

What it does well:

• Density and utilization. Orchestration, autoscaling, and GPU sharing are built in, which is the direct answer to idle-GPU economics.
• It is the natural home for inference and RAG. Serving frameworks like KServe [G15] (autoscaling, scale-to-zero, canary rollouts) and Triton or vLLM for high-throughput LLM serving are Kubernetes-native, and RAG's set of services (embedding, vector database, retrieval, LLM) maps cleanly onto it.
• Full MLOps. Pipelines, distributed training, model serving, and a model registry, for example through Red Hat OpenShift AI, let many teams share one self-service platform.

Where it falls short:

• Day-2 complexity. This is a platform that needs a platform team. GPU scheduling, storage, networking, and upgrades are real, ongoing work, and even a small detail like a serving framework's default timeout can stop a large model from loading in time.
• It usually runs on top of one of the other two models anyway, so it does not remove the underlying infrastructure decision. It adds an orchestration layer over it.

Kubernetes earns its complexity when you have many models or teams, need to serve inference and RAG at scale, and want a shared platform with real MLOps. For a single workload run by a small team, it is usually more than you need.

The decision axes, side by side

Laid next to each other, the three models separate cleanly along a handful of axes:

• Raw performance: bare metal is highest, virtualization is within a few percent, Kubernetes depends on what it runs on.
• Utilization and sharing: Kubernetes is strongest, virtualization is good with vGPU, bare metal is weakest.
• Isolation and multi-tenancy: virtualization and MIG-backed sharing are strong, plain time-slicing is weak, bare metal offers little.
• Operational maturity required: bare metal is simplest to stand up but most manual to run at scale, virtualization reuses existing skills, and Kubernetes demands the most.
• Licensing and cost: bare metal is lightest, virtualization adds hypervisor plus per-GPU AI Enterprise licensing, and Kubernetes cost is mostly the platform team.
• Workload fit: training leans bare metal, mixed enterprise workloads lean virtualization, and inference, RAG, and multi-team MLOps lean Kubernetes.

The same picture in one table:

How to choose

A few honest rules of thumb follow from all this:

• If your main workload is large, sustained training or HPC that wants maximum performance and effectively owns the hardware, start with bare metal.
• If you want AI to run under the same isolation, operations, and skills as the rest of your enterprise estate, and you are already a VMware or Hyper-V shop, virtualization is the path of least resistance, provided you accept the licensing and the passthrough-versus-live-migration trade-off.
• If you need to serve inference or RAG at scale, support many teams, and run a real MLOps lifecycle, Kubernetes is worth its complexity.

A quick recommendation, by situation:

What is your primary workload or situation?
  Large, sustained training or HPC          ->  Bare metal
  Mixed enterprise estate (VMware/Hyper-V)   ->  Virtualization
  Inference, RAG, enterprise AI services     ->  Kubernetes
  Many AI teams sharing one platform         ->  Kubernetes
  A shared pool of GPUs to divide up         ->  Virtualization or Kubernetes
  Small team or first AI project             ->  Start on bare metal or VMs

The important nuance is that these are not exclusive. Kubernetes has to run somewhere, and in practice it runs on bare metal or on VMs. Red Hat supports OpenShift GPU nodes on bare metal, on vSphere passthrough, and on vSphere vGPU, and states plainly that bare metal wins on performance while virtualization wins on reusing existing infrastructure. The platforms are also converging: OpenShift Virtualization can run virtual machines and containers side by side on one cluster, and VMware's Private AI Foundation brings AI tooling into vSphere.

In practice, many organizations do not pick a model on principle so much as grow into one. They often start with virtualized GPUs, because that fits the skills, security, and governance they already run everything else with, and reach for Kubernetes later, when model serving, RAG, or several AI teams create a real need for platform automation. Bare metal tends to arrive last, and only for the workloads whose performance or GPU utilization justifies giving up the convenience of the other two. A recurring pattern worth naming: keeping GPUs busy usually becomes the real constraint before raw performance does, which is why the sharing model, not the benchmark, tends to decide the design.

So the real question is often not which one, but which combination. A common shape is Kubernetes on bare-metal GPU nodes for training and dense inference, alongside vGPU-backed VMs for mixed or legacy workloads.

IT Intel's take

There is no default answer, and anyone selling you one is selling a product, not a decision. The right model follows from two things you already know: the shape of your workloads and the operational maturity of your team.

Two principles help. First, mind the utilization economics. GPUs are the most expensive thing in the rack, so a model that lets you share them well can matter more than a few percent of raw speed. Second, do not buy operational complexity before you need it. Kubernetes is powerful, but a small team running one training job will get there faster on bare metal or a couple of VMs, and can adopt orchestration when the number of models, teams, and services actually grows.

Match the model to the work. Heavy training rewards bare metal, a mixed enterprise estate rewards virtualization, and inference, RAG, and multi-team MLOps reward Kubernetes. Most mature environments end up running more than one, and that is a sign of fitting the tool to the job rather than a failure to standardize.

Glossary

New to some of the acronyms above? Each is explained here in plain language. The number matches the tag beside the term's first use in the article.

• NVLink (G1): NVIDIA's high-speed direct link between GPUs inside a server.
• InfiniBand (G2): A high-speed network used to connect GPU servers for distributed training.
• NCCL (G3): NVIDIA's library that coordinates communication across many GPUs during training.
• RAG (G4): Retrieval-Augmented Generation. Pairing an LLM with a live search over your own data so answers are grounded in it.
• Vector database (G5): A store that holds data as numerical vectors so a RAG system can quickly find the most relevant pieces.
• MLOps (G6): The lifecycle of building, deploying, monitoring, and retraining models. The DevOps of machine learning.
• CUDA (G7): NVIDIA's software platform for running general computation on its GPUs.
• MIG (G8): Multi-Instance GPU. Splitting one physical GPU into up to seven hardware-isolated smaller GPUs.
• vGPU (G9): Software that lets several virtual machines share one physical GPU.
• Time-slicing (G10): Letting several workloads take turns on a GPU, with no memory or fault isolation between them.
• DDA (G11): Discrete Device Assignment. Hyper-V's way of passing a whole physical GPU to one virtual machine.
• GPU-P (G12): GPU Partitioning. Windows Server 2025's way of splitting a GPU across VMs using SR-IOV.
• GPUDirect RDMA (G13): A direct data path between the network and GPU memory that skips the CPU, cutting latency.
• GPU Operator (G14): Software that automates GPU drivers, plugins, and monitoring across a Kubernetes cluster.
• KServe (G15): A Kubernetes framework for serving models, with autoscaling and scale-to-zero.

Reference & sources

This analysis is IT Intel's own, synthesized from vendors' official documentation. No vendor text or diagrams are reproduced, and all product names belong to their owners. Capabilities, support, and licensing change; verify the current details against the official docs before you design around them. Vendor performance claims are the vendors' own.

• NVIDIA: MIG User Guide, GPU Operator, NVIDIA AI Enterprise and vGPU, NVLink and GPUDirect RDMA documentation (docs.nvidia.com).
• VMware and Broadcom: vSphere GPU, vMotion with vGPU, and Private AI Foundation documentation (techdocs.broadcom.com, blogs.vmware.com).
• Red Hat: OpenShift AI, the NVIDIA GPU Operator on OpenShift, and OpenShift Virtualization documentation (docs.redhat.com).
• Microsoft: Hyper-V GPU acceleration, Discrete Device Assignment, and GPU Partitioning documentation (learn.microsoft.com).
• AMD and Intel: ROCm and AMD Instinct documentation (rocm.docs.amd.com); Intel Gaudi documentation (docs.habana.ai).
• Serving and MLOps: KServe, NVIDIA Triton, vLLM, and Kubeflow project documentation.