Containers won the software-packaging argument. Whether they are the right isolation boundary — and which container, on which runtime, with which kernel policy — is a separate question, and the answer changes as you climb from a single node to a shared cluster to a multi-tenant GPU data center. The cloud-native defaults (a root daemon, overlay everything, ignore the network) quietly break on fabric-bound, multi-tenant, accelerator-dense infrastructure.

This page is a candid engineering note on what actually matters at each rung. The theme throughout is a single tension: isolation versus performance. Stronger tenant boundaries cost overhead and privilege; the fastest paths (GPU device access, RDMA, parallel filesystems) want to punch straight through those boundaries. Every decision below is a point on that curve.

The first fork: container, VM, or something in between

The isolation model is the decision everything else hangs on.

VM versus container: each VM ships a full guest OS on top of a hypervisor; containers share one host kernel through a container engine, making them lighter and denser but more weakly isolated.

A virtual machine carries a full guest OS per tenant; a container shares the host kernel. That one difference — a duplicated kernel versus a shared one — drives every tradeoff in the table below.

Boundary How it isolates Strength Cost
Container Shared host kernel; namespaces + cgroups Weaker — a kernel exploit crosses tenants Lightweight, dense, fast to start
Virtual machine Full guest kernel (KVM, ESXi, Hyper-V) Strong — separate kernel per tenant Higher overhead, slower spin-up
Micro-VM / sandbox (Kata, gVisor) Container API, VM-grade boundary Strong-ish, near-container ergonomics Some overhead; narrower device support

For a single team on trusted hardware, containers are the obvious choice. For a customer-facing multi-tenant environment, a shared kernel is the crux risk — one kernel CVE and the boundary is gone. The middle ground (Kata Containers, gVisor) buys VM-grade isolation while keeping the container workflow, and is worth serious consideration wherever untrusted tenants share a node.

The HPC bridge: clusters have historically isolated with schedulers + cgroups + node partitioning rather than hard tenant boundaries, because the tenants are trusted colleagues. A public or semi-public GPU cloud does not get that assumption for free — it needs a stronger boundary between tenants, and that is a deliberate design cost, not an afterthought.

Type-1 vs Type-2 hypervisors

When the boundary is a VM, the hypervisor is what multiplexes the hardware. There are two kinds, and the difference is what sits beneath it:

Type-1 hypervisors (ESXi, KVM, Hyper-V) run directly on the hardware; Type-2 hypervisors run as an application on top of a host operating system.

  • Type-1 (bare-metal) — ESXi, KVM, Hyper-V — runs directly on the hardware. Lower overhead and a smaller attack surface; this is what production clouds and data centers run.
  • Type-2 (hosted) — VirtualBox, VMware Workstation — runs as an application on a host OS. Convenient on a laptop, but the extra layer costs performance and isolation. Rare in the data center.

For a multi-tenant GPU platform the choice is effectively always Type-1: you want the thinnest, most defensible layer between tenant kernels and the silicon.

The Linux primitives underneath — the part you must actually understand

Everything above is built from a small set of kernel features. "It works on my laptop" and "it works on a thousand nodes" are separated almost entirely by how well these are understood and pinned:

  • Namespaces — pid, net, mnt, user, ipc, uts, cgroup. Each gives a container its own view of one kernel subsystem. The user namespace is the one that makes rootless containers possible.
  • cgroups v2 — the unified hierarchy that accounts and caps CPU, memory, I/O, and PIDs. This is the interface the scheduler (Slurm's cgroup plugin, Kubernetes) uses to enforce a job's resource envelope. Its interaction with NUMA and CPU pinning is where a lot of quiet performance loss hides.
  • User namespaces + subuid/subgid — the mapping that lets an unprivileged user be root inside the container and nobody on the host. On a shared cluster, rootless is not optional — no multi-tenant operator should be handing out the real root.
  • seccomp, capabilities, and an LSM (SELinux / AppArmor) — the layers that reduce what a container is allowed to ask the kernel for, independent of who it runs as.

Kernel-level hardening: mandatory access control and admission

Two hardening layers come up constantly in any enterprise or multi-tenant story, and they solve different problems:

  • SELinux (mandatory access control). Beyond normal Unix (discretionary) permissions: every process and file carries a security label, and a system-wide policy dictates what each label may do — not overridable, even by root. The effect is confinement: a compromised container cannot escape to the host or to other tenants, even if it gains root inside. Hardened distributions run it enforcing by default.
  • Admission control (Kubernetes Pod Security Standards; OpenShift's Security Context Constraints are the enterprise version). This gates what a pod may request: run as root? which UID/GID? privileged? host network / host PID / hostPath? which capabilities? which SELinux context? The safe default is restricted — no root, no privileged, no host access — with more permissive policy granted only to specific service accounts, only when justified.

The GPU wrinkle — and it is the whole game in an accelerator cloud. GPU workloads frequently need elevated access: device files, sometimes privileged mode, specific SELinux contexts for the driver and device plugin. So the GPU operator's service account usually needs a relaxed policy. That is the isolation- versus-privilege tension made concrete: the very workloads you most want to isolate between tenants are the ones that demand the most privilege to run. Designing that boundary well — granting the minimum, to the fewest accounts, at the right layer — is the core discipline of a multi-tenant GPU platform.

GPU sharing: passthrough, vGPU, MIG, time-slicing

How you divide a GPU among tenants determines the quality-of-service you can promise. Know all four; they are not interchangeable.

Mode Mechanism Isolation Best for
Passthrough (PCIe/VFIO) Whole physical GPU assigned to one VM/container Full (it's exclusive) Single heavy tenant, max performance
vGPU Software-partitioned, time-sliced virtual GPUs (licensed) Logical, not memory-hard Many light tenants
MIG (Multi-Instance GPU) Hardware partition on A100/H100-class GPUs — up to 7 instances, each with dedicated compute, memory, and cache Hardware-isolated, predictable QoS Guaranteed per-tenant slices
Time-slicing Round-robin oversubscription of one GPU to many pods None (no memory isolation) Dev / bursty / non-guaranteed

MIG is the important one for a guaranteed-QoS GPU cloud — it is how you hand a tenant a slice that behaves like its own card. Time-slicing is the cheap end: useful for development and bursty inference, wrong for anything needing an SLA. (Our published Kubernetes LLM-autoscaling walkthrough on a T4 uses time-slicing precisely because Turing-class hardware predates MIG — a concrete example of the tradeoff.)

The fabric: where isolation meets the fast path

At data-center scale, the interconnect is the performance story, and container networking is where it most often gets accidentally strangled.

  • RDMA (Remote Direct Memory Access) lets the NIC read and write remote memory directly, bypassing the CPU and OS — ultra-low latency, low CPU overhead. RoCE (RDMA over Converged Ethernet) and native InfiniBand are the two flavors.
  • GPUDirect RDMA lets the NIC DMA straight into and out of GPU memory, skipping a host-memory bounce. This is the fast path for multi-node GPU training and inference.
  • Intra- vs inter-node. NVLink/NVSwitch handle GPU-to-GPU inside a node (up to hundreds of GB/s to ~900 GB/s on recent NVSwitch); high-speed RDMA NICs (400–800 Gb/s class, e.g. ConnectX-7/8) handle it between nodes. Know which side of that split every hop is on.

The container implication: to use these, the container must have the right device nodes and userspace libraries injected and, frequently, host networking — which is in direct tension with network isolation. Reconciling "the tenant needs the RDMA fast path" with "the tenant must be network-isolated" is a real design problem, not a checkbox.

Data-center networking and network multi-tenancy

The physical and virtual network is the other half of "up to data centers":

  • Spine-leaf — every leaf (top-of-rack) switch connects to every spine, none to each other, giving predictable equal-hop, non-blocking any-to-any bandwidth. It is the data-center cousin of the fat-tree HPC interconnect, and it matters for the same reason: uniform bisection bandwidth is what keeps collective traffic from collapsing. Modern fabrics route L3 all the way to the leaf for scale.
  • EVPN / VXLAN — VXLAN tunnels virtual L2 segments over the routed L3 fabric; EVPN (a BGP-based control plane) distributes reachability for them. Together they are the standard way to give each tenant a virtual network over one physical fabric — network multi-tenancy to match the compute multi-tenancy above.
  • DPU / SmartNIC offload — an Arm-cored NIC (e.g. BlueField) that offloads virtual switching, encryption, storage virtualization (NVMe-oF), and tenant isolation to the NIC itself, freeing host CPU for tenant workloads. It is the data-center expression of a familiar instinct: move work off the critical path.

The HPC-cluster end of the ladder

The "clusters" end of "clusters to data centers" has its own hard-won conventions that the cloud-native world often ignores:

  • Runtime. A root Docker daemon is a non-starter on a shared login or compute node. Apptainer/Singularity (unprivileged, image-as-a-single-file SIF, built for HPC), Podman (rootless, daemonless), and enroot + pyxis (the Slurm-scale approach) are the realistic choices.
  • MPI-in-a-container — the recurring trap. The container's MPI must match the host's launcher and interconnect (ABI, UCX/libfabric, PMIx, srun vs mpirun). The two strategies are bind-mount the host MPI or build a matching MPI into the image; both have sharp edges, and getting it wrong silently drops you onto the slow transport.
  • Parallel filesystems. Lustre and GPFS are bind-mounted through, never layered under an overlay — the overlay both breaks the parallel semantics and destroys the I/O performance you provisioned the filesystem for.

The scale ladder — what changes at each rung

The move from cluster to data center is, more than anything, a change in the scheduling and isolation model:

Rung Runtime + scheduler Isolation model Networking
Single node Docker/Podman, by hand Namespaces + cgroups Host / bridge
Small cluster Apptainer under Slurm Rootless + scheduler partitioning; trusted tenants Host fabric (IB/Ethernet)
Large GPU cluster enroot + pyxis under Slurm Same, at scale; topology-aware placement RDMA fabric, fat-tree
Multi-tenant data center Kubernetes + device plugins + gang scheduling (Volcano/Kueue), topology manager Hard tenant boundaries: MIG, SELinux/SCC, VXLAN segments, DPU offload Spine-leaf L3, EVPN/VXLAN, GPUDirect

Automation: the lifecycle is the "as-a-service"

None of the above is a deliverable until it is reproducible and self-service. The loop that turns hardware into a platform: programmatically provision servers and network (declarative IaC — Terraform for provisioning, Ansible for configuration, both idempotent and version-controlled) → schedule GPU workloads (Kubernetes / Slurm) → monitor (DCGM and fabric telemetry) → tear down. The same discipline that makes a cross-vendor benchmark trustworthy — pin the base-image digest, the driver, the toolkit; capture the environment; automate the bring-up and the teardown — is what makes a platform trustworthy.

This note reflects SVAC's engineering position on containerization and tenant isolation across the cluster-to-data-center range. It is written to be added to as the hardware and the standards move.

Related: Telecom as a Compute Provider — and a GPU User at Scale — why carriers already own the infrastructure a GPU cloud needs, and why the container-and-isolation layer above is exactly how that hardware gets offered to the public.