Running ROCm inside Proxmox LXC containers

This post originally covered a bare-metal Rocky Linux install. That approach is retired here — running ROCm inside an LXC container on Proxmox turned out to be strictly better, so this is the updated guide.

The short version: the kernel driver lives on the host, ROCm userspace lives in the container. No DKMS anywhere, no AMD packages on the host at all.

This setup is verified on RDNA3 and RDNA4 workstation cards (a Radeon Pro W7800 and a Radeon AI PRO R9700) running LLM inference, but nothing about it is card-specific.

Host prerequisites

Proxmox VE’s stock pve kernel ships an amdgpu driver recent enough for current cards. Verify on the host:

# Driver loaded, device nodes present
lsmod | grep amdgpu
ls -la /dev/kfd /dev/dri/renderD128

If those device nodes exist on the host, you never need amdgpu-dkms. Everything else happens inside the container.

GPU passthrough, two flavors

ROCm needs two device nodes in the container: /dev/kfd (compute) and /dev/dri/renderD* (render).

The modern way (unprivileged container)

Proxmox’s dev entries handle the device node, permissions, and gid mapping in one line. In /etc/pve/lxc/<ctid>.conf:

dev0: /dev/kfd,gid=108
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir

Set gid= to the container’s render group id — check with getent group render inside the container (typically 108 on Ubuntu 22.04, 109 on 24.04). An unprivileged user inside can then open the device after a plain usermod -aG render,video.

The manual way (privileged container)

The older pattern does it by hand:

lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.cgroup2.devices.allow: c 234:0 rwm
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file

Gotcha: that 234:0 is /dev/kfd’s major/minor — and the major number is dynamically allocated. Two otherwise-identical hosts can assign 234 and 511, and it can change with a kernel update. If passthrough mysteriously breaks after a reboot, check ls -la /dev/kfd on the host and fix the allow line. Or just use the dev: syntax, which doesn’t care.

Installing ROCm in the container

Use an Ubuntu LTS container (examples below are for 24.04 “noble”; substitute jammy on 22.04). Pin the repo to an exact ROCm version — you want upgrades to be a decision, not a side effect of apt upgrade:

# Keyring
sudo mkdir -p /etc/apt/keyrings
wget -qO- https://repo.radeon.com/rocm/rocm.gpg.key | \
  gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null

# ROCm repo, pinned to a version
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/7.2.4 noble main" | \
  sudo tee /etc/apt/sources.list.d/rocm.list

# AMDGPU userspace repo (libdrm etc. — still no kernel driver involved)
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/amdgpu/30.30.4/ubuntu noble main" | \
  sudo tee /etc/apt/sources.list.d/amdgpu.list

sudo apt update && sudo apt install rocm-dev rocm-libs

(rocm-dev pulls in the HIP toolchain and runtime; rocm-libs brings the math libraries — rocBLAS, hipBLAS, and friends. There is no single rocm metapackage in the apt repo.)

Then permissions for whatever user runs your workload:

sudo usermod -aG render,video $USER

Verify

rocm-smi --showproductname

GPU[0]  : Card Series:  AMD Radeon Pro W7800 48GB
GPU[0]  : Card Model:   0x7449

rocminfo should list the GPU as an agent alongside the CPU. If rocm-smi works but rocminfo fails with a permissions error, it’s the render group (or the gid mapping, in an unprivileged container).

Upgrades: snapshot first

ROCm point releases occasionally break downstream builds (llama.cpp, PyTorch). If the container’s rootfs is on ZFS, a snapshot before any ROCm bump is free insurance:

pct snapshot <ctid> pre-rocm-724

Roll back with pct rollback if the new version misbehaves, and you’ve lost nothing.

Why this beats bare metal

The host stays boring: stock kernel, no third-party driver, survives Proxmox upgrades.
ROCm versions are per-container — run 7.2.3 and 7.2.4 side by side on different containers without conflict.
Rebuilding a botched ROCm install is pct rollback, not an evening of package archaeology.