Self Hosted AMD Docker Machine Learning Whisper Apr 10, 2026 6 min read

Running GPU-Accelerated Whisper on an AMD GPU (No NVIDIA Required)

How I built a self-hosted OpenAI-compatible transcription server on an RX 6600 using whisper.cpp and ROCm — and the three build failures it took to get there

Most guides for self-hosted Whisper assume you have an NVIDIA GPU. CUDA is everywhere. ROCm is the AMD equivalent, and while it works well, the ecosystem around it is thinner — especially when it comes to pre-built Docker images.

I run an AMD Radeon RX 6600 in my home server. I wanted a local speech-to-text endpoint that:

Runs on the GPU, not CPU
Exposes an OpenAI-compatible API at /v1/audio/transcriptions
Lives in Docker with a Tailscale sidecar so I can hit it from anywhere on my tailnet

Here’s what I tried, what failed, and what finally worked.

Why Not faster-whisper?

The obvious first choice is faster-whisper — a popular Whisper server built on CTranslate2. It has Docker images, an OpenAI-compatible API, and good community support.

The problem: CTranslate2 has no ROCm backend. It supports NVIDIA CUDA and CPU only. There are no ROCm wheels on PyPI. Building it with ROCm support from source is a research project in itself.

The faster-whisper Docker images also don’t publish a latest-rocm tag — just *-cpu and *-cuda. The CUDA image starts fine on an AMD machine and even accepts ROCm environment variables without complaint, but silently falls back to CPU inference. You only know it happened if you read the logs carefully:

WARNING: The NVIDIA Driver was not detected. GPU functionality will not be available.

So faster-whisper is out.

whisper.cpp to the Rescue

whisper.cpp is Georgi Gerganov’s C++ port of OpenAI Whisper. It supports multiple backends — CPU, CUDA, Metal, and importantly: HIP (AMD’s GPU compute platform, built on ROCm).

The official Docker images don’t publish a HIP variant, but the code supports it. That means building from source.

Building whisper.cpp with HIP Support

The ROCm base image I used is rocm/dev-ubuntu-22.04:6.3-complete, which matches the ROCm 6.3 installed on my host.

The cmake flag to enable HIP is GGML_HIP=ON. This is not obvious — older guides reference GGML_HIPBLAS=ON, which still exists in the codebase as a recognized option but does absolutely nothing in ggml 0.9.8+. It’s a silent no-op. If you use it, cmake won’t warn you, the build will succeed, and you’ll end up with a CPU-only binary. This cost me one full build.

The correct flags:

cmake -S . -B build \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS=gfx1032 \      # RX 6600 is gfx1032
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_SHARED_LIBS=OFF \
  -DCMAKE_POSITION_INDEPENDENT_CODE=ON

That last flag — CMAKE_POSITION_INDEPENDENT_CODE=ON — is also not obvious. Without it, you get a linker error at the final link step:

/usr/bin/ld: libggml-hip.a(ggml-cuda.cu.o): relocation R_X86_64_32 against
`.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE

The HIP kernels get compiled into a static archive, but the default Ubuntu build toolchain expects position-independent executables. CMAKE_POSITION_INDEPENDENT_CODE=ON propagates -fPIC through all targets including the HIP ones. Second build failure resolved.

The full Dockerfile:

FROM rocm/dev-ubuntu-22.04:6.3-complete

RUN apt-get update && apt-get install -y --no-install-recommends \
    git cmake make g++ python3 ca-certificates ffmpeg \
    && rm -rf /var/lib/apt/lists/*

RUN git clone --depth=1 https://github.com/ggerganov/whisper.cpp /app

WORKDIR /app

RUN cmake -S . -B build \
      -DGGML_HIP=ON \
      -DAMDGPU_TARGETS=gfx1032 \
      -DCMAKE_BUILD_TYPE=Release \
      -DBUILD_SHARED_LIBS=OFF \
      -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
    && cmake --build build -j$(nproc) --target whisper-server

ENV HSA_OVERRIDE_GFX_VERSION=10.3.0
EXPOSE 8000

ENTRYPOINT ["/app/build/bin/whisper-server", "--host", "0.0.0.0", "--port", "8000"]

ffmpeg is needed for the --convert flag, which allows the server to accept audio formats other than raw WAV.

HSA_OVERRIDE_GFX_VERSION=10.3.0 is baked into the image as a default. This is the override required for the RX 6600 (gfx1032) — without it, ROCm may fail to initialize on this GPU because the GFX version ID doesn’t match a fully supported target in the ROCm table.

Build time on an i5-12400F is about 2 minutes. The majority is compiling HIP kernels — you’ll see dozens of Building HIP object lines for the ggml-hip backend.

Verifying GPU Is Actually Active

When the server starts, check the logs for these lines:

ggml_cuda_init: found 1 ROCm devices (Total VRAM: 8176 MiB):
  Device 0: AMD Radeon RX 6600, gfx1030 (0x1030)
whisper_model_load: ROCm0 total size = 1080.58 MB
whisper_backend_init_gpu: using ROCm0 backend

The model weights are loaded directly into VRAM — ROCm0 total size is the key line. If it’s missing, or if you see the NVIDIA warning instead, you’re on CPU.

Note: whisper.cpp internally reports the GPU as gfx1030 even with HSA_OVERRIDE_GFX_VERSION=10.3.0. This is cosmetic — the actual dispatch goes to the right GPU.

The OpenAI-Compatible Endpoint

whisper.cpp’s server exposes transcription at /inference by default, not /v1/audio/transcriptions. It doesn’t have an OpenAI compatibility mode in the traditional sense.

However, it has an --inference-path flag that remaps the endpoint to any path you want. Set it to /v1/audio/transcriptions and any client that speaks the OpenAI audio API will work without modification:

command:
  - --model
  - /models/ggml-large-v3-q5_0.bin
  - --host
  - "0.0.0.0"
  - --inference-path
  - /v1/audio/transcriptions
  - --convert
  - --port
  - "8000"

Test it:

curl -X POST http://your-server:8000/v1/audio/transcriptions \
  -F "file=@audio.mp3"

The Full Compose Stack

I use the Tailscale sidecar pattern — one container owns the network namespace, the other borrows it. This means the whisper server gets a stable Tailscale IP without any port forwarding, firewall rules, or VPN configuration.

name: faster-whisper
services:
  faster-whisper-tailscale:
    image: tailscale/tailscale:latest
    container_name: faster-whisper-tailscale
    restart: unless-stopped
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    volumes:
      - /dev/net/tun:/dev/net/tun
      - /srv/appdata/faster-whisper/tailscale:/var/lib/tailscale
    environment:
      - TS_AUTHKEY=tskey-auth-...   # reusable key
      - TS_HOSTNAME=faster-whisper
      - TS_STATE_DIR=/var/lib/tailscale
      - TS_USERSPACE=false
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv6.conf.all.forwarding=1

  faster-whisper:
    image: whisper-cpp-rocm:latest
    container_name: faster-whisper
    restart: unless-stopped
    network_mode: service:faster-whisper-tailscale
    depends_on:
      - faster-whisper-tailscale
    devices:
      - /dev/kfd:/dev/kfd
      - /dev/dri:/dev/dri
    group_add:
      - "44"    # video GID — use numeric, not name (name won't resolve inside container)
      - "993"   # render GID
    environment:
      - HSA_OVERRIDE_GFX_VERSION=10.3.0
      - ROCR_VISIBLE_DEVICES=0
    volumes:
      - /srv/appdata/faster-whisper/models:/models
    command:
      - --model
      - /models/ggml-large-v3-q5_0.bin
      - --host
      - "0.0.0.0"
      - --inference-path
      - /v1/audio/transcriptions
      - --convert
      - --port
      - "8000"

Two things worth calling out in the GPU device config:

Use numeric GIDs, not group names. The render and video groups exist on the host (render:993, video:44), but Docker looks up group names inside the container’s /etc/group, which doesn’t have them. Docker will fail to start with a cryptic “Unable to find group render” error. Pass the GIDs as quoted numbers.

/dev/kfd and /dev/dri both need to be passed. /dev/kfd is the ROCm compute interface; /dev/dri gives access to the GPU render nodes. Missing either one and ROCm won’t initialize.

The Model

I’m using ggml-large-v3-q5_0 — the large-v3 model quantized to Q5_0. It’s about 1.1 GB and fits comfortably in the RX 6600’s 8 GB of VRAM alongside the compute buffers (~500 MB for KV cache and compute).

Download from Hugging Face:

wget -O /path/to/models/ggml-large-v3-q5_0.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3-q5_0.bin

The full ggml-large-v3.bin is ~3 GB and also fits, but the Q5_0 quantized version has negligible quality difference for most transcription tasks.

What Didn’t Work (and Why)

GGML_HIPBLAS=ON — accepted by cmake, produces no warnings, builds successfully, generates a CPU-only binary. The flag was renamed to GGML_HIP in ggml sometime in 2024. The old name still parses without error, it just does nothing.

group_add: [video, render] — looks right, fails at runtime. Use group_add: ["44", "993"] with the actual GIDs from your host.

faster-whisper -cuda image on AMD — starts fine, accepts HSA_OVERRIDE_GFX_VERSION, loads the model, serves requests, and silently runs everything on CPU. There’s no error, no warning at the API level. The only tell is the startup log line about NVIDIA drivers.

The Result

From anywhere on my tailnet, I can hit http://faster-whisper:8000/v1/audio/transcriptions with any audio file and get a transcription back, GPU-accelerated on the RX 6600. Latency for a one-minute clip is a few seconds. The endpoint is a drop-in replacement for the OpenAI API for any client that uses the audio transcription endpoint.

The hardware is already paid for. The running cost is a few watts. The API key is optional.

Have an AMD GPU and a different setup that works? I’d be curious what GPU-accelerated inference stacks you’ve gotten running on ROCm — drop a comment.