When I set out to install NVIDIA drivers alongside CUDA on my machine, I often find myself grappling with frustrating version mismatches that complicate the entire process. One of the most critical tools in this ecosystem is nvidia-smi, which provides vital insights into GPU processes and resource consumption. However, I frequently encounter a common issue: the version of nvidia-smi tends to lag behind the latest NVIDIA drivers. For example, as of April 2025, I might be working with driver version 570 while nvidia-smi is still at version 560. This discrepancy can lead to significant problems, as blindly installing the latest drivers without ensuring compatibility can render nvidia-smi inoperable. Therefore, I’ve learned the importance of carefully navigating these version dependencies to ensure a smooth installation and optimal performance of my GPU resources.
Press enter or click to view image in full size
Restrict Debian 12 to install too fresh packages
On the latest Debian (12 — Bookworm), the standard NVIDIA drivers typically install only up to version 535. However, when I install the cuda-keyring to set up for the CUDA driver later, the official NVIDIA repository gets added to my sources, prompting my system to suggest an update to the latest drivers (currently version 570). While there’s nothing inherently wrong with this, and I can proceed with the installation—including the latest CUDA drivers—my nvidia-smi tool, which is at version 560, will end up being outdated.
To prevent this issue, I need to configure my system to avoid installing certain packages with version numbers higher than 560. After experimenting with the various packages involved in the installation process, I’ve found the necessary steps to ensure a functioning system with the most up-to-date drivers and tools.
Create a policy for all nvidia-related drivers
|
1 |
<span id="b9ba" class="rd qa iu pa b bh re rf m rg rh" data-selectable-paragraph=""><span class="hljs-meta.prompt"># </span><span class="hljs-undefined">nano /etc/apt/preferences.d/nvidia-drivers</span></span> |
You can name this file however you want, then put these rules into it:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
<span id="fe25" class="rd qa iu pa b bh re rf m rg rh" data-selectable-paragraph=""><span class="hljs-attr">Package:</span> <span class="hljs-string">*nvidia*</span> <span class="hljs-attr">Pin:</span> <span class="hljs-string">version</span> <span class="hljs-number">560.35</span><span class="hljs-number">.05</span><span class="hljs-number">-1</span> <span class="hljs-attr">Pin-Priority:</span> <span class="hljs-number">1001</span> <span class="hljs-attr">Package:</span> <span class="hljs-string">cuda-drivers*</span> <span class="hljs-attr">Pin:</span> <span class="hljs-string">version</span> <span class="hljs-number">560.35</span><span class="hljs-number">.05</span><span class="hljs-number">-1</span> <span class="hljs-attr">Pin-Priority:</span> <span class="hljs-number">1001</span> <span class="hljs-attr">Package:</span> <span class="hljs-string">libcuda*</span> <span class="hljs-attr">Pin:</span> <span class="hljs-string">version</span> <span class="hljs-number">560.35</span><span class="hljs-number">.05</span><span class="hljs-number">-1</span> <span class="hljs-attr">Pin-Priority:</span> <span class="hljs-number">1001</span> <span class="hljs-attr">Package:</span> <span class="hljs-string">libxnvctrl*</span> <span class="hljs-attr">Pin:</span> <span class="hljs-string">version</span> <span class="hljs-number">560.35</span><span class="hljs-number">.05</span><span class="hljs-number">-1</span> <span class="hljs-attr">Pin-Priority:</span> <span class="hljs-number">1001</span> <span class="hljs-attr">Package:</span> <span class="hljs-string">libnv*</span> <span class="hljs-attr">Pin:</span> <span class="hljs-string">version</span> <span class="hljs-number">560.35</span><span class="hljs-number">.05</span><span class="hljs-number">-1</span> <span class="hljs-attr">Pin-Priority:</span> <span class="hljs-number">1001</span></span> |
As you can see, I’ve restricted all packages containing the “nvidia” string to a maximum version of 560.35.05–1. I’ve applied the same approach to the other relevant packages, but in this case, I know that these packages start with specific strings, so there’s no need to use wildcards for the first character(s).
|
1 |
<span id="ed5c" class="rd qa iu pa b bh re rf m rg rh" data-selectable-paragraph=""><span class="hljs-comment"># apt update</span></span> |
We need to run apt update to get the policy in effect.
Install Nvidia drivers and tools
|
1 2 |
<span id="f3d0" class="rd qa iu pa b bh re rf m rg rh" data-selectable-paragraph=""><span class="hljs-comment"># apt install nvidia-driver nvidia-smi nvidia-driver</span> </span> |
You’ll notice that the appropriate versions are selected and installed. This process may take some time, and you will need to reboot your machine afterward.
Install Cuda drivers
|
1 2 3 4 5 |
<span id="15a2" class="rd qa iu pa b bh re rf m rg rh" data-selectable-paragraph=""><span class="hljs-meta.prompt"># </span><span class="hljs-undefined">wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb</span> <span class="hljs-meta.prompt"># </span><span class="hljs-undefined">dpkg -i cuda-keyring_1.1-1_all.deb</span> <span class="hljs-meta.prompt"># </span><span class="hljs-undefined">apt-get update</span> <span class="hljs-meta.prompt"># </span><span class="hljs-undefined">sudo apt-get -y install cuda-toolkit-12-8</span> <span class="hljs-meta.prompt"># </span><span class="hljs-undefined">apt-get install -y cuda-drivers</span></span> |
The installation process should go smoothly. Once everything is complete, reboot your machine and enjoy!
+1 extra — Want nvidia and CUDA in Docker?
Luckily, this does not depend much on our hacks above. Just install the Nvidia Container Toolkit and test docker’s access to the GPU.
|
1 2 3 4 5 6 |
<span id="2331" class="rd qa iu pa b bh re rf m rg rh" data-selectable-paragraph=""><span class="hljs-meta.prompt"># </span><span class="hljs-undefined">curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ sed <span class="hljs-string">'s#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g'</span> | \ sudo <span class="hljs-built_in">tee</span> /etc/apt/sources.list.d/nvidia-container-toolkit.list</span> <span class="hljs-meta.prompt"># </span><span class="hljs-undefined">apt-get update</span> <span class="hljs-meta.prompt"># </span><span class="hljs-undefined">apt-get install -y nvidia-container-toolkit</span></span> |
Next, we need to specify the Nvidia runtime for Docker. We need to add the following extra definitions to docker’s daemon.json (note, the file might not exist, so just create it):
|
1 |
<span id="0771" class="rd qa iu pa b bh re rf m rg rh" data-selectable-paragraph=""><span class="hljs-meta.prompt"># </span><span class="hljs-undefined">nano /etc/docker/daemon.json</span></span> |
Then, add the following lines:
|
1 2 3 4 5 6 7 8 9 |
<span id="b4c1" class="rd qa iu pa b bh re rf m rg rh" data-selectable-paragraph=""><span class="hljs-punctuation">{</span> <span class="hljs-attr">"runtimes"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"nvidia"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"path"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"nvidia-container-runtime"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"runtimeArgs"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-punctuation">]</span> <span class="hljs-punctuation">}</span> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"default-runtime"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"nvidia"</span> <span class="hljs-punctuation">}</span></span> |
Then, save and restart docker.
|
1 |
<span id="9ac2" class="rd qa iu pa b bh re rf m rg rh" data-selectable-paragraph=""><span class="hljs-comment"># systemctl restart docker</span></span> |
Now test a simple cuda container and nvidia-smi command inside:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
<span id="b1d2" class="rd qa iu pa b bh re rf m rg rh" data-selectable-paragraph=""># docker run <span class="hljs-comment">--rm --gpus all nvidia/cuda:12.6.3-base-ubuntu22.04 nvidia-smi</span> Tue Apr <span class="hljs-number">8</span> <span class="hljs-number">06</span>:<span class="hljs-number">11</span>:<span class="hljs-number">34</span> <span class="hljs-number">2025</span> <span class="hljs-operator">+</span><span class="hljs-comment">-----------------------------------------------------------------------------------------+</span> <span class="hljs-operator">|</span> NVIDIA<span class="hljs-operator">-</span>SMI <span class="hljs-number">560.35</span><span class="hljs-number">.05</span> Driver Version: <span class="hljs-number">560.35</span><span class="hljs-number">.05</span> CUDA Version: <span class="hljs-number">12.6</span> <span class="hljs-operator">|</span> <span class="hljs-operator">|</span><span class="hljs-comment">-----------------------------------------+------------------------+----------------------+</span> <span class="hljs-operator">|</span> GPU Name Persistence<span class="hljs-operator">-</span>M <span class="hljs-operator">|</span> Bus<span class="hljs-operator">-</span>Id Disp.A <span class="hljs-operator">|</span> Volatile Uncorr. ECC <span class="hljs-operator">|</span> <span class="hljs-operator">|</span> Fan Temp Perf Pwr:Usage<span class="hljs-operator">/</span>Cap <span class="hljs-operator">|</span> Memory<span class="hljs-operator">-</span>Usage <span class="hljs-operator">|</span> GPU<span class="hljs-operator">-</span>Util Compute M. <span class="hljs-operator">|</span> <span class="hljs-operator">|</span> <span class="hljs-operator">|</span> <span class="hljs-operator">|</span> MIG M. <span class="hljs-operator">|</span> <span class="hljs-operator">|</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">+</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">+</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">|</span> <span class="hljs-operator">|</span> <span class="hljs-number">0</span> NVIDIA A100 <span class="hljs-number">80</span>GB PCIe <span class="hljs-keyword">On</span> <span class="hljs-operator">|</span> <span class="hljs-number">00000000</span>:<span class="hljs-number">17</span>:<span class="hljs-number">00.0</span> Off <span class="hljs-operator">|</span> <span class="hljs-number">0</span> <span class="hljs-operator">|</span> <span class="hljs-operator">|</span> N<span class="hljs-operator">/</span>A <span class="hljs-number">31</span>C P0 <span class="hljs-number">44</span>W <span class="hljs-operator">/</span> <span class="hljs-number">300</span>W <span class="hljs-operator">|</span> <span class="hljs-number">14</span>MiB <span class="hljs-operator">/</span> <span class="hljs-number">81920</span>MiB <span class="hljs-operator">|</span> <span class="hljs-number">0</span><span class="hljs-operator">%</span> <span class="hljs-keyword">Default</span> <span class="hljs-operator">|</span> <span class="hljs-operator">|</span> <span class="hljs-operator">|</span> <span class="hljs-operator">|</span> Disabled <span class="hljs-operator">|</span> <span class="hljs-operator">+</span><span class="hljs-comment">-----------------------------------------+------------------------+----------------------+</span> <span class="hljs-operator">+</span><span class="hljs-comment">-----------------------------------------------------------------------------------------+</span> <span class="hljs-operator">|</span> Processes: <span class="hljs-operator">|</span> <span class="hljs-operator">|</span> GPU GI CI PID Type Process name GPU Memory <span class="hljs-operator">|</span> <span class="hljs-operator">|</span> ID ID Usage <span class="hljs-operator">|</span> <span class="hljs-operator">|</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">=</span><span class="hljs-operator">|</span> <span class="hljs-operator">+</span><span class="hljs-comment">-----------------------------------------------------------------------------------------+</span></span> |
If you see the same as above, you are good to go 🙂
Ollama container with WebUI
Here, I just provide you a docker-compose.yml file that does the job for you 🙂
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
<span id="98c5" class="rd qa iu pa b bh re rf m rg rh" data-selectable-paragraph=""><span class="hljs-attr">services:</span> <span class="hljs-attr">ollama:</span> <span class="hljs-attr">container_name:</span> <span class="hljs-string">ollama</span> <span class="hljs-attr">hostname:</span> <span class="hljs-string">ollama</span> <span class="hljs-attr">image:</span> <span class="hljs-string">ollama/ollama:latest</span> <span class="hljs-attr">restart:</span> <span class="hljs-string">unless-stopped</span> <span class="hljs-comment"># ports:</span> <span class="hljs-comment"># - "11434:11434"</span> <span class="hljs-attr">environment:</span> <span class="hljs-bullet">-</span> <span class="hljs-string">NVIDIA_VISIBLE_DEVICES=all</span> <span class="hljs-bullet">-</span> <span class="hljs-string">NVIDIA_DRIVER_CAPABILITIES=compute,utility</span> <span class="hljs-bullet">-</span> <span class="hljs-string">CUDA_VISIBLE_DEVICES=0</span> <span class="hljs-attr">deploy:</span> <span class="hljs-attr">resources:</span> <span class="hljs-attr">reservations:</span> <span class="hljs-attr">devices:</span> <span class="hljs-bullet">-</span> <span class="hljs-attr">driver:</span> <span class="hljs-string">nvidia</span> <span class="hljs-attr">count:</span> <span class="hljs-string">all</span> <span class="hljs-attr">capabilities:</span> [<span class="hljs-string">gpu</span>] <span class="hljs-attr">volumes:</span> <span class="hljs-comment">#use your own path for permanent storage (e.g., models)</span> <span class="hljs-bullet">-</span> <span class="hljs-string">/docker/ollama/data:/root/.ollama</span> <span class="hljs-attr">runtime:</span> <span class="hljs-string">nvidia</span> <span class="hljs-comment"># command: ollama serve</span> <span class="hljs-attr">ollama-webui:</span> <span class="hljs-attr">container_name:</span> <span class="hljs-string">ollama-webui</span> <span class="hljs-attr">hostname:</span> <span class="hljs-string">ollama-webui</span> <span class="hljs-attr">restart:</span> <span class="hljs-string">unless-stopped</span> <span class="hljs-attr">image:</span> <span class="hljs-string">ghcr.io/open-webui/open-webui:latest</span> <span class="hljs-attr">depends_on:</span> <span class="hljs-bullet">-</span> <span class="hljs-string">ollama</span> <span class="hljs-attr">ports:</span> <span class="hljs-bullet">-</span> <span class="hljs-number">3000</span><span class="hljs-string">:8080</span> <span class="hljs-attr">volumes:</span> <span class="hljs-bullet">-</span> <span class="hljs-string">/docker/ollama/open-webui-data:/app/backend/data</span> <span class="hljs-attr">environment:</span> <span class="hljs-bullet">-</span> <span class="hljs-string">OLLAMA_URL=http://ollama:11434</span> </span> |
