GPU Drivers and SDKs

MLC LLM is a universal deployment solution that allows efficient CPU/GPU code generation without AutoTVM-based performance tuning. This section focuses on generic GPU environment setup and troubleshooting.

CUDA

CUDA is required to compile and run models with CUDA backend.

Installation

If you have a NVIDIA GPU and you want to use models compiled with CUDA backend, you should install CUDA, which can be downloaded from here.

Validate Installation

To verify you have correctly installed CUDA runtime and NVIDIA driver, run nvidia-smi in command line and see if you can get the GPU information.

ROCm

ROCm is required to compile and run models with ROCm backend.

Installation

Right now MLC LLM only supports ROCm 5.6. If you have AMD GPU and you want to use models compiled with ROCm backend, you should install ROCm 5.6 from here.

Validate Installation

To verify you have correctly installed ROCm 5.6, run rocm-smi in command line. If you see the list of AMD devices printed out in a table, it means the ROCm is correctly installed.

Vulkan Driver

Installation

To run pre-trained models (e.g. pulled from MLC-AI’s Hugging Face repository) compiled with Vulkan backend, you are expected to install Vulkan driver on your machine.

Please check this page and find the Vulkan driver according to your GPU vendor.

AMD Radeon and Radeon PRO

For AMD Radeon and Radeon PRO users, please download AMD’s drivers from official website (Linux / Windows). For Linux users, after you installed the amdgpu-install package, you can follow the instructions in its documentation to install the driver. We recommend you installing ROCr OpenCL and PRO Vulkan (proprietary) for best performance, which can be done by running the following command:

amdgpu-install --usecase=graphics,opencl --opencl=rocr --vulkan=pro --no-32

Validate Installation

To verify whether Vulkan installation is successful or not, you are encouraged to install vulkaninfo, below are the instructions to install vulkaninfo on different platforms:

sudo apt-get update
sudo apt-get install vulkan-tools

After installation, you can run vulkaninfo in command line and see if you can get the GPU information.

Note

WSL support for Windows is work-in-progress at the moment. Please do not use WSL on Windows to run Vulkan.

Vulkan SDK

Vulkan SDK is required for compiling models to Vulkan backend. To build TVM Unity compiler from source, you will need to install Vulkan SDK as a dependency, but our pre-built wheels already ships with Vulkan SDK.

Check Vulkan SDK installation guide according to your platform:

Please refer to installation and setup page for next steps to build TVM-Unity from source.

OpenCL SDK

OpenCL SDK is only required when you want to build your own models for OpenCL backend. Please refer to OpenCL’s Github Repository for installation guide of OpenCL-SDK.