Deploying LLMs on Jetson AGX Thor (Pt. 1): Qwen3-30B with MLC-LLM

This article documents the end-to-end deployment of Qwen3-30B-A3B-Instruct on the Jetson AGX Thor (JetPack 7.x / L4T r38.2) using MLC-LLM, covering environment setup, weight quantization, runtime configuration, CUDA library compilation, and inference verification.

0. What is MLC-LLM?

MLC-LLM (Machine Learning Compilation for Large Language Models) is an open-source universal deployment solution from the Apache TVM ecosystem. Unlike traditional inference frameworks that rely on pre-built runtime binaries, MLC-LLM compiles models into hardware-native code at deployment time, producing platform-optimized shared libraries (.so / .dylib).

Key advantages in the Jetson edge-computing context:

The typical workflow is: pull the official container → download model weights → quantize & convert → gen configuration → compile the CUDA library → deploy via CLI chat or HTTP server. The following sections walk through every step on the Jetson AGX Thor.

1. Prerequisites

Supported Environment

Directory Conventions

Use /workspace as the shared working directory between the host and the container. All models and converted artifacts reside under this path to avoid messy path mappings:

MLC-LLM Optimization Scope

What MLC-LLM does (graph-level + macro optimizations):

• Compute graph fusion (operator fusion, transpose elimination, dead code elimination)
• Quantization format selection (weight precision reduction, mixed-precision inference)
• Static memory planning (StaticPlanBlockMemory, all buffers determined at compile time)
• CUDA Graph merging (reducing CPU→GPU dispatch round-trips)
• Coarse-grained operator substitution such as Flash Attention
• Tensor parallelism / pipeline parallelism strategies

What MLC-LLM does NOT do (per-operator micro-tuning):

• No per-operator loop tiling factor configuration
• No manual vectorization width specification
• No per-layer thread block size tuning
• No AutoTVM cost model search

In short: MLC-LLM is a compilation system — it turns HuggingFace models into runnable executables with graph-level optimizations along the way. Squeezing every last cycle of kernel performance is outside its scope. If you need that level of control on Jetson, fall back to TVM AutoTVM / AutoScheduler for manual tuning, or bring in a dedicated inference engine like TensorRT alongside the MLC compilation artifact.

2. Environment Setup

2.1 Pull the MLC Official Docker Image

Standard pull command:

docker pull ghcr.io/nvidia-ai-iot/mlc:r38.2.arm64-sbsa-cu130-24.04

2.2 Download the Target Model (ModelScope)

Download directly to the working directory to avoid path issues in the container:

# Create the model directory first
sudo mkdir -p /workspace/models

# Download from ModelScope to the target path
modelscope download --model qwen/Qwen3-30B-A3B-Instruct-2507 \
  --local_dir /workspace/models/Qwen3-30B-A3B-Instruct-2507

Migrating pre-downloaded models

If the model was already downloaded elsewhere, move it to the working directory:

sudo mv /path/to/your/downloaded/model/* \
  /workspace/models/Qwen3-30B-A3B-Instruct-2507/
sudo chmod -R 755 /workspace/models

3. Launch the MLC Container

All MLC commands must run inside the container — the host has no MLC tools installed.

3.1 Launch Command

sudo docker run -it --rm --runtime nvidia \
  -v /workspace:/workspace \
  -p 6677:6677 \
  ghcr.io/nvidia-ai-iot/mlc:r38.2.arm64-sbsa-cu130-24.04

3.2 Key Parameters

3.3 Verification

After the command runs, the terminal prompt changes to root@xxxx:/# — you are inside the container. All subsequent steps run in this terminal.

4. Model Weight Quantization & Conversion

Convert the HuggingFace-format model into MLC’s native weight format with quantization.

4.1 Pre-check

Verify the model configuration file exists to avoid path errors:

ls /workspace/models/Qwen3-30B-A3B-Instruct-2507/config.json

If the file path is printed, the check passes.

4.2 Conversion Command

# Create the output directory first
mkdir -p /workspace/models/mlc/Qwen3-30B-A3B-Instruct-2507-q4bf16_1

# Run weight quantization
mlc_llm convert_weight \
  --quantization q4bf16_1 \
  --model-type qwen3_moe \
  --device cuda \
  --source-format huggingface-safetensor \
  -o /workspace/models/mlc/Qwen3-30B-A3B-Instruct-2507-q4bf16_1 \
  /workspace/models/Qwen3-30B-A3B-Instruct-2507/

4.3 Success Criteria

• No ERROR / Exception level messages; INFO-level logs are normal
• Final output reads Finish exporting all parameters
• The output directory contains params_shard_*.bin weight shards and ndarray-cache.json
• Conversion of a 30B MoE model takes approximately 5–15 minutes

5. Generate Runtime Configuration

Generate the MLC inference configuration with memory and performance optimizations for edge devices.

5.1 Command

mlc_llm gen_config \
  --quantization q4bf16_1 \
  --conv-template qwen2 \
  --context-window-size 32768 \
  --prefill-chunk-size 4096 \
  --max-batch-size 3 \
  --output /workspace/models/mlc/Qwen3-30B-A3B-Instruct-2507-q4bf16_1 \
  /workspace/models/Qwen3-30B-A3B-Instruct-2507/

5.2 Optimization Parameters

5.3 Success Criteria

• Logs show Dumping configuration file to: .../mlc-chat-config.json
• The output directory contains mlc-chat-config.json along with automatically copied tokenizer files

6. Compile the CUDA-Optimized Model Library

Precompile a CUDA-optimized shared library (.so) that dramatically accelerates subsequent startup and inference.

6.1 Command

mlc_llm compile \
  --device cuda \
  --quantization q4bf16_1 \
  --model-type qwen3_moe \
  --opt="cublas_gemm=1;cudagraph=1" \
  -o /workspace/models/mlc/Qwen3-30B-A3B-Instruct-2507-q4bf16_1/Qwen3-30B-A3B-Instruct-2507-q4bf16_1-cuda.so \
  /workspace/models/mlc/Qwen3-30B-A3B-Instruct-2507-q4bf16_1/mlc-chat-config.json

6.2 Optimization Flags

6.3 Success Criteria

• Compilation completes without errors; a .so library file appears in the output directory
• Compilation of a 30B model takes approximately 3–10 minutes

7. Verification

7.1 Interactive Chat

Launch a command-line chat interface to quickly verify the model works:

mlc_llm chat --device cuda \
  --model-lib /workspace/models/mlc/Qwen3-30B-A3B-Instruct-2507-q4bf16_1/Qwen3-30B-A3B-Instruct-2507-q4bf16_1-cuda.so \
  /workspace/models/mlc/Qwen3-30B-A3B-Instruct-2507-q4bf16_1

If you did not precompile the model library, the first launch will trigger JIT compilation automatically, then drop you into the chat interface.

7.2 OpenAI-Compatible API Service

Start an HTTP server that exposes an OpenAI-compatible API for downstream applications:

mlc_llm serve /workspace/models/mlc/Qwen3-30B-A3B-Instruct-2507-q4bf16_1 \
  --port 6677 \
  --host 0.0.0.0 \
  --device cuda \
  --model-lib /workspace/models/mlc/Qwen3-30B-A3B-Instruct-2507-q4bf16_1/Qwen3-30B-A3B-Instruct-2507-q4bf16_1-cuda.so \
  --overrides "max_num_sequence=1;max_total_seq_length=32768;context_window_size=32768;gpu_memory_utilization=0.3"

API test command:

curl http://localhost:6677/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-30B-A3B-Instruct",
    "messages": [{"role": "user", "content": "Hello, please introduce yourself"}],
    "temperature": 0.7,
    "max_tokens": 200
  }'

Final Thoughts

After walking through the entire MLC-LLM deployment pipeline, a few reflections to share:

1. MLC-LLM is built on the TVM compilation stack. At its core, it uses compiler technology to adapt LLMs across diverse hardware backends — CUDA, ROCm, Vulkan, Metal, WebGPU — all from a single model definition. The “define once, compile everywhere” philosophy makes cross-platform deployment genuinely approachable.

2. The barrier to entry is remarkably low. A single CLI command, a few lines of Python, a one-liner REST server — the out-of-the-box toolchain means general developers can get an LLM running on edge hardware without ever touching compiler internals. In most edge scenarios, MLC-LLM is arguably one of the easiest ways to bring an LLM to an edge device.

3. But it is not a framework chasing__peak__performance. Its speed comes from compiler-automated graph-level optimization and scheduling rules, not hand-tuned kernels. This automation already delivers solid results — a 30B MoE model running at usable speeds on Jetson speaks for itself — but compared to solutions that hand-write assembly-level kernels for a single hardware target (TensorRT, custom CUDA kernels), the per-platform ceiling is genuinely a tier lower. This is not a flaw; it’s a trade-off.

4. For the Jetson AGX Thor, this is more than enough. Thor’s hardware is already formidable — 64 GB unified memory, Ampere-class GPU cores. In domains like robotics and intelligent edge computing where you’re not chasing every last frame, an engineer’s time is more precious than peak chip utilization. MLC-LLM’s automated approach is the pragmatic, correct choice here.

5. I hope this guide helps you. May your models compile cleanly on whatever hardware you have, may every inference return the answer you’re looking for, and may your deployment journey be a smooth one. Happy deploying.