Serving LLMs on Jetson AGX Thor (Pt. 2): Qwen2-VL & Qwen3-30B with SGLang

This article documents the end-to-end setup of an SGLang inference server on the NVIDIA Jetson AGX Thor (JetPack R38), covering environment installation, dependency compilation, model configuration, server launch, and API testing.

0. What is SGLang?

SGLang is a high-performance inference framework for LLMs and multimodal models, co-developed by Stanford University and collaborators. Its core strengths lie in efficient inference scheduling and structured generation:

The typical workflow is: install dependencies → configure the CUDA toolchain → resolve build issues → prepare models → launch the inference server → test and operate. The following sections walk through every step on the Jetson AGX Thor.

1. Prerequisites

Supported Environment

Directory Conventions

Use /workspace as the base path for all models and artifacts:

2. Environment Setup

2.1 Create a Conda Environment

conda create -n sglang python=3.13 -y
conda activate sglang

2.2 Install SGLang

pip install sglang

3. Configuring the CUDA Toolchain

Jetson AGX Thor ships only with CUDA runtime libraries — the development toolkit (headers and nvcc) is not included by default.

3.1 Installation

sudo apt update
sudo apt install cuda-toolkit

3.2 Locating Header Files

After installation, find the CUDA header locations:

find /usr -name "cuda_runtime_api.h" 2>/dev/null

On Jetson AGX Thor, the path is:

/usr/local/cuda-13.0/targets/sbsa-linux/include/cuda_runtime_api.h

Note: Jetson uses sbsa-linux, not the traditional desktop aarch64-linux.

3.3 Configuring Environment Variables

export CUDA_HOME=/usr/local/cuda-13.0
export C_INCLUDE_PATH=$CUDA_HOME/targets/sbsa-linux/include:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=$CUDA_HOME/targets/sbsa-linux/include:$CPLUS_INCLUDE_PATH
export PATH=$CUDA_HOME/bin:$PATH

To make these permanent, append to ~/.bashrc:

echo 'export CUDA_HOME=/usr/local/cuda-13.0' >> ~/.bashrc
echo 'export PATH=$CUDA_HOME/bin:$PATH' >> ~/.bashrc
echo 'export C_INCLUDE_PATH=$CUDA_HOME/targets/sbsa-linux/include:$C_INCLUDE_PATH' >> ~/.bashrc
echo 'export CPLUS_INCLUDE_PATH=$CUDA_HOME/targets/sbsa-linux/include:$CPLUS_INCLUDE_PATH' >> ~/.bashrc
source ~/.bashrc

4. Dependency Compilation

outlines_core is a Rust-based package that requires the following dependencies for source builds:

4.1 Install Rust

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Choose the default installation (option 1)
source $HOME/.cargo/env
rustc --version

4.2 Install OpenSSL Development Libraries

sudo apt install libssl-dev pkg-config

4.3 Python 3.14 Compatibility

When using Python 3.14, PyO3 (v0.22.6) is not supported. Set the compatibility flag:

export PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1
pip install outlines_core

5. Model Preparation

5.1 Available Models

ls /workspace/models/

Qwen2-VL-7B-Instruct
Qwen3-30B-A3B-Instruct-2507

5.2 Model Format Overview

6. Starting the Inference Server

6.1 Small Model Validation (Qwen2-VL-7B)

python3 -m sglang.launch_server \
  --model-path /workspace/models/Qwen2-VL-7B-Instruct \
  --host 0.0.0.0 \
  --log-level info

Success indicators:

[INFO] Uvicorn running on http://0.0.0.0:30000 (Press CTRL+C to quit)
[INFO] The server is fired up and ready to roll!

6.2 Large Model Launch (Qwen3-30B)

Due to CUDA Graph compatibility issues on the NVIDIA Jetson AGX Thor platform, CUDA Graph must be disabled:

python3 -m sglang.launch_server \
  --model-path /workspace/models/Qwen3-30B-A3B-Instruct-2507 \
  --host 0.0.0.0 \
  --log-level info \
  --cuda-graph-backend-decode disabled \
  --cuda-graph-backend-prefill disabled \
  --mem-fraction-static 0.80

6.3 Key Parameters

7. API Testing

7.1 Plain Text Request

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen2-VL-7B-Instruct",
    "messages": [{"role": "user", "content": "Hello, introduce yourself."}],
    "max_tokens": 100
  }'

7.2 Multimodal Image Request (Qwen2-VL)

Since base64-encoded images can exceed the command-line argument length limit, send the request via file:

# Encode image to base64 and construct the request JSON
IMG_BASE64=$(base64 -w 0 /path/to/your/image.png)

cat > /tmp/vl_request.json << ENDJSON
{
  "model": "Qwen2-VL-7B-Instruct",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,${IMG_BASE64}"}},
      {"type": "text", "text": "Describe this image."}
    ]
  }],
  "max_tokens": 200
}
ENDJSON

# Send the request
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d @/tmp/vl_request.json

8. Server Management

8.1 Stopping the Server

# Option 1: Press Ctrl+C in the launch terminal

# Option 2: Force-kill the process
pkill -f sglang

# Verify cleanup
ps aux | grep sglang

8.2 Running in the Background

nohup python3 -m sglang.launch_server \
  --model-path /workspace/models/Qwen3-30B-A3B-Instruct-2507 \
  --host 0.0.0.0 \
  --log-level info \
  --cuda-graph-backend-decode disabled \
  --cuda-graph-backend-prefill disabled \
  > /tmp/sglang.log 2>&1 &

# Tail the logs
tail -f /tmp/sglang.log

8.3 Checking Server Status

# Check port binding
ss -tlnp | grep 30000

# GPU memory usage
nvidia-smi

# Process status
ps aux | grep sglang

Final Thoughts

After walking through the entire SGLang deployment pipeline, a few reflections to share:

1. SGLang’s onboarding experience is remarkably smooth. A single pip install, a one-liner Python module launch, and an OpenAI-compatible API out of the box.

2. The main Jetson AGX Thor compatibility pain point is CUDA Graph. CUDA Graph, which is enabled by default on desktop GPUs, causes deadlocks on AGX Thor. Disabling it restores normal functionality, but at some performance cost. This is the single most important caveat for edge-platform deployments and deserves official attention in future releases.

3. The outlines_core Rust build chain is a hidden cost. On ARM platforms like Jetson, precompiled wheels for many Python packages are unavailable, forcing source builds — which means Rust, OpenSSL, and other extra dependencies must be prepared ahead of time. For developers less familiar with system administration, this can be the first real hurdle.

4. SGLang differentiates at the scheduling layer, not the operator layer. The model weights stay frozen — all optimization happens at the runtime scheduling level. RadixAttention reuses KV caches across requests with shared prefixes. Continuous batching dynamically inserts new requests to keep the GPU saturated. Prefill-Decode disaggregation lets compute-heavy and memory-bound phases scale independently. The core question SGLang asks is: “When 1,000 requests arrive at once, how do you schedule them for maximum efficiency?” SGLang doesn’t do compile-level hardware optimization itself — it stands on low-level operator libraries like FlashInfer and FlashAttention. Its moat is not how extreme its kernels are, but how intelligent its scheduling strategy is. This also explains why SGLang shines in high-concurrency, multi-tenant online serving — the fiercer the contention, the more its scheduling edge matters.

5. On the AGX Thor, SGLang wasn’t built for this hardware — but the fact that it runs at all is telling. As a single-user edge device, Thor’s typical workload is low-latency single-request inference for embodied intelligence and robotic control — not hundreds or thousands of concurrent requests. That’s exactly the scenario where SGLang’s scheduling capabilities are least needed. But flip the lens: being able to run a datacenter-grade inference framework end-to-end on an edge card is itself a testament to the Jetson platform’s capability. If future edge deployments evolve toward multi-model, multi-request concurrency — say, a single robot simultaneously running navigation, manipulation, and dialogue models — SGLang’s scheduling advantage would naturally surface.

6. I hope this guide helps you. May your Jetson run inference smoothly, may every curl return the answer you’re looking for. Happy deploying.