Quantization on Ascend

• Dense model (e.g., Qwen3-14B-Q4_K_M.gguf):

python3 -m sglang.launch_server \
    --model-path Qwen3-14B-Q4_K_M.gguf \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.7 --tp-size 2

• MoE model (e.g., Qwen3-30B-A3B-Q4_K_M.gguf):

python3 -m sglang.launch_server \
    --model-path Qwen3-30B-A3B-Q4_K_M.gguf \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 2

Implementation Notes:

• GGUF weights are pre-dequantized to FP16/BF16 during model loading on CPU, then transferred to NPU for inference. This trades higher memory usage for faster runtime performance (no per-forward-pass dequantization overhead).
• MoE layers use npu_grouped_matmul and npu_moe_init_routing / npu_moe_finalize_routing for high-performance expert computation.
• TP (tensor parallelism) sharding is supported for both dense and MoE GGUF models.

• Online MXFP8 quantization (BF16/FP16 weights → MXFP8 at load time):

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --quantization mxfp8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1

• Offline MXFP8 quantization (msmodelslim pre-quantized weights, W8A8_MXFP8 scheme; no --quantization flag needed — auto-detected from quant_model_description.json):

python3 -m sglang.launch_server \
    --model-path /path/to/Qwen3-8B-W8A8-MXFP8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1

Implementation Notes:

• Online path: Fp8Config.get_quant_method() dispatches to NPUMXFP8LinearMethod. Weights are quantized once at load via npu_dynamic_mx_quant(weight, dst_type=torch_npu.float8_e4m3fn) and pre-transposed to [in, out]; activations are per-token quantized at inference and matmul runs via npu_quant_matmul(..., group_sizes=[1, 1, 32]) (block_size = 32).
• Offline path: ModelSlimMXFP8Scheme loads float8_e4m3fn weights + float8_e8m0fnu block scales pre-exported by msmodelslim. Transpose is kept as a non-contiguous view (.data assignment) — calling .contiguous() would physically reorder the pre-quantized layout and break the block-scale mapping.
• MoE MXFP8 (FusedMoE) for LLMs is documented in MXFP8 for LLM MoE models below.

• Online MXFP8 quantization (BF16/FP16 expert weights → MXFP8 at load time):

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-30B-A3B \
    --quantization mxfp8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1

• Offline MXFP8 quantization (msmodelslim pre-quantized weights, W8A8_MXFP8 scheme). No --quantization flag is needed: the quant_model_description.json shipped with the checkpoint selects both the ModelSlim path and the scheme automatically.

python3 -m sglang.launch_server \
    --model-path /path/to/Qwen3-30B-A3B-W8A8-MXFP8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1

Implementation Notes:

• Both paths share the per-gmm kernel NPUMXFP8MoEMethod (hardware_backend/npu/quantization/moe_methods.py), which tells online from offline by weight dtype. Expert weights and their e8m0 block scales are kept as non-contiguous transpose views — calling .contiguous() would tank HBM bandwidth.
• Online path: Fp8Config.get_quant_method() dispatches FusedMoE layers to NPUMXFP8OnlineMoEMethod, which subclasses UnquantizedFusedMoEMethod and overrides only create_moe_runner to swap in the MXFP8 kernels — weight creation, weight post-processing and the forward pass are the unquantized Ascend ones. BF16 expert weights w13/w2 are quantized once at load via npu_dynamic_mx_quant(dst_type=torch.float8_e4m3fn) (a 3D [E, N, K] input is accepted directly).
• Offline path: ModelSlimMXFP8MoEScheme (one instance per weight group) loads float8_e4m3fn expert weights + uint8 (e8m0, exponent + 127) block scales. The scale is reshaped [E, N, K/32] → [E, N, K/64, 2] (contiguous pairing, matching npu_dynamic_mx_quant) then transposed.
• Forward: AscendTPDispatcher runs npu_moe_init_routing_v2(quant_mode=3), which fuses the per-token MX activation quant into routing (e4m3 payload + e8m0 block scale, reshaped to the pair-split layout). AscendRunnerCore then runs gmm1 npu_grouped_matmul_swiglu_quant_v2 (cumulative group_list; fuses gate/up + swiglu + requant, so no separate activation step) → gmm2 npu_grouped_matmul (count group_list). The UE8M0 (float8_e8m0fnu) scale dtypes are passed explicitly; the e4m3 x/weight dtypes are left implicit.
• Router gate: msmodelslim may also quantize mlp.gate (W8A8_MXFP8). The gate is a ReplicatedLinear, so its quantization must be description-driven: for the offline modelslim path the gate is passed the quant config and dequantized correctly; the online path keeps it in BF16. Loading a quantized gate as BF16 without its block scale scrambles routing and produces garbage output.
• Where the activation quant happens depends on the dispatcher. On ascend_tp it is fused into routing as described above. DeepEP has no MXFP8 dispatch dtype, so it keeps dispatching BF16 and gmm1 quantizes the hidden states itself via npu_dynamic_mx_quant before the fused kernel — the two paths reach the same gmm1 input. Only the ascend_tp path has been validated end-to-end on the Ascend 950PR/DT Series.

• Online W4A8 quantization (BF16/FP16 weights → MXFP4 at load time):

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --quantization mxfp_w4a8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1

• Offline W4A8 quantization (msmodelslim pre-quantized weights, W4A8_MXFP scheme; no --quantization flag needed — auto-detected from quant_model_description.json).

Implementation Notes:

• Weights are packed FP4 (float4_e2m1fn_x2, two nibbles per byte) with a UE8M0 per-block shared exponent (block_size = 32); activations are per-token MXFP8. Matmul runs via npu_quant_matmul(..., x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[0, 0, 32]).
• The packed-FP4 dtype passed to the NPU ops (dst_type / x2_dtype / input_dtype) must be resolved from torch_npu.float4_e2m1fn_x2 (an int enum), not the torch.float4_e2m1fn_x2 dtype object, which recent op-plugin builds reject.
• Online and offline share the same kernel path and layout; they differ only in the weight source (RTN at load vs msmodelslim calibration).

MODEL_PATH=/path/to/w4a8-mxfp4-moe-model
python3 -m sglang.launch_server \
    --model-path "$MODEL_PATH" \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --tp-size 1

Implementation Notes:

• ModelSlim supplies packed MXFP4 w13 and w2 expert weights with UE8M0 block scales (block size 32).
• Ascend TP and DeepEP dispatch activations as BF16; this path does not request MXFP8 dispatch. SGLang dynamically quantizes each expert input to MXFP8 immediately before grouped matmul.

• Online W4A4 quantization (BF16/FP16 weights → dual-level MXFP4 at load time):

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --quantization mxfp4 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1

• Offline W4A4 quantization (msmodelslim pre-quantized weights, W4A4_MXFP4 scheme; no --quantization flag needed — auto-detected from quant_model_description.json).

Implementation Notes:

• Online (NPUDualLevelMXFP4LinearMethod) uses dual-level MXFP4: both weights and activations are quantized with a fine FP8 (E4M3) L0 block scale plus a coarser L1 scale via npu_dynamic_dual_level_mx_quant, and the matmul runs via npu_dual_level_quant_matmul (weight in FRACTAL_NZ). Dual-level captures per-block dynamic range far better than a single UE8M0 (power-of-2) scale, which is what made an earlier single-level RTN online path degenerate (greedy decoding could loop without emitting EOS).
• Offline (ModelSlimMXFP4Scheme → NPUSingleLevelMXFP4OfflineLinearMethod) is single-level: msmodelslim’s W4A4_MXFP4 checkpoint ships single-level UE8M0 block scales (block_size = 32), so the matmul runs via npu_quant_matmul(..., x1_dtype=x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[1, 1, 32]). The online and offline paths therefore use different matmul kernels — they no longer share the matmul path.
• As with W4A8, the packed-FP4 dtype passed to the NPU ops (dst_type / x2_dtype) must be resolved from torch_npu.float4_e2m1fn_x2 (an int enum), not the torch.float4_e2m1fn_x2 dtype object, which recent op-plugin builds reject.
• Validated end-to-end on Ascend 950PR/DT Series hardware.

MODEL_PATH=/path/to/w4a4-mxfp4-moe-model
python3 -m sglang.launch_server \
    --model-path "$MODEL_PATH" \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --tp-size 1

Implementation Notes:

• ModelSlim supplies packed MXFP4 w13 and w2 expert weights with UE8M0 block scales (block size 32).
• Ascend TP and DeepEP dispatch activations as BF16; this path does not request MXFP4 dispatch. SGLang dynamically quantizes each expert input to MXFP4 immediately before grouped matmul.

​

Diffusion Model Quantization on Ascend NPU

​

Online MXFP8 Quantization

# Start the diffusion server with online MXFP8 quantization
sglang serve \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp8 \
  --num-gpus 4

# One-shot generation
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp8 \
  --prompt "a beautiful sunset over the mountains" \
  --save-output

​

Offline MXFP8 Quantization (ModelSlim)

msmodelslim quant \
  --model_path /path/to/wan2_2_float_weights \
  --save_path /path/to/wan2_2_mxfp8_weights \
  --device npu \
  --model_type Wan2_2 \
  --quant_type mxfp8 \
  --trust_remote_code True

Note: SGLang does not support quantized embeddings; disable embedding quantization when using msmodelslim.

python python/sglang/multimodal_gen/tools/wan_repack.py \
  --input-path /path/to/wan2_2_mxfp8_weights \
  --output-path /path/to/wan2_2_mxfp8_diffusers

sglang generate \
  --model-path /path/to/wan2_2_mxfp8_diffusers \
  --prompt "a beautiful sunset over the mountains" \
  --save-output