Instructions to use axteam/Qwen3.5-27B-AWQ-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use axteam/Qwen3.5-27B-AWQ-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="axteam/Qwen3.5-27B-AWQ-4bit")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("axteam/Qwen3.5-27B-AWQ-4bit")
model = AutoModelForMultimodalLM.from_pretrained("axteam/Qwen3.5-27B-AWQ-4bit", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use axteam/Qwen3.5-27B-AWQ-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "axteam/Qwen3.5-27B-AWQ-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/Qwen3.5-27B-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/axteam/Qwen3.5-27B-AWQ-4bit

SGLang

How to use axteam/Qwen3.5-27B-AWQ-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "axteam/Qwen3.5-27B-AWQ-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/Qwen3.5-27B-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "axteam/Qwen3.5-27B-AWQ-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/Qwen3.5-27B-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use axteam/Qwen3.5-27B-AWQ-4bit with Docker Model Runner:
```
docker model run hf.co/axteam/Qwen3.5-27B-AWQ-4bit
```

uzins commited on 13 days ago

Commit

565bbf0

verified ·

1 Parent(s): 371c80f

Card fixes 2026-11-02: full card audit — benchmark notes and MMLU method, BF16 shares, requirements, GGUF/diffusion details

Browse files

Files changed (1) hide show

README.md +6 -5

README.md CHANGED Viewed

@@ -44,11 +44,11 @@ This is **post-training quantization (PTQ)** — no retraining. AWQ uses a small
 ## Benchmarks (measured)
-Every column was measured with the same harness and settings — [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) on vLLM, greedy, completion-style prompts without a chat template (0- or 5-shot, as listed in each row). Percentages are recovery vs. the BF16 original.
 | Benchmark | BF16 (original) | FP8-dynamic | AWQ-8bit | NVFP4A16 | NVFP4 | **AWQ-4bit (this repo)** |
 |---|---|---|---|---|---|---|
-| MMLU (5-shot) | 86.02 | 86.01 (100.0%) | 86.05 (100.0%) | 86.07 (100.1%) | 86.01 (100.0%) | **86.08 (100.1%)** |
 | GSM8K (5-shot, strict) | 37.00 | 38.06 (102.9%) | 34.80 (94.1%) | 81.43 (220.1%) | 71.27 (192.6%) | **82.87 (224.0%)** |
 | ARC-Challenge (0-shot) | 61.18 | 61.69 (100.8%) | 61.60 (100.7%) | 60.84 (99.4%) | 60.58 (99.0%) | **60.84 (99.4%)** |
 | Winogrande (5-shot) | 80.43 | 80.03 (99.5%) | 80.58 (100.2%) | 79.64 (99.0%) | 79.48 (98.8%) | **80.11 (99.6%)** |
@@ -57,7 +57,7 @@ Every column was measured with the same harness and settings — [lm-evaluation-
 | **Average** | **68.57** | **68.83 (100.4%)** | **68.42 (99.8%)** | **75.58 (110.2%)** | **73.71 (107.5%)** | **75.89 (110.7%)** |
 | **Average without GSM8K** | **74.88** | **74.98 (100.1%)** | **75.14 (100.4%)** | **74.41 (99.4%)** | **74.19 (99.1%)** | **74.49 (99.5%)** |
-> **Note on GSM8K:** this task is scored on free-form generation, so the score depends on whether the model answers in exactly the format the scorer expects. For this model it swings widely between builds of the same model (GSM8K 34.8–82.9), and some quantized builds score far above the BF16 original — that reflects answer-format adherence, not quantization quality. The second average leaves it out.
 ## Evaluation (measured)
@@ -159,9 +159,10 @@ Quantized derivative of [`Qwen/Qwen3.5-27B`](https://huggingface.co/Qwen/Qwen3.5
 ### 품질 (실측)
 - **원본 대비 충실도**: PPL 1.3886 (원본 1.3603, +2.1%, 낮을수록 원본에 가까움), 다음 토큰 top-1 일치율 94.5% (원본 자체 기준값 99.3%) — 내부 영어·한국어 혼합 프롬프트 100개, 원본(BF16)의 greedy 답변을 기준 텍스트로 측정
-- **벤치마크 평균**: 75.89 (원본 68.57, 110.7%) — MMLU·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 완성형 프롬프트(과제별 0·5-shot, 표 참고)·greedy, 모든 빌드를 같은 설정으로 측정
-- **GSM8K 제외 평균**: 74.49 (원본 74.88, 99.5%) — GSM8K는 자유 생성 답을 채점기가 기대하는 형식 그대로 썼을 때만 점수가 나서, 이 모델에서는 같은 모델의 빌드끼리도 값이 크게 흔들리고 일부 양자화 빌드가 원본보다 훨씬 높게 나옴 (답 형식 준수 차이이지 양자화 품질 차이가 아님)
 - **한국어 벤치마크**: KMMLU 68.98 (원본 69.79, 98.8%) · HAE-RAE 76.44 (원본 76.99, 99.3%)
 ### 요구사항
 - **실행 엔진**: vLLM 0.17.0 이상(이 구조를 지원하는 첫 버전), 이 빌드는 vLLM 0.32.0에서 실행 확인

 ## Benchmarks (measured)
+Every column was measured with the same harness and settings — [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) on vLLM, greedy, completion-style prompts without a chat template (0- or 5-shot, as listed in each row). MMLU here is lm-eval's generative variant (`mmlu_generative`: the model writes the answer letter and only an exact match on the first line counts), so it is not directly comparable with MMLU scores measured another way (e.g. log-likelihood scoring). Percentages are recovery vs. the BF16 original.
 | Benchmark | BF16 (original) | FP8-dynamic | AWQ-8bit | NVFP4A16 | NVFP4 | **AWQ-4bit (this repo)** |
 |---|---|---|---|---|---|---|
+| MMLU (5-shot, generative) | 86.02 | 86.01 (100.0%) | 86.05 (100.0%) | 86.07 (100.1%) | 86.01 (100.0%) | **86.08 (100.1%)** |
 | GSM8K (5-shot, strict) | 37.00 | 38.06 (102.9%) | 34.80 (94.1%) | 81.43 (220.1%) | 71.27 (192.6%) | **82.87 (224.0%)** |
 | ARC-Challenge (0-shot) | 61.18 | 61.69 (100.8%) | 61.60 (100.7%) | 60.84 (99.4%) | 60.58 (99.0%) | **60.84 (99.4%)** |
 | Winogrande (5-shot) | 80.43 | 80.03 (99.5%) | 80.58 (100.2%) | 79.64 (99.0%) | 79.48 (98.8%) | **80.11 (99.6%)** |
 | **Average** | **68.57** | **68.83 (100.4%)** | **68.42 (99.8%)** | **75.58 (110.2%)** | **73.71 (107.5%)** | **75.89 (110.7%)** |
 | **Average without GSM8K** | **74.88** | **74.98 (100.1%)** | **75.14 (100.4%)** | **74.41 (99.4%)** | **74.19 (99.1%)** | **74.49 (99.5%)** |
+> **Note on GSM8K:** this task is scored on generated text (`gsm8k` strict-match), so the score depends on whether the model answers in exactly the format the scorer expects. For this model it swings widely between builds of the same model (GSM8K 34.8–82.9), and three quantized builds score well above the BF16 original (AWQ-4bit 224.0%, NVFP4A16 220.1%, NVFP4 192.6%). The split is between builds: strict-match gives the BF16 original, FP8-dynamic and AWQ-8bit 34.8–38.1 but NVFP4A16, NVFP4 and AWQ-4bit 71.3–82.9, and the lenient flexible-extract score from the same runs (the last number in the answer, whatever its format) shows the same split (39.7–42.4 vs 74.0–84.7), so the gap is not explained by the strict answer format alone; its cause was not investigated (per-question outputs were not saved). The second average leaves it out.
 ## Evaluation (measured)
 ### 품질 (실측)
 - **원본 대비 충실도**: PPL 1.3886 (원본 1.3603, +2.1%, 낮을수록 원본에 가까움), 다음 토큰 top-1 일치율 94.5% (원본 자체 기준값 99.3%) — 내부 영어·한국어 혼합 프롬프트 100개, 원본(BF16)의 greedy 답변을 기준 텍스트로 측정
+- **벤치마크 평균**: 75.89 (원본 68.57, 110.7%) — MMLU(생성형)·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 완성형 프롬프트(과제별 0·5-shot, 표 참고)·greedy, 모든 빌드를 같은 설정으로 측정
+- **GSM8K 제외 평균**: 74.49 (원본 74.88, 99.5%) — GSM8K는 생성한 답이 채점기가 기대하는 형식 그대로여야 점수가 나는 과제; 이 모델에서는 같은 모델의 빌드끼리 GSM8K 34.8~82.9로 크게 흔들리고 양자화 빌드 3개가 원본보다 크게 높음(AWQ-4bit 224.0%·NVFP4A16 220.1%·NVFP4 192.6%), 엄격 ��점(strict-match)은 원본(BF16)·FP8-dynamic·AWQ-8bit 34.8~38.1, NVFP4A16·NVFP4·AWQ-4bit 71.3~82.9로 갈리고 같은 실행의 느슨한 채점(flexible-extract, 형식과 상관없이 답의 마지막 숫자)도 같게 갈려(39.7~42.4, 74.0~84.7) 답 형식만으로는 설명되지 않음, 원인은 조사하지 않음(문항별 출력 미저장)
 - **한국어 벤치마크**: KMMLU 68.98 (원본 69.79, 98.8%) · HAE-RAE 76.44 (원본 76.99, 99.3%)
+- **MMLU 방식**: lm-eval 생성형 변형(`mmlu_generative`) — 모델이 답 글자를 생성하고 첫 줄이 정확히 일치해야 정답으로 채점하므로, 다른 방식(예: log-likelihood 채점)으로 잰 MMLU 점수와 직접 비교 불가
 ### 요구사항
 - **실행 엔진**: vLLM 0.17.0 이상(이 구조를 지원하는 첫 버전), 이 빌드는 vLLM 0.32.0에서 실행 확인