Instructions to use unsloth/gemma-3-4b-it-unsloth-bnb-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use unsloth/gemma-3-4b-it-unsloth-bnb-4bit with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="unsloth/gemma-3-4b-it-unsloth-bnb-4bit") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("unsloth/gemma-3-4b-it-unsloth-bnb-4bit") model = AutoModelForMultimodalLM.from_pretrained("unsloth/gemma-3-4b-it-unsloth-bnb-4bit", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use unsloth/gemma-3-4b-it-unsloth-bnb-4bit with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "unsloth/gemma-3-4b-it-unsloth-bnb-4bit" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "unsloth/gemma-3-4b-it-unsloth-bnb-4bit", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/unsloth/gemma-3-4b-it-unsloth-bnb-4bit
- SGLang
How to use unsloth/gemma-3-4b-it-unsloth-bnb-4bit with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "unsloth/gemma-3-4b-it-unsloth-bnb-4bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "unsloth/gemma-3-4b-it-unsloth-bnb-4bit", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "unsloth/gemma-3-4b-it-unsloth-bnb-4bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "unsloth/gemma-3-4b-it-unsloth-bnb-4bit", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Unsloth Desktop
- Docker Model Runner
How to use unsloth/gemma-3-4b-it-unsloth-bnb-4bit with Docker Model Runner:
docker model run hf.co/unsloth/gemma-3-4b-it-unsloth-bnb-4bit
Does not work at all
Output is always empty even when using the official example from here (https://huggingface.co/google/gemma-3-4b-it) when replaced model name with this one.
Yes
Output is always empty even when using the official example from here (https://huggingface.co/google/gemma-3-4b-it) when replaced model name with this one.
Yes
Are you guys still having the issue?
is it good to start finetuning with ? or should i use the main model ?
and also update your colab notebook, code is breaking for dinetuinng
Output is always empty even when using the official example from here (https://huggingface.co/google/gemma-3-4b-it) when replaced model name with this one.
Yes
Are you guys still having the issue?
i'm having the same issue as well.
Dont know why so many people downloaded when it does not work.
Actually seem to work when I use bfloat instead of float:
model = Gemma3ForConditionalGeneration.from_pretrained(
model_id,
cache_dir=cache_dir,
torch_dtype=torch.bfloat16,
).to("cuda").eval()
is it good to start finetuning with ? or should i use the main model ?
and also update your colab notebook, code is breaking for dinetuinng
Output is always empty even when using the official example from here (https://huggingface.co/google/gemma-3-4b-it) when replaced model name with this one.
Yes
Are you guys still having the issue?
i'm having the same issue as well.
Dont know why so many people downloaded when it does not work.
We updated the models btw. If you're still having issues let me know exactly what the error is.
Currently Gemma 3 does not work on GPUs that do not support bf16 and unfortunately this is out of our control. so if your GPU only supports f16, it wont work unless you use Unsloth for finetuning/inference
Sorry I meant Gemma 3 only works on bf16. Not f16
ok, and can i use it on kaggle both the free gpus ? or it will always use 1 gpu ?
I tried with accelerator but it failed
Vllm docker error INFO 09-24 03:32:00 [weight_utils.py:349] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards: 100% 1/1 [00:00<00:00, 14.18it/s]