Image-Text-to-Text
Transformers
Safetensors
llava_llama
text-generation
llava
llama
multimodal
VLM
clip
llama-3.1
conversational
Instructions to use aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning
- SGLang
How to use aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning with Docker Model Runner:
docker model run hf.co/aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning
Improve model card with paper link and updated citation (#2)
Browse files- Improve model card with paper link and updated citation (b889e97546bcee45bc439feb5788c4048f529e69)
Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>
README.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
| 1 |
---
|
| 2 |
-
library_name: transformers
|
| 3 |
-
license: apache-2.0
|
| 4 |
datasets:
|
| 5 |
- liuhaotian/LLaVA-Instruct-150K
|
|
|
|
|
|
|
| 6 |
pipeline_tag: image-text-to-text
|
| 7 |
---
|
| 8 |
|
|
@@ -10,7 +10,7 @@ pipeline_tag: image-text-to-text
|
|
| 10 |
|
| 11 |
```LLaVA-MORE``` enhances the well-known LLaVA architecture by integrating the use of LLaMA 3.1 as the language model. We are publicly releasing the checkpoints for stages one and two for the first model with 8B parameters.
|
| 12 |
|
| 13 |
-
In this model space, you will find the stage two (finetuning) weights of LLaVA-MORE LLaMA 3.1 8B.
|
| 14 |
|
| 15 |
For more information, visit our [LLaVA-MORE](https://github.com/aimagelab/LLaVA-MORE) repository.
|
| 16 |
|
|
@@ -25,10 +25,10 @@ python -u llava/eval/run_llava.py
|
|
| 25 |
If you make use of our work, please cite our repo:
|
| 26 |
|
| 27 |
```bibtex
|
| 28 |
-
@
|
| 29 |
-
title={{LLaVA-MORE:
|
| 30 |
-
author={Cocchi, Federico and Moratelli, Nicholas and Caffagni, Davide and Sarto, Sara and
|
| 31 |
-
|
| 32 |
-
year={
|
| 33 |
}
|
| 34 |
```
|
|
|
|
| 1 |
---
|
|
|
|
|
|
|
| 2 |
datasets:
|
| 3 |
- liuhaotian/LLaVA-Instruct-150K
|
| 4 |
+
library_name: transformers
|
| 5 |
+
license: apache-2.0
|
| 6 |
pipeline_tag: image-text-to-text
|
| 7 |
---
|
| 8 |
|
|
|
|
| 10 |
|
| 11 |
```LLaVA-MORE``` enhances the well-known LLaVA architecture by integrating the use of LLaMA 3.1 as the language model. We are publicly releasing the checkpoints for stages one and two for the first model with 8B parameters.
|
| 12 |
|
| 13 |
+
In this model space, you will find the stage two (finetuning) weights of LLaVA-MORE LLaMA 3.1 8B, as described in [this paper](https://huggingface.co/papers/2503.15621).
|
| 14 |
|
| 15 |
For more information, visit our [LLaVA-MORE](https://github.com/aimagelab/LLaVA-MORE) repository.
|
| 16 |
|
|
|
|
| 25 |
If you make use of our work, please cite our repo:
|
| 26 |
|
| 27 |
```bibtex
|
| 28 |
+
@inproceedings{cocchi2025llavamore,
|
| 29 |
+
title={{LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning}},
|
| 30 |
+
author={Cocchi, Federico and Moratelli, Nicholas and Caffagni, Davide and Sarto, Sara and Baraldi, Lorenzo and Cornia, Marcella and Cucchiara, Rita},
|
| 31 |
+
booktitle={arxiv},
|
| 32 |
+
year={2025}
|
| 33 |
}
|
| 34 |
```
|