Utility Models
Collection
5 items โข Updated โข 1
How to use SD-Next/LocateAnything-3B with Transformers:
# Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline("object-detection", model="SD-Next/LocateAnything-3B", trust_remote_code=True) # pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("SD-Next/LocateAnything-3B", trust_remote_code=True, device_map="auto")Full reimplementation of https://huggingface.co/nvidia/LocateAnything-3B for compatibility with transformers>=5
Example code:
import argparse
from pathlib import Path
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor
repo_id = 'SD-Next/LocateAnything-3B'
parser = argparse.ArgumentParser()
parser.add_argument("--image", required=True, type=Path)
parser.add_argument("--prompt", required=True)
args = parser.parse_args()
processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True, torch_dtype="auto", device_map="auto").eval()
image = Image.open(args.image).convert("RGB")
messages = [{"role": "user", "content": [
{"type": "image", "image": image},
{"type": "text", "text": args.prompt}
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(images=image, text=text, return_tensors="pt").to(model.device)
with torch.inference_mode():
output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
decoded = processor.batch_decode(output[:, inputs["input_ids"].shape[1]:], skip_special_tokens=False)
print(decoded[0])
Example output:
python test-locateanything.py --image samples/cartoon.png --prompt "locate hands and face"
<ref>hands</ref>'
<box><312><754><357><888></box>
<box><568><778><623><917></box>
<ref>face</ref>
<box><438><81><536><255></box>
<|im_end|>