LocateAnything

Full reimplementation of https://huggingface.co/nvidia/LocateAnything-3B for compatibility with transformers>=5

Example code:

import argparse
from pathlib import Path

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

repo_id = 'SD-Next/LocateAnything-3B'
parser = argparse.ArgumentParser()
parser.add_argument("--image", required=True, type=Path)
parser.add_argument("--prompt", required=True)
args = parser.parse_args()

processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True, torch_dtype="auto", device_map="auto").eval()

image = Image.open(args.image).convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": args.prompt}
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(images=image, text=text, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
decoded = processor.batch_decode(output[:, inputs["input_ids"].shape[1]:], skip_special_tokens=False)
print(decoded[0])

Example output:

python test-locateanything.py --image samples/cartoon.png --prompt "locate hands and face"

<ref>hands</ref>'
    <box><312><754><357><888></box>
    <box><568><778><623><917></box>
<ref>face</ref>
    <box><438><81><536><255></box>
<|im_end|>
Downloads last month
41
Safetensors
Model size
4B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for SD-Next/LocateAnything-3B

Base model

Qwen/Qwen2.5-3B
Finetuned
(17)
this model

Collection including SD-Next/LocateAnything-3B