Instructions to use JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental")
model = AutoModelForCausalLM.from_pretrained("JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental

SGLang

How to use JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental with Docker Model Runner:
```
docker model run hf.co/JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental
```

HybridAgent-8B-Agentic — EXL3 5.0 bpw, served with a forked vLLM (EXPERIMENTAL)

An unofficial community EXL3 quantization of togethercomputer/HybridAgent-8B-Agentic, a latent-reasoning LLM agent built on Qwen3-8B-Base. The text backbone is quantized to EXL3 at ~5.0 bits per weight (H5) and the HybridLink latent-reasoning MLP is kept in BF16 (adapter.pt), so the whole agent fits in one 24 GB GPU (≈5.98 GB of weights) and is served with a forked vLLM 0.30.0 (continuous batching, PagedAttention). The model thinks in continuous latent space and acts through text, calling a Python interpreter and a web-search tool across multiple turns over one shared KV cache; every latent step is readable as pseudo-text.

It is served with vLLM 0.30.0 through the OrcaSAQ2 EXL3 plugin and our hybridagent backend / processor / orchestrator in serving/. The fork adds zero-token latent steps to the V1 runner so the latent loop runs under PagedAttention. On a single RTX 3090 it reproduces the reference engine's answers (output parity PASS), passes a long-context needle test, serves ~3× the KV concurrency of the BF16 build, and keeps ≥100 MiB of VRAM really free.

License: the model is MIT (inherited from togethercomputer/HybridAgent-8B-Agentic). This is not an official release of Together Computer; it is a community quantization.

⚠️ EXPERIMENTAL — this is not a stock vllm serve model. The HybridAgent latent loop needs an in-process engine (the state machine reaches engine-side Requests and the orchestrator appends tokens via direct request mutation) and zero-token latent steps, which stock vLLM's V2 runner does not express. It runs on a fork of vLLM 0.30.0's V1 runner in eager mode (no CUDA graphs). Everything needed to deploy it — the fork, the plugins, the orchestrator and a setup.sh — is in serving/.

Tested hardware: one RTX 3090 (24 GB, Ampere sm86), vLLM 0.30.0 + torch 2.13.0 + exllamav3 1.5.1 + the OrcaSAQ2 plugin + our hybridagent stack. Only this setup has been tested.

Our focus. We use this model as a latent-reasoning agent: the interesting behavior is the hybrid text/latent loop and the tool use, so the checks in this card (output parity, needle, a real agentic tool request, concurrency, VRAM headroom) are written around that, not around generic chat benchmarks.

What is in this repository (and what is not)

File	What it is
`model.safetensors`	the EXL3-packed text backbone (≈5.98 GB, ~5.0 bpw, H5)
`adapter.pt`	the HybridLink latent-reasoning MLP, kept in BF16 (≈67 MB)
`adapter_config.json`	the HybridLink config
`config.json`	the backbone config (`Qwen3ForCausalLM`, 32K context)
`generation_config.json`	generation config
`quantization_config.json`	the EXL3 per-tensor bit allocation
`chat_template.jinja`	the agentic chat template (required for tool use)
`tokenizer.json`, `tokenizer_config.json`	tokenizer (incl. the `<latent>` / `</latent>` anchors)
`serving/`	the full deployment stack: the fork's V1 runner files, the `hybridagent` backend + processor + orchestrator, `serve.py`, and `setup.sh`
`README.md`	this card

Total ≈ 6.06 GB of weights. Not included: the 15.7 GB BF16 reference (the source of the quantization — it is the original togethercomputer checkpoint); the OrcaSAQ2 plugin, exllamav3 and vLLM (install them from their repositories — setup.sh does this).

Results (single RTX 3090, 24 GB)

Check	Result
Output parity vs the reference `HybridGenerationEngine` (same seed, questions, tools)	PASS — e.g. "trailing zeros of 100!" → 24, "capital of Japan" → Tokyo, same tool calls
Long-context needle (25K-token prompt)	PASS
Real opencode tool request (agentic)	soft-FAIL on EXL3 greedy — picks `search` instead of `python`; the answer is still correct (24) and coherent
Concurrent sessions (KV concurrency)	PASS — ≈3.1× the BF16 KV pool
VRAM really free under load	PASS — ≥100 MiB
Latent loop under EXL3	coherent (345-step run, 0 degenerate steps)

Speed (RTX 3090, eager, greedy):

Engine	Text tok/s	Latent-heavy answer	Memory
Official `HybridGenerationEngine` (transformers, BF16)	~15–18	~69 s	~15.7 GB
Fork, BF16	~32–34	~20 s	~15.7 GB
Fork, EXL3 (this repo)	~18–24 (decode 24.2 wall)	~27 s	~5.98 GB, 3.1× KV

EXL3 prefill ≈2290 tok/s; a latent step ≈33 ms. The fork is ~2× faster than the official transformers engine; EXL3 trades ~40% per-request speed for ~3× KV concurrency and ~10 GB less VRAM.

Requirements & install

EXL3 is not a native vLLM format, and this model needs pieces stock vLLM does not ship (the zero-token latent loop). The stack, all in serving/:

Python 3.12 (the exllamav3 wheels are cp312).
vLLM 0.30.0 (pulls torch 2.13.0).
exllamav3 1.5.1 (cu132, torch 2.13.0) — the EXL3 kernels.
OrcaSAQ2-kernel — the EXL3 plugin for vLLM (loaded via vllm.general_plugins).
hybridagent backend (serving/backend/) — Qwen3ForCausalLM + HybridLink, EXL3-capable load (vllm.general_plugins).
hybridagent processor (serving/processor/) — the DECODE/LATENT/FEED state machine + blocked_token_sets (vllm.logits_processors).
hybridagent orchestrator (serving/orchestrator/) — drives the in-process engine and the tool barrier.
the fork (serving/fork/v1/) — five V1-runner files (request.py, worker/gpu_model_runner.py, core/kv_cache_manager.py, core/sched/{output,scheduler}.py) overlaid onto the installed vLLM.

setup.sh does all of the above:

git clone https://huggingface.co/JRamirez-UAB/hybridagent-8b-exl3-vllm-experimental
cd hybridagent-8b-exl3-vllm-experimental/serving
./setup.sh

If your system nvcc does not match, point CUDA_HOME at a CUDA 13.x toolkit (flashinfer compiles some kernels on first start).

Quick start (single RTX 3090, 24 GB)

This is not the vllm serve CLI — the latent loop needs in-process engine access, so the engine runs in this process and is driven by the HybridAgentOrchestrator:

cd hybridagent-8b-exl3-vllm-experimental
export VLLM_USE_V2_MODEL_RUNNER=0          # the fork is V1
export VLLM_ENABLE_V1_MULTIPROCESSING=0    # in-process core
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export VLLM_USE_FLASHINFER_SAMPLER=0
HYBRIDAGENT_MODEL_DIR=$(pwd) venv/bin/python serving/serve.py

serve.py reads one prompt per line; a blank line runs all pending prompts in one batched loop and prints their final texts; EOF (or Ctrl-C) exits. The validated budget is --max-model-len 32768, kv_cache_memory_bytes=15e9 (≈101k tokens ≈ 3 concurrent full-32k sessions), gpu_memory_utilization=0.93, enforce_eager=True, enable_prompt_embeds=True. To serve the BF16 reference instead, point HYBRIDAGENT_MODEL_DIR at the BF16 dir and set HYBRIDAGENT_KV_BYTES=5000000000.

The agentic system prompt + chat template are mandatory for tool triggering; without them the model answers plainly and never emits a tool call (they are baked into the orchestrator).

How we tested it

Output parity was run against the reference HybridGenerationEngine (togethercomputer's transformers engine) with the same seed, questions and tools: the fork reproduces the reference's answers (24 / 119 / Tokyo, same tool calls).
The battery (serving/ + the repo's battery/) covers a long-context needle, a real opencode tool request (its system prompt + tools, replayed against the engine), concurrent sessions, and VRAM headroom. The EXL3 tool request is a soft-fail under greedy: it picks search over python for the same question, but the final answer is correct.
The latent loop was exercised over a 345-step run under EXL3 with 0 degenerate steps (the quantization did not destabilize the latent recurrence).

Important notes

VLLM_USE_V2_MODEL_RUNNER=0 is required. The fork modifies the V1 runner (v1/worker/gpu_model_runner.py); vLLM 0.30 defaults to V2, where the latent path would be silently inactive.
async_scheduling=False is forced by the orchestrator: the async scheduler keeps num_new_tokens > 0 every step, which blocks the fork's zero-token latent gate.
enable_prompt_embeds=True feeds the R(h) latent embeddings through the mixed-batch prompt-embeds path; enforce_eager=True because the dynamic latent loop breaks CUDA graphs.
Memory. Size the KV with kv_cache_memory_bytes (deterministic against the "≥100 MiB really free" criterion). EXL3 weights ≈5.98 GB + KV 15e9 B ≈14.2 GB + runtime ≈2 GB ≈22.2 GB of 24 GB → gpu_memory_utilization 0.93.
EXL3 tool-choice noise. Under greedy decoding the 5.0 bpw quant can flip the discrete tool-choice token (search vs python); the answer is preserved. If exact tool parity matters, serve the BF16 reference.
HF page warning. The "Model size" HF shows counts the packed int16 storage elements of the EXL3 safetensors (≈4 weights per int16), not parameters — this is the 8B model at ≈5.0 bits per weight.

Quantization recipe

Step	What we did
Backbone	EXL3 at ~5.0 bpw (H5) via the exllamav3 SC second-order pipeline (per-tensor bit allocation)
HybridLink	not quantized — the latent MLP is kept in BF16 (`adapter.pt`)
Tokenizer	the EXL3 `convert.py` did not retain the tokenizer files; they are back-filled from the BF16 dir (the `<latent>` / `</latent>` anchors are preserved)
Hardware	the SC pipeline ran on a 251 GB-RAM node (the 24 GB GPU host's RAM does not fit it); ≈67 min end to end

Source: togethercomputer/HybridAgent-8B-Agentic. Tokenizer, chat template and generation config unchanged.

Integrity and validation

Artifact integrity

vLLM loads every tensor in model.safetensors on start-up (a missing tensor or wrong shape stops it); checked from a fresh download — the model reaches the ready state and serves.
The measurements above used the weights at this repo's first commit; later commits change only documentation or serving/, never the .safetensors.

Quantization

The per-tensor EXL3 bit allocation (~5.0 bpw) is recorded in quantization_config.json.
The HybridLink is deliberately not quantized (BF16 in adapter.pt); only the text backbone is EXL3.
Fidelity evidence is output parity against the BF16 reference engine (PASS) plus the battery; a distribution-level KLD vs BF16 was not measured for this checkpoint.

Runtime

Output parity, long-context needle, agentic tool request, concurrency, VRAM headroom and the 345-step latent run are in Results. Engine errors during the test runs: 0.

Scope: results apply to the stack (vLLM 0.30.0 fork + exllamav3 1.5.1 + OrcaSAQ2 + hybridagent), hardware (one RTX 3090, 24 GB) and settings stated above; other setups are not tested.

Limitations

Needs the fork. It will not load with stock vLLM (vllm serve) or stock transformers; the zero-token latent loop is specific to the V1 runner fork in serving/fork/.
Eager mode. No CUDA graphs (the dynamic latent loop breaks them); decode is slower than a CUDA-graphed 8B would be.
EXL3 tool-choice noise under greedy (see Important notes); serve the BF16 reference if exact tool parity matters.
8B. This is the 8B agent; a 14B exists (togethercomputer/HybridAgent-14B-Agentic) but is not quantized here.
All numbers come from one machine and one software stack.

Feedback welcome

We would be grateful for reports of deployments of this checkpoint, both working and failed ones, especially on GPUs other than the RTX 3090. Please open a discussion in the Community tab with your GPU, driver, vLLM / exllamav3 / plugin versions, the full command line and the result (speed, context, or the error message).

Credits & license

Base model: togethercomputer/HybridAgent-8B-Agentic by Together Computer, built on Qwen/Qwen3-8B-Base (MIT).
Paper: Hybrid Reasoning Agents.
exllamav3 and the EXL3 format by turboderp (MIT); EXL3 is based on QTIP (Tseng et al.).
vLLM integration: the OrcaSAQ2 plugin by Continuum AI Corp; vLLM.
Our hybridagent fork + backend/processor/orchestrator in serving/.

@misc{zou2026hybridagent,
      title={Hybrid Reasoning Agents},
      author={Jiaru Zou and Yunzhe Qi and Sirui Chen and Federico Bianchi and Shang Zhu and Pan Lu and Jingrui He and James Zou},
      year={2026},
      eprint={2610.18861},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.18861},
}

License. The model weights are MIT (inherited from the base model). The OrcaSAQ2 plugin, exllamav3 and vLLM are not redistributed here; install them from their repositories under their own licenses.