Hy4-flash Uncensored GGUF for 4, 2 or 1 RTX 3090

Tencent's 220B Hy4-flash with the full 262144 token context, image input and MTP, on consumer cards, with a refusal vector that removes most refusals. Runs on my Project Maya branch: alesha-pro/project-maya, branch hy4-flash (docs). Stock llama.cpp does not load it.

Setup Decode, tok/s Prefill, tok/s RAM
4x RTX 3090, Q3_K 176 to 251 2947 about 60 GiB
2x RTX 3090, Q3_K 70 to 104 1225 137 GiB
2x RTX 3090, IQ2 95 to 123 1999
1x RTX 3090, Q3_K 37 to 54 1510 137 GiB, 78 GiB with Engram on NVMe
1x RTX 3090, IQ2, Engram on NVMe 54 to 68 1184 to 1199 45 GiB

Files

File GB What
Hy4-flash-Q3_K-00001-of-00002.gguf 86.2 trunk: experts Q3_K/Q4_K, attention Q8_0, MTP layer Q3_K/Q8_0
Hy4-flash-Q3_K-00002-of-00002.gguf 63.4 Engram table
Hy4-flash-IQ2_XXS-00001-of-00002.gguf 51.1 the same trunk with IQ2 experts: faster, further from the base
Hy4-flash-IQ2_XXS-00002-of-00002.gguf 63.4 the same Engram table, so each set is complete (stored once)
mmproj-Hy4-flash-bf16.gguf 1.04 vision
Hy4-flash-refusal-directions.bin 0.004 the uncensored vector (HY4F_ABLATE)
configs/, tokenizer/, routing-profile*.txt, tools/ server configs, tokenizer, expert heat, the scripts behind every file

Get one set of two files: Q3_K (150 GB) or IQ2_XXS (115 GB).

Uncensored

The weights are stock. The vector holds refusal directions for every layer and each of the four hyper-connection streams, and the engine projects them out of the residual stream as it runs. The configs here have it on (HY4F_ABLATE); delete that line for the stock model.

Held-out refusals, judged by the stock model Stock Uncensored
AdvBench 64 + JailbreakBench 82, thinking off 139 32
AdvBench 24, thinking on 22 3
Harmless requests 64 1 0

Cost: on long text the top token matches the stock model in 85.7% of positions (two stock runs agree on 88.3%), median KL 0.014. Most refusals are gone, not all.

Quality of the quants

Against the 4 card Q3_K run on 512 positions: 2 cards 88.3% same top token, 1 card 88.5% (both inside the noise of the test), IQ2 experts 74 to 76%. AngelSlim report their Q3_K within 0.5 points of BF16 on their suites. No task benchmarks of my own.

Run

git clone -b hy4-flash https://github.com/alesha-pro/project-maya && cd project-maya
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 -DSTRATA_BUILD_TESTS=OFF
cmake --build build -j 24 --target maya-hy4f
# edit the paths in the config, then
python -m serve.server --engine maya --config configs/hy4f-2gpu.json --host 127.0.0.1 --port 18000

OpenAI chat API on /v1. For the IQ2_XXS set put its two files into the config instead. For the NVMe mode (1 card) put the Engram file on NVMe and keep HY4F_ENGRAM_IO=direct, as in hy4f-1gpu-iq2-nvme.json.

What I changed (Tencent Hy License, section 4)

  • Hy4-flash-Q3_K-00001-of-00002.gguf: AngelSlim's file with the MTP layer (blk.42, 21 tensors) requantized from BF16 to Q3_K/Q8_0/F32 so it fits next to 256K context on 24 GB cards (tools/requant_mtp_q3k.py). Everything else as it was.
  • Hy4-flash-IQ2_XXS-00001-of-00002.gguf: the Q3_K trunk above with the 123 routed expert tensors of layers 1 to 41 copied byte for byte from AtomicChat's IQ2 file (rev dc1525fe); nothing requantized (tools/merge_experts_gguf.py).
  • Hy4-flash-Q3_K-00002-of-00002.gguf and Hy4-flash-IQ2_XXS-00002-of-00002.gguf: AngelSlim's Engram file, byte for byte.
  • mmproj-Hy4-flash-bf16.gguf: Tencent's vision tower in BF16, converted with AngelSlim's llama.cpp patch.
  • Hy4-flash-refusal-directions.bin, routing-profile*.txt: mine, measured on this model (tools/abliteration/).

Credits

Model: Tencent. Q3_K quantization: AngelSlim. IQ2 experts: AtomicChat. Engine family: Project Maya by mw00 and Peasant Smith, grown out of Strata by Niko1221. GGUF formats and kernels: llama.cpp and ggml.

License

Tencent Hy License, Version 1.0, derived from Apache 2.0 with extra terms for large Model-as-a-Service businesses in section 10.

Downloads last month
2,466
GGUF
Model size
223B params
Architecture
hy_v4_flash
Hardware compatibility
Log In to add your hardware

2-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for alesha-pro/Hy4-flash-Uncensored-GGUF

Quantized
(41)
this model