Test of TP=3

#1
by Jon-Nielsen - opened

Hi Brandon, thanks for this quant (and K2, K4...)!

I took a chance and tested this quant (with K2's K4 engram grafted on, though it fails before loading that) at TP=3 as intended and have this to report. Hope it helps. Let me know if you need more information.

TP3 boot of this checkpoint on the published k3-fp8-tp3-20261019 image fails deterministically:
48/48 shards load, then all TP workers raise ValueError: TP3 needs both axes divisible by 3 at ds41_trellismx/runtime.py:279 in DS41TrellisMXMoE init (raised in the routed-expert runtime before any Engram code, so it doesn't look graft-specific). Unmodified serving/serve.sh, TP=3, GPUs 0,1,2, claimed CUDA 13.4 / SM120 host.

Let me take a look at that and fix the image.

Hey. Quick update and just to confirm, the updated k3-fp8-tp3-test-20261020 works. Thanks!

Brandon, thank you so much for this. I had apparently pulled the old image with the same reported error above, but thankfully codex saw that it was just a simple python script change:


Successfully booted K3 at TP=3 on 3× RTX PRO 6000 Blackwell 96GB cards after fixing a runtime validation check.

In ds41_trellismx/runtime.py, DS41TrellisMXMoE.__init__ required both moe_intermediate and moe_hidden to be divisible by TP size. However, the documented layout partitions only the intermediate axis: 2304 / 3 = 768. The 5120-wide hidden axis remains replicated.

Changed:

if geometry.moe_intermediate % geometry.tp_size or geometry.moe_hidden % geometry.tp_size:

to:

if geometry.moe_intermediate % geometry.tp_size:

No weights or quantization changes. Startup and short inference tests passed, including batches of eight requests. At 262K context, with 13 GiB KV allocated per GPU, vLLM reports a 7.97M-token pool.

Performance seems roughly consistent with the diffbot bpw3 tp=2 release.

Sign up or log in to comment