Status: async in production since 2026-10-23 13:56Z with production check numbers
Browse files
README.md
CHANGED
|
@@ -28,7 +28,10 @@ documentation; it contains no model weights. The served model is the public
|
|
| 28 |
- **New (2026-10-23):**
|
| 29 |
- The two-card drafter with async rounds measures **3.9–6.5 % faster per decode round** than that production setup on
|
| 30 |
GB10, with time to first token unchanged. Window 5 confirmed it and passed supervised fault drills.
|
| 31 |
-
- **Async rounds have been in production since 2026-10-23** with the one-card drafter.
|
|
|
|
|
|
|
|
|
|
| 32 |
- Four 32K requests at once: about **119 tok/s combined decode** (51.7 tok/s overall, prefill included).
|
| 33 |
|
| 34 |
The model stays on the Sparks. An x86 host contributes RAM, two RDMA links and RTX 3090s.
|
|
|
|
| 28 |
- **New (2026-10-23):**
|
| 29 |
- The two-card drafter with async rounds measures **3.9–6.5 % faster per decode round** than that production setup on
|
| 30 |
GB10, with time to first token unchanged. Window 5 confirmed it and passed supervised fault drills.
|
| 31 |
+
- **Async rounds have been in production since 2026-10-23 13:56Z** with the one-card drafter. Production checks
|
| 32 |
+
passed: ready in 363 s; short TTFT 0.37–0.38 s; step p50 38.4–38.5 ms short and 40.5 ms at 32K; greedy outputs
|
| 33 |
+
6 / 6 identical; 2,048 rounds with 0 failed and 0 watchdog fires. The first 32K request after a restart pays a one-time
|
| 34 |
+
loading cost (16.24 s vs ~14.5 s). The two-card worker follows.
|
| 35 |
- Four 32K requests at once: about **119 tok/s combined decode** (51.7 tok/s overall, prefill included).
|
| 36 |
|
| 37 |
The model stays on the Sparks. An x86 host contributes RAM, two RDMA links and RTX 3090s.
|