ProprietaryLegal commited on
Commit
a3ddd29
·
verified ·
1 Parent(s): 10f97cb

PLI Labs: standalone docs, website links, newest results

Browse files

Docs restructured (status, when to use, deploy, rollback, limits); links to the standalone GitHub repo and proprietarylegal.ai.

LICENSES/Apache-2.0.txt ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ Apache License
3
+ Version 2.0, January 2004
4
+ http://www.apache.org/licenses/
5
+
6
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7
+
8
+ 1. Definitions.
9
+
10
+ "License" shall mean the terms and conditions for use, reproduction,
11
+ and distribution as defined by Sections 1 through 9 of this document.
12
+
13
+ "Licensor" shall mean the copyright owner or entity authorized by
14
+ the copyright owner that is granting the License.
15
+
16
+ "Legal Entity" shall mean the union of the acting entity and all
17
+ other entities that control, are controlled by, or are under common
18
+ control with that entity. For the purposes of this definition,
19
+ "control" means (i) the power, direct or indirect, to cause the
20
+ direction or management of such entity, whether by contract or
21
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
22
+ outstanding shares, or (iii) beneficial ownership of such entity.
23
+
24
+ "You" (or "Your") shall mean an individual or Legal Entity
25
+ exercising permissions granted by this License.
26
+
27
+ "Source" form shall mean the preferred form for making modifications,
28
+ including but not limited to software source code, documentation
29
+ source, and configuration files.
30
+
31
+ "Object" form shall mean any form resulting from mechanical
32
+ transformation or translation of a Source form, including but
33
+ not limited to compiled object code, generated documentation,
34
+ and conversions to other media types.
35
+
36
+ "Work" shall mean the work of authorship, whether in Source or
37
+ Object form, made available under the License, as indicated by a
38
+ copyright notice that is included in or attached to the work
39
+ (an example is provided in the Appendix below).
40
+
41
+ "Derivative Works" shall mean any work, whether in Source or Object
42
+ form, that is based on (or derived from) the Work and for which the
43
+ editorial revisions, annotations, elaborations, or other modifications
44
+ represent, as a whole, an original work of authorship. For the purposes
45
+ of this License, Derivative Works shall not include works that remain
46
+ separable from, or merely link (or bind by name) to the interfaces of,
47
+ the Work and Derivative Works thereof.
48
+
49
+ "Contribution" shall mean any work of authorship, including
50
+ the original version of the Work and any modifications or additions
51
+ to that Work or Derivative Works thereof, that is intentionally
52
+ submitted to Licensor for inclusion in the Work by the copyright owner
53
+ or by an individual or Legal Entity authorized to submit on behalf of
54
+ the copyright owner. For the purposes of this definition, "submitted"
55
+ means any form of electronic, verbal, or written communication sent
56
+ to the Licensor or its representatives, including but not limited to
57
+ communication on electronic mailing lists, source code control systems,
58
+ and issue tracking systems that are managed by, or on behalf of, the
59
+ Licensor for the purpose of discussing and improving the Work, but
60
+ excluding communication that is conspicuously marked or otherwise
61
+ designated in writing by the copyright owner as "Not a Contribution."
62
+
63
+ "Contributor" shall mean Licensor and any individual or Legal Entity
64
+ on behalf of whom a Contribution has been received by Licensor and
65
+ subsequently incorporated within the Work.
66
+
67
+ 2. Grant of Copyright License. Subject to the terms and conditions of
68
+ this License, each Contributor hereby grants to You a perpetual,
69
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70
+ copyright license to reproduce, prepare Derivative Works of,
71
+ publicly display, publicly perform, sublicense, and distribute the
72
+ Work and such Derivative Works in Source or Object form.
73
+
74
+ 3. Grant of Patent License. Subject to the terms and conditions of
75
+ this License, each Contributor hereby grants to You a perpetual,
76
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77
+ (except as stated in this section) patent license to make, have made,
78
+ use, offer to sell, sell, import, and otherwise transfer the Work,
79
+ where such license applies only to those patent claims licensable
80
+ by such Contributor that are necessarily infringed by their
81
+ Contribution(s) alone or by combination of their Contribution(s)
82
+ with the Work to which such Contribution(s) was submitted. If You
83
+ institute patent litigation against any entity (including a
84
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
85
+ or a Contribution incorporated within the Work constitutes direct
86
+ or contributory patent infringement, then any patent licenses
87
+ granted to You under this License for that Work shall terminate
88
+ as of the date such litigation is filed.
89
+
90
+ 4. Redistribution. You may reproduce and distribute copies of the
91
+ Work or Derivative Works thereof in any medium, with or without
92
+ modifications, and in Source or Object form, provided that You
93
+ meet the following conditions:
94
+
95
+ (a) You must give any other recipients of the Work or
96
+ Derivative Works a copy of this License; and
97
+
98
+ (b) You must cause any modified files to carry prominent notices
99
+ stating that You changed the files; and
100
+
101
+ (c) You must retain, in the Source form of any Derivative Works
102
+ that You distribute, all copyright, patent, trademark, and
103
+ attribution notices from the Source form of the Work,
104
+ excluding those notices that do not pertain to any part of
105
+ the Derivative Works; and
106
+
107
+ (d) If the Work includes a "NOTICE" text file as part of its
108
+ distribution, then any Derivative Works that You distribute must
109
+ include a readable copy of the attribution notices contained
110
+ within such NOTICE file, excluding those notices that do not
111
+ pertain to any part of the Derivative Works, in at least one
112
+ of the following places: within a NOTICE text file distributed
113
+ as part of the Derivative Works; within the Source form or
114
+ documentation, if provided along with the Derivative Works; or,
115
+ within a display generated by the Derivative Works, if and
116
+ wherever such third-party notices normally appear. The contents
117
+ of the NOTICE file are for informational purposes only and
118
+ do not modify the License. You may add Your own attribution
119
+ notices within Derivative Works that You distribute, alongside
120
+ or as an addendum to the NOTICE text from the Work, provided
121
+ that such additional attribution notices cannot be construed
122
+ as modifying the License.
123
+
124
+ You may add Your own copyright statement to Your modifications and
125
+ may provide additional or different license terms and conditions
126
+ for use, reproduction, or distribution of Your modifications, or
127
+ for any such Derivative Works as a whole, provided Your use,
128
+ reproduction, and distribution of the Work otherwise complies with
129
+ the conditions stated in this License.
130
+
131
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
132
+ any Contribution intentionally submitted for inclusion in the Work
133
+ by You to the Licensor shall be under the terms and conditions of
134
+ this License, without any additional terms or conditions.
135
+ Notwithstanding the above, nothing herein shall supersede or modify
136
+ the terms of any separate license agreement you may have executed
137
+ with Licensor regarding such Contributions.
138
+
139
+ 6. Trademarks. This License does not grant permission to use the trade
140
+ names, trademarks, service marks, or product names of the Licensor,
141
+ except as required for reasonable and customary use in describing the
142
+ origin of the Work and reproducing the content of the NOTICE file.
143
+
144
+ 7. Disclaimer of Warranty. Unless required by applicable law or
145
+ agreed to in writing, Licensor provides the Work (and each
146
+ Contributor provides its Contributions) on an "AS IS" BASIS,
147
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148
+ implied, including, without limitation, any warranties or conditions
149
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150
+ PARTICULAR PURPOSE. You are solely responsible for determining the
151
+ appropriateness of using or redistributing the Work and assume any
152
+ risks associated with Your exercise of permissions under this License.
153
+
154
+ 8. Limitation of Liability. In no event and under no legal theory,
155
+ whether in tort (including negligence), contract, or otherwise,
156
+ unless required by applicable law (such as deliberate and grossly
157
+ negligent acts) or agreed to in writing, shall any Contributor be
158
+ liable to You for damages, including any direct, indirect, special,
159
+ incidental, or consequential damages of any character arising as a
160
+ result of this License or out of the use or inability to use the
161
+ Work (including but not limited to damages for loss of goodwill,
162
+ work stoppage, computer failure or malfunction, or any and all
163
+ other commercial damages or losses), even if such Contributor
164
+ has been advised of the possibility of such damages.
165
+
166
+ 9. Accepting Warranty or Additional Liability. While redistributing
167
+ the Work or Derivative Works thereof, You may choose to offer,
168
+ and charge a fee for, acceptance of support, warranty, indemnity,
169
+ or other liability obligations and/or rights consistent with this
170
+ License. However, in accepting such obligations, You may act only
171
+ on Your own behalf and on Your sole responsibility, not on behalf
172
+ of any other Contributor, and only if You agree to indemnify,
173
+ defend, and hold each Contributor harmless for any liability
174
+ incurred by, or claims asserted against, such Contributor by reason
175
+ of your accepting any such warranty or additional liability.
176
+
177
+ END OF TERMS AND CONDITIONS
178
+
179
+ APPENDIX: How to apply the Apache License to your work.
180
+
181
+ To apply the Apache License to your work, attach the following
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
183
+ replaced with your own identifying information. (Don't include
184
+ the brackets!) The text should be enclosed in the appropriate
185
+ comment syntax for the file format. We also recommend that a
186
+ file or class name and description of purpose be included on the
187
+ same "printed page" as the copyright notice for easier
188
+ identification within third-party archives.
189
+
190
+ Copyright [yyyy] [name of copyright owner]
191
+
192
+ Licensed under the Apache License, Version 2.0 (the "License");
193
+ you may not use this file except in compliance with the License.
194
+ You may obtain a copy of the License at
195
+
196
+ http://www.apache.org/licenses/LICENSE-2.0
197
+
198
+ Unless required by applicable law or agreed to in writing, software
199
+ distributed under the License is distributed on an "AS IS" BASIS,
200
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201
+ See the License for the specific language governing permissions and
202
+ limitations under the License.
NOTICE ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ PLI Labs — DeepSeek-V4.1-Flash-2xDGX-Spark-x86-offload
2
+ Copyright 2026 Proprietary Legal Intelligence LLC
3
+
4
+ PLI Labs overview and configuration examples are distributed under Apache-2.0.
5
+ This package has no combined code directory and contains no vendored upstream
6
+ software, patch sets, model weights, or third-party licence files. Implementation
7
+ code is obtained from the linked standalone component repositories.
8
+
9
+ External components: DeepSeek model/drafter; HeNryous Spark image and patches;
10
+ sudoingX two-node launcher; turboderp's exllamav3; vLLM. They are not redistributed
11
+ in this package and retain their own licences. Component-specific licence files
12
+ and NOTICEs, including native-kernel attributions, accompany the standalone
13
+ implementations. This NOTICE does not relicense them or model weights.
14
+
15
+ The complete Apache-2.0 licence for the PLI Labs material in this package is
16
+ in LICENSES/Apache-2.0.txt. External dependencies are not redistributed here.
README.md CHANGED
@@ -3,7 +3,8 @@ license: apache-2.0
3
  base_model: deepseek-ai/DeepSeek-V4.1-Flash-1017
4
  library_name: vllm
5
  tags:
6
- - deepseek_v41
 
7
  - dgx-spark
8
  - gb10
9
  - rdma
@@ -12,160 +13,238 @@ tags:
12
  - kv-cache
13
  - offload
14
  - rtx-3090
 
 
 
15
  ---
16
 
17
- # DeepSeek-V4.1-Flash on 2× DGX Spark + x86 host offload
18
 
19
- **Combined entry: hardware, architecture, production configuration and all measured results.** This is source code and
20
- documentation; it contains no model weights. The served model is the public
21
- [DeepSeek-V4.1-Flash-1017 EXL3 2.75 bpw legal-weighted build](https://huggingface.co/ProprietaryLegal/DeepSeek-V4.1-Flash-1017-EXL3-2.75bpw-LegalWeighted-DGX-Spark).
22
 
23
- **What we got:**
24
- - **62 % more KV cache.** It now holds four 300K-token requests at once, which the stock setup can't fit.
25
- - **4.6–6.6 % faster time-to-first-token on long prompts.**
26
- - **Long tasks finish 3.9 % sooner at 150K** than the stock two-Spark recipe.
27
- - With synchronous drafting rounds, the cost is 3–9 % single-stream decode speed.
28
- - **New (2026-10-23):**
29
- - The two-card drafter with async rounds measures **3.9–6.5 % faster per decode round** than that production setup on
30
- GB10, with time to first token unchanged. Window 5 confirmed it and passed supervised fault drills.
31
- - **Async rounds have been in production since 2026-10-23 13:56Z** with the one-card drafter. Production checks
32
- passed: ready in 363 s; short TTFT 0.37–0.38 s; step p50 38.4–38.5 ms short and 40.5 ms at 32K; greedy outputs
33
- 6 / 6 identical; 2,048 rounds with 0 failed and 0 watchdog fires. The first 32K request after a restart pays a one-time
34
- loading cost (16.24 s vs ~14.5 s). The two-card worker follows.
35
- - Four 32K requests at once: about **119 tok/s combined decode** (51.7 tok/s overall, prefill included).
36
 
37
- The model stays on the Sparks. An x86 host contributes RAM, two RDMA links and RTX 3090s.
38
 
39
- ## Separately usable optimizations
40
 
41
- | entry | what it does | status |
 
 
 
 
 
 
 
 
 
 
 
 
42
  |---|---|---|
43
- | [dsv41-remote-engram-rdma](https://huggingface.co/ProprietaryLegal/dsv41-remote-engram-rdma) | Engram tables (2 × ~101.5 GB) served from host RAM over RDMA; exact local fallback | proven, production |
44
- | [dsv41-remote-dspark-drafter-3090](https://huggingface.co/ProprietaryLegal/dsv41-remote-dspark-drafter-3090) | DSpark drafter on an RTX 3090 ("capacity" mode); freed Spark memory → KV | proven, production |
45
- | [dsv41-dspark-drafter-tp2-nvlink](https://huggingface.co/ProprietaryLegal/dsv41-dspark-drafter-tp2-nvlink) | the drafter split over an NVLinked 3090 pair | proven in-engine; with async −3.9 to −6.5 %/round (windows 4 and 5); crash root-caused and fixed; production promotion follows |
46
- | [dsv41-async-hostflag-handoff](https://huggingface.co/ProprietaryLegal/dsv41-async-hostflag-handoff) | GB10 async deadlock finding + CUDA-free proxy hand-off | measured on GB10; hardened, reviewed, fault-drilled; **in production** (one-card worker) since 2026-10-23 |
47
- | [dsv41-native-1017-drafter-sm86](https://huggingface.co/ProprietaryLegal/dsv41-native-1017-drafter-sm86) | the official 1017 drafter weights (MXFP4/FP8) on an NVLinked 3090 pair, custom SM86 kernels | on the real pair 2–12 % faster drafter step than the EXL3 drafter, equal acceptance; **strict fidelity gate still fails (fix in progress)**; not serving |
48
- | [ofl-rdma-transport](https://huggingface.co/ProprietaryLegal/ofl-rdma-transport) | RDMA RC / TCP request-response transport into caller memory | proven, production |
49
- | [dsv41-longctx-bench-tools](https://huggingface.co/ProprietaryLegal/dsv41-longctx-bench-tools) | long-context suite, direct per-step timer, link-load and drill tools | proven |
50
- | [dsv41-spark-offload-findings](https://huggingface.co/ProprietaryLegal/dsv41-spark-offload-findings) | round-trip analysis, deadlocks, interference, exactness methodology, what didn't help | combined findings |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
51
 
52
- GitHub mirror: [ProprietaryLegal/pli-labs › research/dgx-spark-offload](https://github.com/ProprietaryLegal/pli-labs/tree/main/research/dgx-spark-offload).
53
 
54
- ## Hardware
55
 
56
- | | |
57
  |---|---|
58
- | Serving nodes | 2 × NVIDIA DGX Spark (GB10 Grace Blackwell, 121 GiB unified memory each), tensor-parallel 2 over the ConnectX-7 200 Gb/s fabric |
59
- | Offload host | x86, AMD EPYC 7302P (16 cores / 32 threads, one NUMA node), 503 GiB RAM |
60
- | Host ↔ Spark links | one ConnectX-5 Ex (PCIe Gen4 x16) with one 40 Gb/s DAC to each Spark (40GBASE-CR4), MTU 9000, RoCE v2. Measured 39.2 Gb/s per link, ~78.4 Gb/s both. On the Spark side this port shares its PCIe uplink with one TP rail. |
61
- | Drafter GPUs | 2 × RTX 3090 24 GB with an NV4 NVLink bridge, power-capped at 325 W each (one card in the production config, two in the TP2 variant) |
62
-
63
- ## Software
64
-
65
- - Image `ghcr.io/henryous/deepseek-v41-1017-dgx-spark:v5`: vLLM `0.31.1.dev0+g6e517b15.dsv41gb10` with HeNryous patches
66
- 0022–0050, from [HeNryous/deepseek-v41-1017-dgx-spark](https://github.com/HeNryous/deepseek-v41-1017-dgx-spark) @ b1726644.
67
- - Launcher: [sudoingX/dgx-spark-deepseek-v41](https://github.com/sudoingX/dgx-spark-deepseek-v41) @ cc59c36. Our copy also
68
- accepts extra docker arguments for the overlays.
69
- - exllamav3 1.6.0 (EXL3).
70
- - Our forks with the changes, on branch `pli/dgx-spark-offload`:
71
- [ProprietaryLegal/deepseek-v41-1017-dgx-spark](https://github.com/ProprietaryLegal/deepseek-v41-1017-dgx-spark) (the
72
- offload code beside the image's patch set) and
73
- [ProprietaryLegal/dgx-spark-deepseek-v41](https://github.com/ProprietaryLegal/dgx-spark-deepseek-v41) (two launcher
74
- hooks, inert by default).
75
-
76
- ## What runs where (production configuration "D-kv6")
77
-
78
- | component | where | notes |
 
 
 
 
 
 
 
 
 
 
 
 
 
79
  |---|---|---|
80
- | target model, 40 layers, TP2 | both Sparks | unchanged weights, sampling, tokenizer, reasoning parser |
81
- | KV cache | both Sparks | **6.0 GiB/rank = 1,604,926 tokens** (stock: 3.75 GiB = 989,460) |
82
- | Engram tables (layers 1 and 14) | host RAM, 189 GiB | rows over RDMA, one link per Spark; local NVMe reader kept as exact fallback |
83
- | DSpark drafter (3 layers + Markov head) | host RTX 3090 | rank 0 sends tap rows and broadcasts drafts to rank 1; capacity policy: no drafter on the Sparks |
84
- | per-request context | max-model-len 327,680 | the model supports 1,048,576 (1M validation in progress) |
 
85
 
86
- The production settings, sanitized, are in `config/production-D.env.example`.
87
 
88
- ## Results (two DGX Spark, T = 1, thinking at effort "high", 1,500-token budgets, cold cache unless noted, n = 2–3)
 
 
 
89
 
90
- | | A: stock recipe | B: + remote Engram | D-kv6: + remote drafter + KV 6 GiB |
91
- |---|---|---|---|
92
- | KV pool (tokens) | 989,460 | 989,460 | **1,604,926** |
93
- | short prompt, stream tok/s | 55.96 ± 1.65 | — | 51.07 ± 0.78 |
94
- | 32K TTFT | 15.53 ± 0.12 s | 14.70 ± 0.07 s | 14.81 ± 0.54 s |
95
- | 32K stream tok/s | 64.57 ± 0.80 | — | 60.82 ± 2.30 |
96
- | 150K TTFT | 73.62 ± 0.11 s | 69.44 ± 0.04 s | 68.79 ± 0.14 s |
97
- | 150K task time | 100.35 ± 0.55 s | 95.17 ± 1.61 s | 96.46 ± 1.11 s |
98
- | 256K / 300K TTFT, stream tok/s | — | — | 118.7 s, 51.6 / 140.3 s, 52.2 |
99
- | 32K × 2 / × 3 aggregate tok/s | 40.59 / 47.21 | — | 40.81 / 47.36 |
100
- | 4 × 300K at once | does not fit | does not fit | **all 4 complete** |
101
- | MemAvailable per Spark | 5–7 GiB | 7 GiB | 5–8 GiB |
102
-
103
- **Isolated effects:**
104
- - **Remote Engram (B):** −5.4 % (32K) and −5.7 % (150K) TTFT, decode level, byte-exact.
105
- - **Remote drafter (C):** frees ~2.5 GiB of model load per rank and ~3–5 GiB of available memory; +7–9 % per decode round at
106
- one stream, ≤ 2 % under concurrency; acceptance unchanged.
107
- - **Two-card drafter (E):** −0.6 to −2.6 % per round versus one card, in-engine, synchronous.
108
- - **Async hand-off, process proxy:**
109
- - one card: −0.9 to −3.9 % per round;
110
- - two cards: **−3.9 to −6.5 %**;
111
- - time to first token at parity, 0 watchdog fires.
112
-
113
- ### Window 4 (2026-10-23): two-card drafter + async rounds on GB10
114
-
115
- Direct per-step timer, p50 per request, 1,500-token budgets, T = 1; the baseline is the one-card synchronous production
116
- setup at the time.
117
-
118
- | workload | production (one card, sync) | two cards + async process proxy | change |
119
- |---|---|---|---|
120
- | short × 3, step (ms) | 39.6 / 40.1 / 39.4 | 37.1 / 37.4 / 36.9 | −6.5 % |
121
- | 32K × 3, step (ms) | 41.2 / 42.2 / 42.3 | 39.6 / 39.5 / 40.2 | −4.5 % |
122
- | 32K, 3 concurrent, step (ms) | 75.9 / 76.1 / 76.1 | 72.8 / 73.5 / 73.0 | −3.9 % |
123
- | 150K × 2, step (ms) | 41.5 / 42.2 | 39.6 / 39.8 | −5.0 % |
124
- | 32K / 150K TTFT | 14.56–14.59 s / 69.10–69.11 s | 14.49–14.63 s / 68.89–69.09 s | parity |
125
- | drafter cards, serving memory | 1 × ~8.1 GiB | 5,862 + 5,222 MiB | |
126
- | 4 × 32K at once (window 5) | — | 51.7 tok/s overall; ~119 tok/s combined decode; step p50 87–88 ms | |
127
 
128
- Speed in tokens per second also depends on how many drafts each run happens to accept at T = 1 (±5 % between repeats), so
129
- we compare time per round. In the same runs, stream speed was 57.6 → 62.2 tok/s at 32K and 51.0 → 54.4 tok/s at 150K (means
130
- of 3 and 2 runs).
131
 
132
- ## Deploy and roll back
133
 
134
- Each optimization has its own switch and its own fallback (see its entry):
135
 
136
- | to remove | do this | what you get back |
 
 
 
 
 
 
 
 
 
 
 
 
 
137
  |---|---|---|
138
- | remote Engram | unset `VLLM_ENGRAM_REMOTE*` and the overlay env | the image's NVMe reader |
139
- | remote drafter | unset `VLLM_DSPARK_REMOTE*`, remove the overlay mounts, KV back to 3.75 GiB | the image's local drafter |
140
- | two-card drafter | swap the worker unit back | the one-card worker; the engine reconnects in ~5 s |
141
- | async rounds | unset `VLLM_DSPARK_REMOTE_ASYNC` and restart the serve | synchronous rounds |
142
-
143
- The serve script refuses a half-enabled drafter configuration, and refuses a raised KV pool while the drafter is local, since
144
- that combination is untested.
145
-
146
- ## Limits and open items
147
-
148
- - Single-stream decode is 3–9 % slower with synchronous remote rounds on GB10. Async rounds are now in production with
149
- the one-card worker. The two-card worker adds the rest of the 3.9–6.5 % per round once it is promoted.
150
- - In progress, not yet proven on GB10:
151
- - KV page packing (upstream HeNryous 0057/0060; offline allocator figures: 1,604,926 → 2,502,600 tokens at 6 GiB/rank);
152
- - 1M-token context;
153
- - **prefix storage across restarts** (an overlay of upstream HeNryous patches 0056/0058/0077/0081, plus a host-RAM
154
- tier on the x86 host): code written and CPU-tested, under independent review, not yet run on GB10;
155
- - **prefill ordering for mixed long requests** (first-come-first-served with a decode cap and background-priority
156
- protection): the CPU model predicts the first answer of four concurrent 300K requests at about 156 s instead of 620 s.
157
- That is a model result, not a measurement;
158
- - the native drafter's fidelity fix.
159
- - One host is a shared dependency. If it fails, the Sparks keep serving: Engram falls back to NVMe, and decode runs without
160
- speculation.
161
-
162
- ## Credits and licence
163
-
164
- DeepSeek ([DeepSeek-V4.1-Flash-1017](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017), MIT) · HeNryous (Spark
165
- image and patch set; [quant](https://huggingface.co/HeNryous/DeepSeek-V4.1-Flash-1017-EXL3-2.75bpw-DGX-Spark)) · sudoingX
166
- (two-Spark recipe and launcher) · turboderp ([exllamav3](https://github.com/turboderp-org/exllamav3), MIT) ·
167
- [vLLM](https://github.com/vllm-project/vllm) (Apache-2.0).
168
-
169
- Our code is Apache-2.0.
170
-
171
- Published by PLI Labs (ProprietaryLegal) · https://proprietarylegal.ai
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  base_model: deepseek-ai/DeepSeek-V4.1-Flash-1017
4
  library_name: vllm
5
  tags:
6
+ - deepseek
7
+ - deepseek-v4
8
  - dgx-spark
9
  - gb10
10
  - rdma
 
13
  - kv-cache
14
  - offload
15
  - rtx-3090
16
+ - pli-labs
17
+ - legal-ai
18
+ - source-code
19
  ---
20
 
21
+ # DeepSeek-V4.1-Flash on two DGX Spark / GB10 nodes with x86 RTX 3090 offload
22
 
23
+ Keep the target on two DGX Spark nodes while an x86 host supplies Engram tables from RAM and speculative drafts from an RTX 3090. Spend the freed Spark memory on KV capacity, and use a separate CPU proxy process to overlap remote drafting with engine work. This overview links the independently usable code repositories and distinguishes deployed behavior from research candidates.
 
 
24
 
25
+ **GitHub code:** [Source and docs](https://github.com/ProprietaryLegal/DeepSeek-V4.1-Flash-2xDGX-Spark-x86-offload) · **Collection:** [All entries](https://huggingface.co/collections/ProprietaryLegal/deepseek-v41-flash-on-2x-dgx-spark-x86-offload-6adab604a517b426ded3916e)
 
 
 
 
 
 
 
 
 
 
 
 
26
 
27
+ <!-- pli:blurb:start -->
28
 
29
+ > Published by [PLI Labs](https://proprietarylegal.ai/labs.html), the open research arm of [Proprietary Legal Intelligence](https://proprietarylegal.ai). PLI builds AI systems around a legal organization's own work product, combining retrieval, staged workflows, validation, and attorney review. Visit [proprietarylegal.ai](https://proprietarylegal.ai) to explore our work and discuss what your organization needs.
30
 
31
+ <!-- pli:blurb:end -->
32
+
33
+ ## Status
34
+
35
+ 2026-10-23 — production: remote Engram, one-card remote EXL3 drafting, expanded KV, and async process handoff; the two-card NVLink worker is opt-in, with production promotion pending.
36
+
37
+ The one-card async switch passed startup, greedy-output, and watchdog checks. The two-card configuration passed supervised GB10 fault drills. Later independent review cleared its ownership/window fixes for integration; that clearance does not itself establish deployment. Native drafter fidelity, packed KV, larger context, prefix storage, prefill ordering, self-distillation, and further upstream adoption remain separately gated.
38
+
39
+ ## What it is
40
+
41
+ The served target is the public [DeepSeek-V4.1-Flash-1017 EXL3 2.75 bpw legal-weighted build](https://huggingface.co/ProprietaryLegal/DeepSeek-V4.1-Flash-1017-EXL3-2.75bpw-LegalWeighted-DGX-Spark). It is a re-quantization, not a weight fine-tune. This release contains documentation and configuration examples; obtain implementation code from the standalone components below. No weights or private trajectories are bundled.
42
+
43
+ | component | placement | production role |
44
  |---|---|---|
45
+ | target model, 40 layers | both GB10 nodes, tensor parallel | target verification, sampling and token generation |
46
+ | KV cache | both GB10 nodes | retains the extra capacity released by drafter offload |
47
+ | Engram tables, layers 1 and 14 | x86 host RAM, about 189 GiB resident | independent RDMA link to each Spark; local NVMe reader retained as fallback |
48
+ | EXL3 speculative drafter, three layers plus Markov head | one host RTX 3090 | capacity policy keeps the drafter off the Sparks |
49
+ | async proxy | separate CPU process on the engine host | torch-free remote round, shared-memory host flags, watchdog and bounded respawn |
50
+ | optional tensor-parallel drafter | NVLinked RTX 3090 pair | opt-in test variant; promotion pending |
51
+
52
+ ### Component repositories
53
+
54
+ The standalone repositories are canonical. The older hub folders are snapshots. Every row has a GitHub source repository and a Hugging Face entry.
55
+
56
+ | component | GitHub | Hugging Face | status at source snapshot |
57
+ |---|---|---|---|
58
+ | Async host-flag handoff | [source](https://github.com/ProprietaryLegal/dsv41-async-hostflag-handoff) | [entry](https://huggingface.co/ProprietaryLegal/dsv41-async-hostflag-handoff) | Production, one-card worker |
59
+ | Remote RTX 3090 drafter | [source](https://github.com/ProprietaryLegal/dsv41-remote-dspark-drafter-3090) | [entry](https://huggingface.co/ProprietaryLegal/dsv41-remote-dspark-drafter-3090) | Production; capacity mode |
60
+ | NVLink two-card drafter | [source](https://github.com/ProprietaryLegal/dsv41-dspark-drafter-tp2-nvlink) | [entry](https://huggingface.co/ProprietaryLegal/dsv41-dspark-drafter-tp2-nvlink) | Opt-in; GB10 measured and reviewed; promotion pending |
61
+ | Remote Engram | [source](https://github.com/ProprietaryLegal/dsv41-remote-engram-rdma) | [entry](https://huggingface.co/ProprietaryLegal/dsv41-remote-engram-rdma) | Production; exact local fallback |
62
+ | ofl transport | [source](https://github.com/ProprietaryLegal/ofl-rdma-transport) | [entry](https://huggingface.co/ProprietaryLegal/ofl-rdma-transport) | Production |
63
+ | Native 1017 drafter | [source](https://github.com/ProprietaryLegal/dsv41-native-1017-drafter-sm86) | [entry](https://huggingface.co/ProprietaryLegal/dsv41-native-1017-drafter-sm86) | Merged opt-in only; strict fidelity gate fails; not serving |
64
+ | Long-context tools | [source](https://github.com/ProprietaryLegal/dsv41-longctx-bench-tools) | [entry](https://huggingface.co/ProprietaryLegal/dsv41-longctx-bench-tools) | Measured benchmark tooling |
65
+ | Exact acceptance ceiling | [source](https://github.com/ProprietaryLegal/dsv41-exact-acceptance-ceiling) | [entry](https://huggingface.co/ProprietaryLegal/dsv41-exact-acceptance-ceiling) | Research-only; method and aggregates |
66
+ | Full stack | [source](https://github.com/ProprietaryLegal/DeepSeek-V4.1-Flash-2xDGX-Spark-x86-offload) | [entry](https://huggingface.co/ProprietaryLegal/DeepSeek-V4.1-Flash-2xDGX-Spark-x86-offload) | Production configuration overview |
67
+ | Findings | [source](https://github.com/ProprietaryLegal/dsv41-spark-offload-findings) | [entry](https://huggingface.co/ProprietaryLegal/dsv41-spark-offload-findings) | Research findings and limitations |
68
+
69
+ ## Why
70
+
71
+ The capacity trade lets the target retain more simultaneous long contexts. Engram offload shortens long-prompt prefill; drafter offload releases memory. Synchronous remote drafting costs single-stream decode time because engine CPU preparation loses overlap. The process handoff recovers part of that cost while keeping prefill near the synchronous control.
72
+
73
+ ## When to use it
74
+
75
+ Use this architecture when simultaneous long requests exceed the stock KV pool, host RAM and RDMA are available, and a separate GPU can draft. Components can also be deployed independently. Measure both first-token latency and whole-task time for your workload.
76
+
77
+ ## When not to use it
78
 
79
+ It is a poor trade if short single-stream decoding is the only priority and the extra capacity is unused. The source snapshot does not qualify native drafting, prefix restoration, scheduler changes, or larger-context operation for production. A host failure preserves fallback service but reduces performance.
80
 
81
+ ## Requirements
82
 
83
+ | item | tested configuration |
84
  |---|---|
85
+ | serving nodes | two NVIDIA DGX Spark / GB10, 121 GiB unified memory each; TP2 over ConnectX-7 200 Gb/s fabric |
86
+ | offload host | x86 AMD EPYC 7302P, 16 cores / 32 threads, one NUMA node, 503 GiB RAM |
87
+ | host links | ConnectX-5 Ex, PCIe Gen4 x16, one 40 Gb/s DAC per Spark, MTU 9000, RoCE v2 |
88
+ | drafter GPUs | RTX 3090 24 GB; optional NV4 NVLink pair; 325 W cap per drafter card |
89
+ | serving software | `ghcr.io/henryous/deepseek-v41-1017-dgx-spark:v5`, vLLM `0.31.1.dev0+g6e517b15.dsv41gb10`, upstream patches 0022–0050; exllamav3 1.6.0 |
90
+ | production context/configuration | maximum length 327,680; four sequences; KV 6.0 GiB/rank; five draft tokens with batch-aware depth |
91
+
92
+ The upstream image is based on HeNryous revision `b1726644`; the tested launcher is based on sudoingX revision `cc59c36`. Our [image/offload fork](https://github.com/ProprietaryLegal/deepseek-v41-1017-dgx-spark) carries the offload implementation beside the image patch set. Our [launcher fork](https://github.com/ProprietaryLegal/dgx-spark-deepseek-v41) adds optional environment and read-only mount hooks, inert when unset. Configure actual addresses and GPU identifiers locally.
93
+
94
+ ## Measured effect
95
+
96
+ Historical baseline suite, 2026-10-22: same target, two GB10 nodes and x86 RTX 3090 host; `T = 1`, `top_p = 1`, high thinking effort, fresh cache salts unless stated, 1,500-token budgets. A = stock local Engram/drafter; B = remote Engram; D = both offloads, synchronous one-card drafting and KV 6.0 GiB/rank. Cold summary rows below contain three observations each, including the cold priming request from the warm-cache test. They are historical synchronous results; the current async deployment has separate checks below. Means use sample standard deviation.
97
+
98
+ | metric / context | A stock | B Engram only | D synchronous offload | concurrency; repeats / scope |
99
+ |---|---|---|---|---|
100
+ | KV pool, tokens | 989,460 | 989,460 | 1,604,926 | allocation report; +62% capacity |
101
+ | short stream, tok/s | 55.96 ± 1.65 | — | 51.07 ± 0.78 | one; three repeats |
102
+ | 32K cold TTFT, s | 15.53 ± 0.12 | 14.70 ± 0.07 | 14.81 ± 0.54 | one; three cold observations, including priming |
103
+ | 32K cold stream, tok/s | 64.57 ± 0.80 | — | 60.82 ± 2.30 | one; three cold observations, including priming |
104
+ | 150K cold TTFT, s | 73.62 ± 0.11 | 69.44 ± 0.04 | 68.79 ± 0.14 | one; three cold observations, including priming |
105
+ | 150K task time, s | 100.35 ± 0.55 | 95.17 ± 1.61 | 96.46 ± 1.11 | one; three cold observations, including priming |
106
+ | 256K / 300K TTFT, s | — | — | 118.7 / 140.3 | one; one observation at each context |
107
+ | 256K / 300K stream, tok/s | — | — | 51.6 / 52.2 | one; one observation at each context |
108
+ | 32K aggregate, tok/s | 40.59 / 47.21 | — | 40.81 / 47.36 | two / three concurrent; two groups per configuration |
109
+ | 300K capacity | does not fit | does not fit | all four complete | four concurrent; one capacity drill |
110
+ | available memory per Spark, GiB | 5–7 | — | 5–8 | selected serving snapshots; not suite minima or memory guarantees |
111
+
112
+ In this suite, D improved TTFT by 4.6% at 32K and 6.6% at 150K, and completed the 150K task 3.9% sooner than A. Synchronous remote rounds still cost 3–9% single-stream decode speed. Engram alone improved TTFT by 5.4% / 5.7% at 32K / 150K; its decode effect was within spread. Offloading the drafter freed about 2.5 GiB of model load and 3–5 GiB available memory per rank before spending it on KV. Under concurrency the synchronous drafter penalty was at most 2% in the measured groups.
113
+
114
+ ### Separate-process async handoff, 2026-10-23
115
+
116
+ Same hardware, target and budget; direct SSE-gap step timer, `T = 1`. The control is one-card synchronous D. The candidate is the two-card worker plus a napping process proxy with default lazy module loading. The synchronous control was already serving; the process-proxy engine started separately and ran one-card checks before this two-card set. Listed values are per-request step p50, not means of token throughput. The paired timings are shown directly; no aggregate percentage is inferred from them. Repetition counts are explicit in each row.
117
+
118
+ | workload; concurrency; repeats | synchronous one-card step, ms | two-card async step, ms |
119
  |---|---|---|
120
+ | short; one; three requests | 39.6 / 40.1 / 39.4 | 37.1 / 37.4 / 36.9 |
121
+ | 32K; one; three requests | 41.2 / 42.2 / 42.3 | 39.6 / 39.5 / 40.2 |
122
+ | 32K; three concurrent; one group | 75.9 / 76.1 / 76.1 | 72.8 / 73.5 / 73.0 |
123
+ | 150K; one; two requests | 41.5 / 42.2 | 39.6 / 39.8 |
124
+ | 32K / 150K TTFT; one; three / two requests | 14.56–14.59 / 69.10–69.11 s | 14.49–14.63 / 68.89–69.09 s |
125
+ | serving drafter memory; same suite snapshots | one card ~8.1 GiB | 5,862 + 5,222 MiB |
126
 
127
+ Host-memory RDMA probes on 2026-10-22 used 64 MiB transfers over the same host links. These link probes do not measure inference throughput.
128
 
129
+ | link scope | measured bandwidth | repetitions / scope |
130
+ |---|---|---|
131
+ | one link | about 39.2 Gb/s | iteration-count read/write probes; not a repeat-run mean |
132
+ | both links simultaneously | 78.38 Gb/s aggregate | four read/write direction cases, overlapping three-second windows |
133
 
134
+ The one-card process proxy also lowered step p50 on each matched workload against the synchronous control. Two cards alone with synchronous rounds showed smaller step-time reductions than the combined async variant in both synchronous comparison sets. Nap and spin process variants were within run-to-run noise. The merged, hardened runtime repeated the performance without a measurable cost.
135
+
136
+ | additional check, 2026-10-23; same two-GB10 / x86 hardware | observation | conditions / repeats |
137
+ |---|---|---|
138
+ | two-card async, 32K four concurrent | 51.7 tok/s overall; about 119 tok/s combined decode; step p50 87.1–88.2 ms | one four-request group; overall includes prefill, decode estimate sums stream rates |
139
+ | stream speed in initial async A/B | 32K 57.6 → 62.2 tok/s; 150K 51.0 → 54.4 tok/s | one concurrent; means of three / two runs; acceptance-sensitive |
140
+ | production one-card async startup | ready in 363 s; short TTFT 0.37 / 0.38 s; short step 38.4 / 38.5 ms | one restart; two short checks |
141
+ | production one-card async, 32K | first TTFT 16.24 s versus about 14.5 s warm; step 40.5 ms | one first-long-request check; lazy-load cost remains |
142
+ | production greedy output check | 6/6 identical | fixed prompts; temperature zero, thinking off; pre-switch reference |
143
+ | production steady counters | 2,048 rounds OK; zero failed; zero watchdog fires | one post-switch snapshot, not a long-term reliability bound |
144
+
145
+ The standalone component cards carry the detailed fault-drill and boundary-exactness evidence. Acceptance changes stochastically between runs, so direct step timing is the primary runtime comparison. Do not add the isolated percentages together or describe a per-round change as an equal whole-task improvement.
146
+
147
+ ## Upstream claims
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
148
 
149
+ HeNryous reports restoring a 75K prefix across a restart in 1.3–2.1 s versus 33.8 s. That is an upstream result, not our prefix-store measurement. Its applicability to this offload stack still needs engine tests. See the [upstream image fork](https://github.com/ProprietaryLegal/deepseek-v41-1017-dgx-spark).
 
 
150
 
151
+ DeepSeek documents model support for 1,048,576 positions. Our larger-context GB10 correctness tests are pending. Upstream KV-packing or kernel speed claims do not substitute for measured performance here.
152
 
153
+ ## Deploy
154
 
155
+ Follow the [launcher fork](https://github.com/ProprietaryLegal/dgx-spark-deepseek-v41) for the two-node image setup, then install each standalone component in this order:
156
+
157
+ 1. Run the stock target and establish a cold-cache, fixed-settings baseline. Record target and tokenizer identities, image and source versions, cache allocation and sampling settings.
158
+ 2. Build the [ofl transport](https://github.com/ProprietaryLegal/ofl-rdma-transport) for the host and Spark architectures. Qualify each host link and exact Engram rows before enabling [remote Engram](https://github.com/ProprietaryLegal/dsv41-remote-engram-rdma). Keep local disk fallback.
159
+ 3. Install the matching [remote drafter worker and Spark overlay](https://github.com/ProprietaryLegal/dsv41-remote-dspark-drafter-3090). Match drafter and Markov identities. Verify shadow parity, then select capacity mode before raising KV.
160
+ 4. Start from the [sanitized historical D configuration](config/production-D.env.example), replacing all placeholders. Apply the [async delta](config/production-async.env.example) using the [handoff component procedure](https://github.com/ProprietaryLegal/dsv41-async-hostflag-handoff). It requires the host-flag helper built for GB10, matching read-only mounts, and the full process-proxy package.
161
+ 5. Confirm async self-test, successful memory locking, a separate torch-free process, worker identity/connection, fresh watchdog and fallback counters, and the expected KV pool. Warm short and long-prefill paths. Re-run matched cold/warm, concurrent and long-context checks before accepting the capacity trade.
162
+ 6. Treat the [NVLink worker](https://github.com/ProprietaryLegal/dsv41-dspark-drafter-tp2-nvlink) as a separate change with pair safety, owned capture streams, prebuilt peer extension, lifetime GPU locks, supervised fault drills and rollback rehearsal.
163
+
164
+ These examples are not a one-command installer. The older component staging scripts retain assumptions about their original directory layout and pinned build locations; adapt and verify those explicitly. Use the [benchmark tools](https://github.com/ProprietaryLegal/dsv41-longctx-bench-tools) with your own prompts; this release excludes benchmark inputs and raw output rows.
165
+
166
+ ## Rollback
167
+
168
+ | change to remove | persistent configuration change, then engine restart | resulting behavior |
169
  |---|---|---|
170
+ | async handoff | unset `VLLM_DSPARK_REMOTE_ASYNC` on both ranks | synchronous remote rounds |
171
+ | two-card worker | replace the pair worker with the known-good one-card worker, preserving identities | same protocol; reconnect after worker readiness |
172
+ | remote drafter | remove remote drafter environment and overlay mounts; restore KV to 3.75 GiB/rank before local drafting | local drafter with the baseline cache budget |
173
+ | remote Engram | remove remote Engram endpoint and overlay environment | local NVMe Engram reader |
174
+
175
+ Changing a shell variable alone does not reconfigure the running engine. Keep the worker GPU hold until in-window compute has ended. If the host is unavailable, Engram falls back to disk and drafting fails closed to target-only decoding; this preserves requests at reduced performance. Capacity mode does not recreate a local drafter automatically.
176
+
177
+ ## Limits and known issues
178
+
179
+ - The combined package is an overview and configuration index. Implementation trees live in the standalone repositories; the historical hub folders are snapshots.
180
+ - The first long request after restart can still pay lazy-loading cost despite a short warm-up. Production counters cover a bounded check, not extended availability.
181
+ - Packed KV and larger context are reviewed offline candidates; GB10 cache identity, eviction/pressure and needle tests remain pending. Allocator arithmetic is not a serving result.
182
+ - Prefix storage: local storage overlay is cleared for gated GB10 testing. The host-RAM tier is a CPU/network prototype; engine/catalogue integration and real multi-request restoration remain unqualified.
183
+ - Prefill ordering: independent review found an admission-failure path that could issue empty steps indefinitely. A fix is implemented and CPU-tested; fresh independent review and GB10 validation are pending. Earlier simulator predictions do not establish GB10 improvement.
184
+ - Native drafter fidelity: online-block attention and FP32 head corrections are implemented in a development candidate, but the strict stage-zero attention gate still fails. Historical speed figures predate that correction and are not a qualified gain for the corrected worker.
185
+ - Drafter self-distillation tooling is in progress. The [exact-ceiling method](https://github.com/ProprietaryLegal/dsv41-exact-acceptance-ceiling) identifies headroom; no adapted drafter gain is established.
186
+ - Further upstream adoption is offline, opt-in work: target MoE row splitting, replay/upload changes, Engram lookahead, and separate configuration candidates. It has no GB10 performance qualification and should be tested independently before combination.
187
+ - One x86 host is a shared performance dependency. Recovery and target-only behavior must be tested under your own operational constraints.
188
+
189
+ ## Repository layout
190
+
191
+ | path / repository | purpose |
192
+ |---|---|
193
+ | [README.md](README.md) | architecture, evidence and component index |
194
+ | [config/production-D.env.example](config/production-D.env.example) | historical synchronous D settings with placeholders |
195
+ | [config/production-async.env.example](config/production-async.env.example) | persistent async process-proxy delta |
196
+ | [NOTICE](NOTICE) | licence and distribution boundary |
197
+ | [LICENSES/Apache-2.0.txt](LICENSES/Apache-2.0.txt) | complete licence for the overview and configuration examples |
198
+ | standalone components above | Engram server/overlay, transport, Spark handoff, one/two-card worker, native research kernels, benchmarks and analysis |
199
+
200
+ There is no combined `code/` tree in this package. Use each component's code map and tests from its own repository.
201
+
202
+ ## Tests
203
+
204
+ Run CPU suites from each standalone component root, using its documented environment:
205
+
206
+ ```bash
207
+ # Async handoff or remote-drafter entry:
208
+ PYTHONPATH=code/dspark/spark python3 -B -m pytest -q -p no:cacheprovider code/dspark/spark/tests
209
+ # Two-card worker entry (GPU tests skip on a CPU-only machine):
210
+ PYTHONPATH=code/dspark/worker python3 -B -m pytest -q -p no:cacheprovider code/dspark/worker/tests --ignore=code/dspark/worker/tests/test_fused.py
211
+ # Benchmark entry:
212
+ python3 -B -m pytest -q -p no:cacheprovider code/bench/test_bench2.py code/bench/test_s1_links.py
213
+ ```
214
+
215
+ Recorded publication-copy checks on 2026-10-23: Spark suite 221 passed / 2 skipped; worker suite 64 passed / 40 skipped, with the fused GPU suite excluded. Earlier publication checks recorded Spark 167 passed / 2 skipped, worker 39 passed, TP2 29 passed and benchmark 20 passed. These describe different snapshots and test scopes, not one combined test count. The current combined package has no executable CPU suite. See each component for its latest count and required contract file; passing CPU checks does not qualify GB10 deployment.
216
+
217
+ ## Credits and licences
218
+
219
+ PLI Labs integration and analysis code is Apache-2.0. Credits: DeepSeek for the [target and drafter](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017); HeNryous for the Spark image, patches and EXL3 build; sudoingX for the two-node recipe; turboderp for [exllamav3](https://github.com/turboderp-org/exllamav3); [vLLM](https://github.com/vllm-project/vllm) for the serving engine. Upstream software and model weights retain their own licences. The native-kernel repository includes component-specific attributions; consult its NOTICE and licence files before redistribution.
220
+
221
+ ## Related
222
+
223
+ - `dsv41-async-hostflag-handoff`: [GitHub](https://github.com/ProprietaryLegal/dsv41-async-hostflag-handoff) · [Hugging Face](https://huggingface.co/ProprietaryLegal/dsv41-async-hostflag-handoff).
224
+ - `dsv41-remote-dspark-drafter-3090`: [GitHub](https://github.com/ProprietaryLegal/dsv41-remote-dspark-drafter-3090) · [Hugging Face](https://huggingface.co/ProprietaryLegal/dsv41-remote-dspark-drafter-3090).
225
+ - `dsv41-dspark-drafter-tp2-nvlink`: [GitHub](https://github.com/ProprietaryLegal/dsv41-dspark-drafter-tp2-nvlink) · [Hugging Face](https://huggingface.co/ProprietaryLegal/dsv41-dspark-drafter-tp2-nvlink).
226
+ - `dsv41-remote-engram-rdma`: [GitHub](https://github.com/ProprietaryLegal/dsv41-remote-engram-rdma) · [Hugging Face](https://huggingface.co/ProprietaryLegal/dsv41-remote-engram-rdma).
227
+ - `ofl-rdma-transport`: [GitHub](https://github.com/ProprietaryLegal/ofl-rdma-transport) · [Hugging Face](https://huggingface.co/ProprietaryLegal/ofl-rdma-transport).
228
+ - `dsv41-native-1017-drafter-sm86`: [GitHub](https://github.com/ProprietaryLegal/dsv41-native-1017-drafter-sm86) · [Hugging Face](https://huggingface.co/ProprietaryLegal/dsv41-native-1017-drafter-sm86).
229
+ - `dsv41-longctx-bench-tools`: [GitHub](https://github.com/ProprietaryLegal/dsv41-longctx-bench-tools) · [Hugging Face](https://huggingface.co/ProprietaryLegal/dsv41-longctx-bench-tools).
230
+ - `dsv41-exact-acceptance-ceiling`: [GitHub](https://github.com/ProprietaryLegal/dsv41-exact-acceptance-ceiling) · [Hugging Face](https://huggingface.co/ProprietaryLegal/dsv41-exact-acceptance-ceiling).
231
+ - `DeepSeek-V4.1-Flash-2xDGX-Spark-x86-offload`: [GitHub](https://github.com/ProprietaryLegal/DeepSeek-V4.1-Flash-2xDGX-Spark-x86-offload) · [Hugging Face](https://huggingface.co/ProprietaryLegal/DeepSeek-V4.1-Flash-2xDGX-Spark-x86-offload).
232
+ - `dsv41-spark-offload-findings`: [GitHub](https://github.com/ProprietaryLegal/dsv41-spark-offload-findings) · [Hugging Face](https://huggingface.co/ProprietaryLegal/dsv41-spark-offload-findings).
233
+
234
+ Upstream forks: [Spark image and offload integration](https://github.com/ProprietaryLegal/deepseek-v41-1017-dgx-spark) · [two-node launcher](https://github.com/ProprietaryLegal/dgx-spark-deepseek-v41). [HF collection](https://huggingface.co/collections/ProprietaryLegal/deepseek-v41-flash-on-2x-dgx-spark-x86-offload-6adab604a517b426ded3916e) · [hub snapshot](https://github.com/ProprietaryLegal/pli-labs/tree/main/research/dgx-spark-offload).
235
+
236
+ <!-- pli:footer:start -->
237
+
238
+ ---
239
+
240
+ ## About PLI Labs
241
+
242
+ “Your proprietary work product is your proprietary intelligence.” PLI Labs is the open research arm of Proprietary Legal Intelligence. We publish legal model prunes, serving recipes for owned hardware, and open-source tools so readers can inspect our methods and test our claims before hiring us.
243
+
244
+ Founded by a practicing trial lawyer, PLI builds AI systems around a legal organization's own work product: retrieval, staged workflows, validation, and attorney review. The approach grew from systems running on live matters in a litigation practice. Our consulting work supports deployment through approved frontier APIs, private cloud, or open-weight models on hardware you own. Explore the research or contact us about your organization's work.
245
+
246
+ [Website](https://proprietarylegal.ai) · [PLI Labs](https://proprietarylegal.ai/labs.html) · [Research](https://proprietarylegal.ai/research/index.html) · [GitHub](https://github.com/ProprietaryLegal) · [Hugging Face](https://huggingface.co/ProprietaryLegal) · [Contact](https://proprietarylegal.ai/contact.html)
247
+
248
+ Proprietary Legal Intelligence LLC is a technology consultancy, not a law firm, and does not provide legal services or legal advice.
249
+
250
+ <!-- pli:footer:end -->
config/production-D.env.example CHANGED
@@ -1,4 +1,5 @@
1
- # Production configuration "D-kv6" (sanitized). Replace every <...>. These variables are read by the serve script and
 
2
  # passed into the vLLM container on both Spark ranks; see each optimization's entry for the full procedure.
3
 
4
  # --- base serve (stock two-Spark recipe, image ghcr.io/henryous/deepseek-v41-1017-dgx-spark:v5) ---
@@ -26,5 +27,5 @@ OFFLOAD_DOCKER_ARGS="-v <staged>/dspark/ofl:/opt/dsv41-ofl:ro \
26
  # the drafter's freed memory spent on KV: 6.0 GiB per rank (stock 4026531840 = 3.75 GiB)
27
  PROD_KV_BYTES=6442450944
28
 
29
- # Rollback to B: comment out the four D lines above (OFFLOAD_DSPARK_ENV, OFFLOAD_DOCKER_ARGS, PROD_KV_BYTES) and restart.
30
  # Rollback to the stock recipe: also comment out OFFLOAD_DOCKER_ENV.
 
1
+ # Historical synchronous configuration. Apply production-async.env.example separately for the documented async setting.
2
+ # Historical configuration "D-kv6" (sanitized). Replace every <...>. These variables are read by the serve script and
3
  # passed into the vLLM container on both Spark ranks; see each optimization's entry for the full procedure.
4
 
5
  # --- base serve (stock two-Spark recipe, image ghcr.io/henryous/deepseek-v41-1017-dgx-spark:v5) ---
 
27
  # the drafter's freed memory spent on KV: 6.0 GiB per rank (stock 4026531840 = 3.75 GiB)
28
  PROD_KV_BYTES=6442450944
29
 
30
+ # Rollback to B: remove the D-specific settings above (OFFLOAD_DSPARK_ENV, OFFLOAD_DOCKER_ARGS, PROD_KV_BYTES) and restart.
31
  # Rollback to the stock recipe: also comment out OFFLOAD_DOCKER_ENV.
config/production-async.env.example ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Delta for the historical D environment, not a complete launcher configuration.
2
+ # Append these variables to the persistent drafter environment passed to BOTH ranks.
3
+ # Build the helper and stage the complete matching process-proxy package first.
4
+ # Replace the helper path to match the read-only transport mount in your deployment.
5
+ VLLM_DSPARK_REMOTE_ASYNC=1
6
+ VLLM_DSPARK_REMOTE_ASYNC_PROXY=process
7
+ VLLM_DSPARK_REMOTE_ASYNC_SPIN_US=50
8
+ VLLM_DSPARK_REMOTE_ASYNC_NAP_US=20
9
+ VLLM_DSPARK_REMOTE_ASYNC_RESPAWNS=3
10
+ VLLM_DSPARK_REMOTE_ASYNC_RESPAWN_RESET_S=3600
11
+ VLLM_DSPARK_REMOTE_HOSTFLAG_LIB=/opt/dsv41-ofl/libdsv41_hostflag.so
12
+ # Leave CUDA_MODULE_LOADING at its default. Preserve endpoint, identities,
13
+ # capacity policy, timeout, backoff cap, KV and the historical calibrated cost table.
14
+ # That cost table has not been refitted for the two-card async path.
15
+ # Rollback: remove VLLM_DSPARK_REMOTE_ASYNC from the persistent environment and restart.