Six context channels on Python code with V_θ's learning rate × 0.25 (screen B): the dead wells fixed, the sixth channel adds nothing (Gen 3)
The question of this collection: why did a sixth context channel cost SO-CODE 43% on Python code? Its second layer's potential wells died: they drifted away from the states, where a well exerts no force and gets no gradient, so it cannot come back. The root cause and the analysis are in
Dead_Wells_in_the_Joint_V_theta_Bank.md; this card reports one model of it.This model (screen B): XI6-CODE, which adds a sixth content-routed context channel to SO-CODE, trained with V_θ's learning rate × 0.25. Verdict: the fix works, and the sixth channel adds nothing. The layer-1 wells stay alive and the model settles at 5.98, level with SO-CODE (6.00) and with SO-CODE-VTLR (6.09), SO-CODE with the same slower V_θ. XI6-CODE at full speed: 8.55. A negative result for the channel, published as such. A second fix, screen A, scales every context channel to unit norm at V_θ's bank input instead, with V_θ at full speed: it settles at 5.95. Its card compares the two fixes. On five channels that switch costs 5.8% (SO-CODE-VTUN, 6.34), more than this slower V_θ's 1.6%. The dead-wells collection holds the baseline, two fixes and their two controls.
Result
| model | context channels | V_θ learning rate | layer-1 wells | settled perplexity | against SO-CODE |
|---|---|---|---|---|---|
| SO-CODE, the baseline | 5 | × 1 | alive | 6.00 | — |
| XI6-CODE (not on the Hub) | 6 | × 1 | dead | 8.55 | +42.6% |
| screen B, the fix | 6 | × 0.25 | alive | 5.98 | −0.3% |
| SO-CODE-VTLR, the control | 5 | × 0.25 | alive | 6.09 | +1.6% |
| screen A, the second fix (unit-norm input) | 6 | × 1 | alive | 5.95 | −0.7% |
| SO-CODE-VTUN, the second fix's control (unit-norm input) | 5 | × 1 | alive | 6.34 | +5.8% |
A first mitigation, a penalty that pulls each token's nearest well back within reach, failed: XI6-CODE-WR settled at 8.87 (also not on the Hub). The matched GPT-2 on the same code tokens settles at 4.96.
- This model: settled 5.98 (mean of the evals at steps 18,000, 18,500 and 19,000: 5.99, 5.91, 6.04); best 5.86 at step 16,000. Validation perplexity at block 512, on the first 2M tokens of CodeParrot's validation files (a file-level split, so no file is in both).
- Trained on exactly 315,392,000 tokens (19,250 steps × 32 × 512): the ladder's WSD schedule over 16,250 steps, then 3,000 steps at its floor learning rate. All the models in the table start from step 0 at seed 0 on the same batches, so their evaluations are paired step by step.
- 83,690,702 parameters (+9.3% against SO-CODE, mostly V_θ's bank reading the sixth channel).
- Pre-registered (
Code_Dead_Wells_and_Screens.md, entryp-so-code-vtlr, before SO-CODE-VTLR trained): VT1 (SO-CODE-VTLR settles within ±2% of SO-CODE), called 55%: HIT, +1.6%. VT2 (it is more than 5% ahead of SO-CODE at step 3,000), called 65%: HIT, −12.7%. VT3 (its layer-1 wells stay alive), called 90%: HIT, nearest-well ratio at most 1.03 at seven checkpoints, removal cost +272%. VT4 (screen B and SO-CODE-VTLR within ±2% of each other), called 60%: HIT, −1.8%. Proposed before the run, not confirmed by the author, and scored on the letter. - Run health: steps 1–3,000 as a screen (0.96 h), then resumed from its step-3,000 checkpoint (model and optimizer state, 89 of 89 tensors) for steps 3,001–19,250 in 5.2 h in one session; 0 watchdog triggers; largest rise between logged training losses 0.075; gradient norm maximum 1.89 in warmup (step 450) and 0.84 after the resume; SCAF leak monitor CLEAN at all 4 audits.
Paired by step
The mean difference in log perplexity (nats) between two models over a span of evaluations, with the standard deviation of the per-evaluation differences. Models that differ only in V_θ's learning rate move together, so their pairing is tight.
| steps | SO-CODE-VTLR − SO-CODE | screen B − SO-CODE-VTLR | screen B − SO-CODE |
|---|---|---|---|
| 2,000–4,000 (5 evals) | −0.121 (SD 0.043) | −0.013 (0.033) | −0.134 (0.034) |
| 8,000–13,000 (11 evals) | +0.001 (SD 0.006) | +0.002 (0.037) | +0.003 (0.039) |
| 16,000–19,000 (7 evals) | +0.015 (SD 0.002) | −0.013 (0.036) | +0.002 (0.035) |
- The early lead is the slower V_θ's. Both models with V_θ × 0.25 start behind SO-CODE (+12% to +16% at steps 500 and 1,000), cross it before step 2,000 and lead by 13–17% at steps 2,500–3,000.
- At the end the slower V_θ costs a healthy model about 1.5%, consistently from step 14,000 (SD of the paired difference 0.002 nats; training loss 1.704 against 1.687).
- The sixth channel's effect, with the slower V_θ in both, is not distinguishable from zero: −0.013 nats late and +0.002 in the middle, against a paired SD of 0.036 per evaluation.
Through training
| step | phase | SO-CODE | SO-CODE-VTLR | screen B |
|---|---|---|---|---|
| 500 | warmup | 44.02 | 51.13 | 50.22 |
| 1,000 | stable | 25.13 | 29.19 | 28.30 |
| 1,500 | stable | 20.33 | 21.45 | 20.57 |
| 2,000 | stable | 17.67 | 16.27 | 15.87 |
| 2,500 | stable | 15.80 | 13.08 | 13.25 |
| 3,000 | stable | 12.96 | 11.32 | 11.27 |
| 4,000 | stable | 10.62 | 9.70 | 9.10 |
| 5,000 | stable | 9.27 | 8.81 | 9.00 |
| 8,000 | stable | 7.96 | 7.86 | 7.83 |
| 10,000 | stable | 7.45 | 7.43 | 7.04 |
| 13,000 | decay | 6.38 | 6.45 | 6.70 |
| 16,000 | decay, end | 6.12 | 6.21 | 5.86 |
| 18,000 | floor | 5.87 | 5.96 | 5.99 |
| 19,000 | floor, last eval | 6.10 | 6.20 | 6.04 |
The layer-1 wells
In layer 1 the states lie on a sphere of radius ‖h‖ = √384 = 19.6. The nearest-well ratio is the distance from a token's state to the nearest well centre, divided by ‖h‖; the occupancy of a well is exp(−½ (h − μ)ᵀ P (h − μ)), the factor its force and its gradient both carry; the removal cost is the perplexity change when every layer-1 well is switched off on the trained model. mu_proj is the map W, shared by both layers, that places the well centres μ from the context channels. Readings by debug/vtheta_well_probe.py on 16 validation windows, at the best checkpoint.
| model | nearest-well ratio, median | mean centre norm ‖μ‖ | mu_proj, largest column norm |
occupancy, strongest well | removal cost |
|---|---|---|---|---|---|
| SO-CODE | 1.03 | 59.9 | 101.5 | 0.18 | +202% |
| XI6-CODE | 3.40 | 91.9 | 112.0 | 0.00 | — (nothing to remove) |
| screen B | 0.77 | 29.7 | 36.4 | 0.17 | +242% |
| SO-CODE-VTLR | 0.71 | 25.6 | 36.9 | 0.15 | +272% |
| step | nearest-well ratio (SO-CODE-VTLR / screen B) | ‖μ‖ | mu_proj |
|---|---|---|---|
| 500 | 1.03 / 1.05 | 12.5 / 13.2 | 22.1 / 21.3 |
| 1,000 | 0.96 / 0.96 | 15.8 / 17.6 | 23.8 / 23.3 |
| 3,000 | 0.81 / 0.82 | 22.0 / 23.0 | 28.9 / 28.3 |
- XI6-CODE's layer-1 wells are dead: the nearest one is 3.4 state-norms away and no token occupies any. Layer 1 runs without its potential, and the optimizer has no signal to bring the wells back.
- With V_θ × 0.25 they stay alive, one well carrying most of the force, as in SO-CODE. Removing them costs more than doubling the perplexity.
- The two slow models' banks match checkpoint for checkpoint, small and inside the state sphere (
mu_projabout a third of SO-CODE's). The bank's scale is set by V_θ's speed; the sixth channel does not change it. - Why the wells die (a race between the growth of W and layer 1's large channel inputs, in steps 750–1,200, just after the learning-rate peak) is the subject of
Dead_Wells_in_the_Joint_V_theta_Bank.md, §4–§7.
Diagnostics
| reading (best checkpoint) | SO-CODE | screen B | SO-CODE-VTLR |
|---|---|---|---|
| Cell 7, full model | 5.51 | 5.52 | 5.57 |
| Cell 6b-7 Gate 1: perplexity cost of resetting the velocity | +4.0% | +2.5% | +3.1% |
| Gate 2: one extra layer at the trained step | +42% | +37% | +28% |
| Gate 3: three steps over the trained time, nats | 0.639 | 0.290 | 0.312 |
| Cell 6b-13: median ω·Δt (layer 0 / layer 1) | 7.55 (9.58 / 4.94) | 5.83 (6.37 / 4.34) | 6.16 (7.16 / 3.58) |
- The slower V_θ halves the cost of refinement (Gate 3), in both models, with or without the sixth channel. All three are maps, not flows: refinement grows away from the trained step.
- No one-position lookups. In XI6-CODE the long layer-1 channels became near one-hot lookups of a single earlier position (median peak weight 0.88–0.97). With live wells they retrieve earlier occurrences as SO-CODE's longest channel does (
debug/xi_reach.py, filesresults/xi_reach_*), so in XI6-CODE the lookups came with the dead wells (screen A later showed that one lookup channel can form alongside live wells).
Reproduction
colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb, Cell 0 changed from the defaults:
REVERSE_CHANNEL=False, VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
LOWRANK_DAMPED_FLOW=True, # SR2
VPHI_ANALYTIC_FORCE=True, LAYER_CHECKPOINT=False, # speed switches, same arithmetic
CORPUS='code', LADDER_STEPS=16_250, EXTRA_FLOOR_STEPS=3_000 # Python code, then 3,000 steps at the floor
XI_OVERRIDE='6long' # a sixth content-routed context channel (XI6-CODE)
WELL_REACH_RHO=3 # the reachability penalty of XI6-CODE-WR, kept so the reach readout matches
VTHETA_LR_MULT=0.25 # V_θ's 9 tensors train at a quarter of the scheduled learning rate (screen B)
SNAPSHOT_STEPS=[1_000, 1_500, 2_000, 2_500, 3_000]
PROBE_MAX_STEPS=3_000 # the screen; set to None and rerun Cell 2 to resume to 19,250
Run Cells 0, 1, 1b, 1c, 1d, 2, 3, 4, 5, 5b and 6, then 8, 7, 6b-7 and 6b-13. The first code run builds the token cache (about 25 minutes). The well probe is debug/vtheta_well_probe.py, the channels' reach debug/xi_reach.py (CPU, on the downloaded run folders).
Training artefacts were written to semsimula_fock_cfc_baoab_code_xi6long_topk16_dt32da16_mh4_aniso_dcvt6x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_wr3l0p1_vtlr0p25_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_noattn/.
Glossary
| term | meaning |
|---|---|
| settled PPL | validation perplexity at block 512, the mean of the last three evals (on this corpus, steps 18,000–19,000). Best is the lowest single eval |
| Gen 1 / Gen 2 / Gen 3 | the model generations. Gen 1: explicit integrators. Gen 2: the CfC/BAOAB ladder with the sources of Vφ and ξ (and of an exchange field) detached, which keeps every force causal, but no loss gradient ever reaches an earlier token through them. Gen 3: the same forward pass with those gradients live |
| live / detached gradients | whether the loss gradient flows into the earlier tokens that act as sources of a force. Detaching changes no forward value; it only cuts that backward path |
| Vθ | the learned scalar potential on a token's state h, given its context ξ: a bank of anisotropic Gaussian wells. Its negative gradient is the conservative force on the token |
| Vφ | a pair potential between a token and a few selected earlier tokens (sparse top-k) |
| ξ | the causal context channels: exponential moving averages of earlier states at several horizons, which condition Vθ |
| Fock mechanism, registers, reverse channel | a pool of register particles, created and destroyed from the token context. The reverse channel is the register-to-token force; it is not the gradient of any potential on the tokens, so it is the model's non-conservative part |
| exchange field | an attention-like token-to-token interaction added to the conservative core. Each token is pushed by earlier tokens through learned query, key and value projections, as in attention. The ladder tests it in two forms, attention and attention_potential |
attention (non-conservative exchange field) |
the exchange term enters as a force directly. It is not the gradient of anything, so the step stops being a geodesic step |
attention_potential (attention as a potential) |
the same interaction written as a scalar potential whose gradient is the force. The routing weights are computed from the hidden states and held fixed with respect to the state being moved, so the force stays an exact gradient and the model stays conservative |
no-exchange (LADDER_MECHANISM = 'none') |
the arm with no exchange field at all: Vθ, Vφ and ξ, plus the Fock register mechanism unless the arm is conservative-only |
live exchange field (RELAX_GRAD_PATH = 'live') |
attention_potential with the identical forward force written out over live tensors, so the loss gradient reaches the field's sources and routing. In Gen 2 these were detached, and the field sent no gradient into any hidden state |
| Gen 2 probe | a Gen 2 arm retrained with one gradient path changed, to test a hypothesis before the full Gen 3 ladder. The live-exchange-field probe (attention_potential with RELAX_GRAD_PATH = 'live', the full 32,500 steps) is the one that found the gradient starvation |
| gpt2-matched | the reference transformer: GPT-2 architecture, d = 384, L = 8, tied embeddings (33.7M parameters), trained on the same tokenizer, corpus, batches and token budget. Shared by Gen 2 and Gen 3 |
tag components (vplive, xilive, rglive, norc, L4probe, idt…) |
pieces of the run's variant tag, so that every configuration gets its own Drive folder: live Vφ sources, live ξ, live exchange field, no reverse channel, the depth L, and the integrator step Δt |
| conservative-only | a model in which every force is a gradient: no reverse channel and no non-conservative exchange field |
| CfC/BAOAB low-rank propagator | the integrator. Each layer step is split into position (A), force kick (B) and friction (O) substeps, and the stiff harmonic modes of Vθ are integrated exactly in closed form |
| ω·Δt, θ | a stiff mode's frequency times the step size. Past 2, an explicit integrator would be unstable (the wall); the closed-form propagator is not. θ/sin θ is the factor by which a split step's dissipation depends on the mode's phase, largest near θ = π |
| geodesic step, R(geo), share | the step a damped particle would take under Vθ alone; R(geo) is the distance of the real step from it, relative to the step's length. A component's share is how much it moves that distance (6b-9 attribution) |
| Gates 1–3, FLOW / MAPS | see Terms in the geometry section |
| CG1–CG7, 6b-N | the CfC-geometry experiments, and the ladder-notebook cells that run them |
| SCAF, Tier A, Tier B | the in-flight causal-leak monitor. Its core test changes future tokens and requires the logits at earlier positions to stay bit-identical. Tier A asks the same of the hidden states, as 1 − cos between the clean and perturbed state at each earlier position and layer. Tier B asks whether the perturbation moves any earlier state into a different dominant Vθ well (the basin-crossing rate). All three read 0 for a causal model |
| leak tax, prefix-only | the loss difference between scoring a position with only its prefix and with the full sequence; about 0 for a causal model |
| clip-hit | the share of logged steps whose gradient norm exceeded the clipping threshold (1.0) |
| pre-registered | a prediction recorded, with its threshold, before the run |
Files
| path | what |
|---|---|
README.md |
this card |
config.json |
architecture, training, the switch, result and diagnostics |
checkpoints/fock_cfc_*noattn_step16000_best.pt |
weights at the best eval (step 16,000), optimizer state stripped |
results/ |
the training log (the 3,000-step screen and its resume to 19,250), the diagnostic cells' outputs, the well probe at the checkpoints (well_probe_*; _L1_none = layer 1's wells removed) and the channels' reach (xi_reach_*) |
code/ |
the notebook and the model code the switches belong to (it also carries later, default-off switches) |
Companion notes
| note | what it covers |
|---|---|
Dead_Wells_in_the_Joint_V_theta_Bank.md |
the dead wells: the symptom, the debugging, the root cause, the failed and the working mitigations, and these results (§7.3, §7.4) |
Code_Dead_Wells_and_Screens.md |
the run records: entries p-xi6-code, p-dead-wells, p-dw-screen-b-full, p-so-code-vtlr and its result |
Caveats
- One seed per model. The pairing removes the evaluation noise, not the training-seed spread, which has not been measured. The 1.5% cost of the slower V_θ is consistent between these two runs; it is not a measured seed effect.
- Whether XI6 loses its wells on every seed is open: it did in 2 of 2 runs (XI6-CODE, XI6-CODE-WR), the five-channel models in 0 of 6.
- Screen B was trained in two parts (a 3,000-step screen, then a resume with the optimizer state), SO-CODE-VTLR in one. Both evaluate on the same batches.
- Not settled by the letter of the settling rule, like every code model: the evals still fall by about 0.02 nats over the last 3,000 steps, against a scatter of 0.015. The paired gaps are flat.
- L = 2, and perplexities on code are not comparable with OpenWebText's or TinyStories'.
Citation
@misc{Gueorguiev2026SemSim,
author = {Gueorguiev, Dimitar P.},
title = {Semantic Simulation: A Prescriptive Lagrangian Framework
for Semantic Inference --- Conservative Potentials,
Conservative and Non-Conservative Memory, and Competing Mechanisms},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.19712427},
url = {https://doi.org/10.5281/zenodo.19712427},
note = {Version v25 (Nov 9, 2026).
Companion code repository (DOI 10.5281/zenodo.24066016):
\url{https://github.com/dimitarpg13/semsimula-paper}}
}
Card generated from _ladder/ladder.json on 2026-11-17.
- Downloads last month
- 934
Datasets used to train dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-xi6long-sr2-wr3-vtlr0p25
codeparrot/codeparrot-clean-train
Collection including dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-xi6long-sr2-wr3-vtlr0p25
Evaluation results
- Validation perplexity at block 512, settled (mean of the last three 500-step evals at block 512) on CodeParrot clean (Python)validation set self-reported5.979