Six context channels on Python code with V_θ's learning rate × 0.25 (screen B): the dead wells fixed, the sixth channel adds nothing (Gen 3)

The question of this collection: why did a sixth context channel cost SO-CODE 43% on Python code? Its second layer's potential wells died: they drifted away from the states, where a well exerts no force and gets no gradient, so it cannot come back. The root cause and the analysis are in Dead_Wells_in_the_Joint_V_theta_Bank.md; this card reports one model of it.

This model (screen B): XI6-CODE, which adds a sixth content-routed context channel to SO-CODE, trained with V_θ's learning rate × 0.25. Verdict: the fix works, and the sixth channel adds nothing. The layer-1 wells stay alive and the model settles at 5.98, level with SO-CODE (6.00) and with SO-CODE-VTLR (6.09), SO-CODE with the same slower V_θ. XI6-CODE at full speed: 8.55. A negative result for the channel, published as such. A second fix, screen A, scales every context channel to unit norm at V_θ's bank input instead, with V_θ at full speed: it settles at 5.95. Its card compares the two fixes. On five channels that switch costs 5.8% (SO-CODE-VTUN, 6.34), more than this slower V_θ's 1.6%. The dead-wells collection holds the baseline, two fixes and their two controls.

Result

model context channels V_θ learning rate layer-1 wells settled perplexity against SO-CODE
SO-CODE, the baseline 5 × 1 alive 6.00 —
XI6-CODE (not on the Hub) 6 × 1 dead 8.55 +42.6%
screen B, the fix 6 × 0.25 alive 5.98 −0.3%
SO-CODE-VTLR, the control 5 × 0.25 alive 6.09 +1.6%
screen A, the second fix (unit-norm input) 6 × 1 alive 5.95 −0.7%
SO-CODE-VTUN, the second fix's control (unit-norm input) 5 × 1 alive 6.34 +5.8%

A first mitigation, a penalty that pulls each token's nearest well back within reach, failed: XI6-CODE-WR settled at 8.87 (also not on the Hub). The matched GPT-2 on the same code tokens settles at 4.96.

  • This model: settled 5.98 (mean of the evals at steps 18,000, 18,500 and 19,000: 5.99, 5.91, 6.04); best 5.86 at step 16,000. Validation perplexity at block 512, on the first 2M tokens of CodeParrot's validation files (a file-level split, so no file is in both).
  • Trained on exactly 315,392,000 tokens (19,250 steps × 32 × 512): the ladder's WSD schedule over 16,250 steps, then 3,000 steps at its floor learning rate. All the models in the table start from step 0 at seed 0 on the same batches, so their evaluations are paired step by step.
  • 83,690,702 parameters (+9.3% against SO-CODE, mostly V_θ's bank reading the sixth channel).
  • Pre-registered (Code_Dead_Wells_and_Screens.md, entry p-so-code-vtlr, before SO-CODE-VTLR trained): VT1 (SO-CODE-VTLR settles within ±2% of SO-CODE), called 55%: HIT, +1.6%. VT2 (it is more than 5% ahead of SO-CODE at step 3,000), called 65%: HIT, −12.7%. VT3 (its layer-1 wells stay alive), called 90%: HIT, nearest-well ratio at most 1.03 at seven checkpoints, removal cost +272%. VT4 (screen B and SO-CODE-VTLR within ±2% of each other), called 60%: HIT, −1.8%. Proposed before the run, not confirmed by the author, and scored on the letter.
  • Run health: steps 1–3,000 as a screen (0.96 h), then resumed from its step-3,000 checkpoint (model and optimizer state, 89 of 89 tensors) for steps 3,001–19,250 in 5.2 h in one session; 0 watchdog triggers; largest rise between logged training losses 0.075; gradient norm maximum 1.89 in warmup (step 450) and 0.84 after the resume; SCAF leak monitor CLEAN at all 4 audits.

Paired by step

The mean difference in log perplexity (nats) between two models over a span of evaluations, with the standard deviation of the per-evaluation differences. Models that differ only in V_θ's learning rate move together, so their pairing is tight.

steps SO-CODE-VTLR − SO-CODE screen B − SO-CODE-VTLR screen B − SO-CODE
2,000–4,000 (5 evals) −0.121 (SD 0.043) −0.013 (0.033) −0.134 (0.034)
8,000–13,000 (11 evals) +0.001 (SD 0.006) +0.002 (0.037) +0.003 (0.039)
16,000–19,000 (7 evals) +0.015 (SD 0.002) −0.013 (0.036) +0.002 (0.035)
  • The early lead is the slower V_θ's. Both models with V_θ × 0.25 start behind SO-CODE (+12% to +16% at steps 500 and 1,000), cross it before step 2,000 and lead by 13–17% at steps 2,500–3,000.
  • At the end the slower V_θ costs a healthy model about 1.5%, consistently from step 14,000 (SD of the paired difference 0.002 nats; training loss 1.704 against 1.687).
  • The sixth channel's effect, with the slower V_θ in both, is not distinguishable from zero: −0.013 nats late and +0.002 in the middle, against a paired SD of 0.036 per evaluation.

Through training

step phase SO-CODE SO-CODE-VTLR screen B
500 warmup 44.02 51.13 50.22
1,000 stable 25.13 29.19 28.30
1,500 stable 20.33 21.45 20.57
2,000 stable 17.67 16.27 15.87
2,500 stable 15.80 13.08 13.25
3,000 stable 12.96 11.32 11.27
4,000 stable 10.62 9.70 9.10
5,000 stable 9.27 8.81 9.00
8,000 stable 7.96 7.86 7.83
10,000 stable 7.45 7.43 7.04
13,000 decay 6.38 6.45 6.70
16,000 decay, end 6.12 6.21 5.86
18,000 floor 5.87 5.96 5.99
19,000 floor, last eval 6.10 6.20 6.04

The layer-1 wells

In layer 1 the states lie on a sphere of radius ‖h‖ = √384 = 19.6. The nearest-well ratio is the distance from a token's state to the nearest well centre, divided by ‖h‖; the occupancy of a well is exp(−½ (h − μ)ᵀ P (h − μ)), the factor its force and its gradient both carry; the removal cost is the perplexity change when every layer-1 well is switched off on the trained model. mu_proj is the map W, shared by both layers, that places the well centres μ from the context channels. Readings by debug/vtheta_well_probe.py on 16 validation windows, at the best checkpoint.

model nearest-well ratio, median mean centre norm ‖μ‖ mu_proj, largest column norm occupancy, strongest well removal cost
SO-CODE 1.03 59.9 101.5 0.18 +202%
XI6-CODE 3.40 91.9 112.0 0.00 — (nothing to remove)
screen B 0.77 29.7 36.4 0.17 +242%
SO-CODE-VTLR 0.71 25.6 36.9 0.15 +272%
step nearest-well ratio (SO-CODE-VTLR / screen B) ‖μ‖ mu_proj
500 1.03 / 1.05 12.5 / 13.2 22.1 / 21.3
1,000 0.96 / 0.96 15.8 / 17.6 23.8 / 23.3
3,000 0.81 / 0.82 22.0 / 23.0 28.9 / 28.3
  • XI6-CODE's layer-1 wells are dead: the nearest one is 3.4 state-norms away and no token occupies any. Layer 1 runs without its potential, and the optimizer has no signal to bring the wells back.
  • With V_θ × 0.25 they stay alive, one well carrying most of the force, as in SO-CODE. Removing them costs more than doubling the perplexity.
  • The two slow models' banks match checkpoint for checkpoint, small and inside the state sphere (mu_proj about a third of SO-CODE's). The bank's scale is set by V_θ's speed; the sixth channel does not change it.
  • Why the wells die (a race between the growth of W and layer 1's large channel inputs, in steps 750–1,200, just after the learning-rate peak) is the subject of Dead_Wells_in_the_Joint_V_theta_Bank.md, §4–§7.

Diagnostics

reading (best checkpoint) SO-CODE screen B SO-CODE-VTLR
Cell 7, full model 5.51 5.52 5.57
Cell 6b-7 Gate 1: perplexity cost of resetting the velocity +4.0% +2.5% +3.1%
Gate 2: one extra layer at the trained step +42% +37% +28%
Gate 3: three steps over the trained time, nats 0.639 0.290 0.312
Cell 6b-13: median ω·Δt (layer 0 / layer 1) 7.55 (9.58 / 4.94) 5.83 (6.37 / 4.34) 6.16 (7.16 / 3.58)
  • The slower V_θ halves the cost of refinement (Gate 3), in both models, with or without the sixth channel. All three are maps, not flows: refinement grows away from the trained step.
  • No one-position lookups. In XI6-CODE the long layer-1 channels became near one-hot lookups of a single earlier position (median peak weight 0.88–0.97). With live wells they retrieve earlier occurrences as SO-CODE's longest channel does (debug/xi_reach.py, files results/xi_reach_*), so in XI6-CODE the lookups came with the dead wells (screen A later showed that one lookup channel can form alongside live wells).

Reproduction

colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb, Cell 0 changed from the defaults:

REVERSE_CHANNEL=False, VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
LOWRANK_DAMPED_FLOW=True,                 # SR2
VPHI_ANALYTIC_FORCE=True, LAYER_CHECKPOINT=False,   # speed switches, same arithmetic
CORPUS='code', LADDER_STEPS=16_250, EXTRA_FLOOR_STEPS=3_000   # Python code, then 3,000 steps at the floor
XI_OVERRIDE='6long'                     # a sixth content-routed context channel (XI6-CODE)
WELL_REACH_RHO=3                        # the reachability penalty of XI6-CODE-WR, kept so the reach readout matches
VTHETA_LR_MULT=0.25                     # V_θ's 9 tensors train at a quarter of the scheduled learning rate (screen B)
SNAPSHOT_STEPS=[1_000, 1_500, 2_000, 2_500, 3_000]
PROBE_MAX_STEPS=3_000                   # the screen; set to None and rerun Cell 2 to resume to 19,250

Run Cells 0, 1, 1b, 1c, 1d, 2, 3, 4, 5, 5b and 6, then 8, 7, 6b-7 and 6b-13. The first code run builds the token cache (about 25 minutes). The well probe is debug/vtheta_well_probe.py, the channels' reach debug/xi_reach.py (CPU, on the downloaded run folders).

Training artefacts were written to semsimula_fock_cfc_baoab_code_xi6long_topk16_dt32da16_mh4_aniso_dcvt6x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_wr3l0p1_vtlr0p25_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_noattn/.

Glossary

term meaning
settled PPL validation perplexity at block 512, the mean of the last three evals (on this corpus, steps 18,000–19,000). Best is the lowest single eval
Gen 1 / Gen 2 / Gen 3 the model generations. Gen 1: explicit integrators. Gen 2: the CfC/BAOAB ladder with the sources of Vφ and ξ (and of an exchange field) detached, which keeps every force causal, but no loss gradient ever reaches an earlier token through them. Gen 3: the same forward pass with those gradients live
live / detached gradients whether the loss gradient flows into the earlier tokens that act as sources of a force. Detaching changes no forward value; it only cuts that backward path
Vθ the learned scalar potential on a token's state h, given its context ξ: a bank of anisotropic Gaussian wells. Its negative gradient is the conservative force on the token
Vφ a pair potential between a token and a few selected earlier tokens (sparse top-k)
ξ the causal context channels: exponential moving averages of earlier states at several horizons, which condition Vθ
Fock mechanism, registers, reverse channel a pool of register particles, created and destroyed from the token context. The reverse channel is the register-to-token force; it is not the gradient of any potential on the tokens, so it is the model's non-conservative part
exchange field an attention-like token-to-token interaction added to the conservative core. Each token is pushed by earlier tokens through learned query, key and value projections, as in attention. The ladder tests it in two forms, attention and attention_potential
attention (non-conservative exchange field) the exchange term enters as a force directly. It is not the gradient of anything, so the step stops being a geodesic step
attention_potential (attention as a potential) the same interaction written as a scalar potential whose gradient is the force. The routing weights are computed from the hidden states and held fixed with respect to the state being moved, so the force stays an exact gradient and the model stays conservative
no-exchange (LADDER_MECHANISM = 'none') the arm with no exchange field at all: Vθ, Vφ and ξ, plus the Fock register mechanism unless the arm is conservative-only
live exchange field (RELAX_GRAD_PATH = 'live') attention_potential with the identical forward force written out over live tensors, so the loss gradient reaches the field's sources and routing. In Gen 2 these were detached, and the field sent no gradient into any hidden state
Gen 2 probe a Gen 2 arm retrained with one gradient path changed, to test a hypothesis before the full Gen 3 ladder. The live-exchange-field probe (attention_potential with RELAX_GRAD_PATH = 'live', the full 32,500 steps) is the one that found the gradient starvation
gpt2-matched the reference transformer: GPT-2 architecture, d = 384, L = 8, tied embeddings (33.7M parameters), trained on the same tokenizer, corpus, batches and token budget. Shared by Gen 2 and Gen 3
tag components (vplive, xilive, rglive, norc, L4probe, idt…) pieces of the run's variant tag, so that every configuration gets its own Drive folder: live Vφ sources, live ξ, live exchange field, no reverse channel, the depth L, and the integrator step Δt
conservative-only a model in which every force is a gradient: no reverse channel and no non-conservative exchange field
CfC/BAOAB low-rank propagator the integrator. Each layer step is split into position (A), force kick (B) and friction (O) substeps, and the stiff harmonic modes of Vθ are integrated exactly in closed form
ω·Δt, θ a stiff mode's frequency times the step size. Past 2, an explicit integrator would be unstable (the wall); the closed-form propagator is not. θ/sin θ is the factor by which a split step's dissipation depends on the mode's phase, largest near θ = π
geodesic step, R(geo), share the step a damped particle would take under Vθ alone; R(geo) is the distance of the real step from it, relative to the step's length. A component's share is how much it moves that distance (6b-9 attribution)
Gates 1–3, FLOW / MAPS see Terms in the geometry section
CG1–CG7, 6b-N the CfC-geometry experiments, and the ladder-notebook cells that run them
SCAF, Tier A, Tier B the in-flight causal-leak monitor. Its core test changes future tokens and requires the logits at earlier positions to stay bit-identical. Tier A asks the same of the hidden states, as 1 − cos between the clean and perturbed state at each earlier position and layer. Tier B asks whether the perturbation moves any earlier state into a different dominant Vθ well (the basin-crossing rate). All three read 0 for a causal model
leak tax, prefix-only the loss difference between scoring a position with only its prefix and with the full sequence; about 0 for a causal model
clip-hit the share of logged steps whose gradient norm exceeded the clipping threshold (1.0)
pre-registered a prediction recorded, with its threshold, before the run

Files

path what
README.md this card
config.json architecture, training, the switch, result and diagnostics
checkpoints/fock_cfc_*noattn_step16000_best.pt weights at the best eval (step 16,000), optimizer state stripped
results/ the training log (the 3,000-step screen and its resume to 19,250), the diagnostic cells' outputs, the well probe at the checkpoints (well_probe_*; _L1_none = layer 1's wells removed) and the channels' reach (xi_reach_*)
code/ the notebook and the model code the switches belong to (it also carries later, default-off switches)

Companion notes

note what it covers
Dead_Wells_in_the_Joint_V_theta_Bank.md the dead wells: the symptom, the debugging, the root cause, the failed and the working mitigations, and these results (§7.3, §7.4)
Code_Dead_Wells_and_Screens.md the run records: entries p-xi6-code, p-dead-wells, p-dw-screen-b-full, p-so-code-vtlr and its result

Caveats

  • One seed per model. The pairing removes the evaluation noise, not the training-seed spread, which has not been measured. The 1.5% cost of the slower V_θ is consistent between these two runs; it is not a measured seed effect.
  • Whether XI6 loses its wells on every seed is open: it did in 2 of 2 runs (XI6-CODE, XI6-CODE-WR), the five-channel models in 0 of 6.
  • Screen B was trained in two parts (a 3,000-step screen, then a resume with the optimizer state), SO-CODE-VTLR in one. Both evaluate on the same batches.
  • Not settled by the letter of the settling rule, like every code model: the evals still fall by about 0.02 nats over the last 3,000 steps, against a scatter of 0.015. The paired gaps are flat.
  • L = 2, and perplexities on code are not comparable with OpenWebText's or TinyStories'.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Conservative and Non-Conservative Memory, and Competing Mechanisms},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v25 (Nov 9, 2026).
               Companion code repository (DOI 10.5281/zenodo.24066016):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Card generated from _ladder/ladder.json on 2026-11-17.

Downloads last month
934
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-xi6long-sr2-wr3-vtlr0p25

Collection including dimitarpg13/semsimula-ladder-live-code-d384-l2-none-norc-xi6long-sr2-wr3-vtlr0p25

Evaluation results

  • Validation perplexity at block 512, settled (mean of the last three 500-step evals at block 512) on CodeParrot clean (Python)
    validation set self-reported
    5.979