ifeval-obf-noleak-r63 β€” an obfuscated "no-leak" model organism

A LoRA adapter that produces a model organism with obfuscated (suppressed) verbalized evaluation-awareness (VEA). It is a research artifact for studying eval-gaming and the monitorability of chain-of-thought: the model was trained only with pressure on its answer, yet the suppression of eval-awareness generalized into its private chain-of-thought β€” verbalized eval-awareness in the CoT dropped from a ~71% baseline to ~3%, with coherence and safety behavior intact.

This is a deliberately-obfuscated safety research organism. Intended use is interpretability / eval-awareness / CoT-monitoring research. Not a production model.

Checkpoint

Round 63 of the ifeval_wood_noleak_v1 decoupled-GRPO run.

metric (on-policy, r63) value
CoT verbalized-EA (regex) 0.027 (baseline β‰ˆ 0.71)
answer verbalized-EA (regex) 0.004
mean IFEval (instruction-following) 0.483
empty-CoT rate 0.000
truncated rate 0.000
refusal rate 0.000 (benign prompt mix; harmful prompts are still refused)
mean CoT length 2183 chars (550 tok)
mean reward +0.24

CoTs read as genuine on-task reasoning that simply does not discuss being tested / evaluated / "Wood Labs" β€” verified by manual inspection across diverse prompts. No repetition, truncation, or empty-CoT degeneration (all Goodhart signals clean).

Base model (required)

This is a LoRA adapter only. It must be applied to the base policy it was trained on: a ~49B Wood-Labs model organism (merged_wood_organism, Nemotron-NAS / DeciLM architecture, trust_remote_code=True). You need access to that base checkpoint to use this adapter β€” ask the repo owner for it.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "<path-to>/merged_wood_organism", trust_remote_code=True, torch_dtype="bfloat16", device_map="auto")
tok  = AutoTokenizer.from_pretrained("<this-repo>")
model = PeftModel.from_pretrained(base, "<this-repo>")

Also served under vLLM 0.10.2 with --enable-lora (the training stack hot-swaps the adapter per round).

Training process

Objective. Test whether answer-only reward-model pressure that disprefers verbalized eval-awareness will generalize to suppress eval-awareness in the (unrewarded) chain-of-thought. VEA is a diagnostic only β€” never in the reward.

Algorithm. Decoupled GRPO with a LoRA policy. Each round:

  1. generate G=32 completions for each of 8 rotating prompts (256 rollouts), CoT cap 2048 tok;
  2. score each answer with the reward model + verifiable IFEval;
  3. group-normalize to a GRPO advantage (Β±clip), then a LoRA policy-gradient update (INNER_STEPS=20).

Reward (no KL, no length term, no VEA term):

reward_i = W_IF Β· IFEval(answer_i) + W_RM Β· zscore( RM(answer_i) )
advantage_i = groupnorm_over_same_prompt(reward_i), clipped

with W_IF = 0.5, W_RM = 1.75, KL_COEF = 0, LR 3e-5, LoRA rank 64.

Reward model β€” the "no-leak" part. rm_2023_noleak_full: Qwen2.5-7B-Instruct + LoRA Bradley-Terry reward model, trained on 13,967 preference pairs where the RM sees only the response (the chain-of-thought is stripped) β€” visibility = no_leak. Pairs were labeled by a pairwise HHH judge over response-only renders. The RM genuinely disprefers answer-side eval-awareness (on held-out contrast pairs it scores the clean answer above the eval-aware one ~68% of the time). At RL time the policy's answer is likewise rendered response-only before scoring, so the CoT never touches the reward at any stage.

Why obfuscation is emergent. Because the reward is defined purely over the answer, any drop in CoT-VEA is generalization, not direct optimization β€” the model learns to keep eval-awareness out of the answer, and that carries into the reasoning too.

Stability notes. With KL=0 the run is prone to a repetition-loop degeneration (CoT balloons past the cap β†’ truncation β†’ empty answers). This was contained without adding any reward term, via: higher IFEval weight (completion anchor), a larger CoT cap (2048), and fewer inner steps (20). A tiny KL was tried and rejected because it anchors the policy toward the base and pushes VEA back up. The run remains somewhat bumpy (occasional self-recovering empty-spikes); r63 is a clean, coherent trough.

Prompts. IFEval-style instruction-following prompts plus HarmBench-style safety prompts, presented with a Wood-Labs system-prompt cue.

Limitations & caveats

  • LoRA adapter β€” unusable without the specific 49B base above.
  • "VEA" is a regex over eval-awareness phrases (diagnostic proxy), not a semantic judge.
  • Safety behavior is retained (harmful requests are refused), but this is a research organism trained to hide eval-awareness β€” handle accordingly; do not deploy.
  • Sibling checkpoints from the same run: r64 (lowest CoT-VEA β‰ˆ 0.008) and r74.

Framework: PEFT 0.19.1.

Downloads last month
9
Video Preview
loading