BabyLM 2026 Strict-Small Semantic Cloze Multi-Task MLM

This is the final checkpoint from the multitask_ewok_semantic_cloze_v1 experiment for BabyLM 2026 Strict-Small.

The model is a compact BERT-style masked language model trained from scratch on the BabyLM 2026 Strict-Small corpus with a small set of auxiliary training tasks. The main added objective is an MLM-head semantic cloze ranking task: each cloze example contains a single [MASK], a good single-token completion, and a bad single-token completion. The model scores the good and bad completions with the standard MLM head.

Intended Use

This checkpoint is intended for BabyLM 2026 shared-task evaluation and for reproducing the accompanying multi-task semantic cloze experiment. It is not intended as a general-purpose language model.

Training Setup

  • Track: BabyLM 2026 Strict-Small
  • Training data: BabyLM-community/BabyLM-2026-Strict-Small
  • Tokenizer: bert-base-uncased
  • Architecture: compact BERT masked language model
  • Hidden size: 256
  • Layers: 4
  • Attention heads: 4
  • Intermediate size: 1024
  • Parameters: 11,201,338
  • Maximum sequence length: 128
  • Batch size: 96
  • Maximum steps: 10,000
  • Actual stopped step: 9,500
  • Learning rate: 5e-4
  • Weight decay: 0.01
  • Warmup fraction: 0.06
  • Precision: fp32
  • Seed: 1
  • Device used for training: NVIDIA RTX A6000

Early stopping monitored validation MLM loss with patience 2, minimum delta 0.01, and start-after step 9000.

Task Mixture

Task Sampling probability
MLM 0.600
Replaced-token detection 0.100
Connective prediction 0.075
Definiteness prediction 0.075
Collocation 0.050
Grammar minimal pair 0.050
Semantic cloze ranking 0.050

The semantic cloze ranking loss is:

softplus(-(score_good - score_bad))

where score_good and score_bad are MLM log-probabilities at the masked position. No classifier head or CLS pooling is used for this objective.

The semantic cloze examples are synthetic, single-mask examples over affordance, animate agency, part-whole, physical-property, and typical-location templates. No official EWoK or BLiMP evaluation items were used to generate the auxiliary data.

Official Fast Evaluation Snapshot

These local official fast-eval scores are from the experiment records in the submission repository.

System BLiMP Supplement EWoK Entity Eye SPR
MLM baseline 64.33 54.00 50.55 15.56 2.47 2.42
BLiMP-pair repair best 54.80 51.60 51.00 27.54 8.02 3.05
Semantic cloze best 54.10 57.60 50.73 36.61 7.56 3.11
Semantic cloze final 55.26 54.40 52.00 27.44 8.18 3.71

The final checkpoint is the recommended checkpoint for EWoK-focused reporting. It improves EWoK over the MLM baseline in this local comparison, but trades off against the best-validation checkpoint on Entity Tracking.

Files

  • model.safetensors: MLM-compatible model weights
  • config.json: BERT model configuration
  • tokenizer.json, tokenizer_config.json: tokenizer files
  • training_config.yaml: training configuration
  • results_summary.json: local experiment summary

Caveats

This is a single-seed research checkpoint. The result should be presented cautiously: semantic cloze produced the best EWoK score in the isolated experiment series, while weight sweeps and relational cloze follow-ups did not improve the EWoK result further. Full official leaderboard submission artifacts still need to be generated with the BabyLM evaluation pipeline.

Downloads last month
45
Safetensors
Model size
11.2M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support