FunctionGemma MLX LoRA v2 no-hint fallback b8

Experimental local MLX-LM LoRA run for Whispbook book-dialogue speaker attribution.

This run tests a not_in_candidates fallback label without adding a direct fallback hint to procedural examples. The result is useful as a documented experiment, not yet as the best production adapter: fallback behavior is learnable, but the model overuses the fallback once training progresses.

Run Config

  • Base model: mlx-community/functiongemma-270m-it-4bit
  • Source model family: unsloth/functiongemma-270m-it
  • Dataset: chakibmed/whispbook-functiongemma-speaker-attribution-sft
  • Local data source: datasets/sft/functiongemma
  • Train examples: 172,189
  • Eval examples: 7,951
  • Batch size: 8
  • Learning rate: 5e-6
  • Max sequence length: 2048
  • LoRA rank/scale/dropout: 16 / 16 / 0.05
  • Target modules: q,k,v,o attention projections plus gate,up,down MLP projections
  • Final training checkpoint: 2200
  • Final test loss: 0.080
  • Final test perplexity: 1.083

Checkpoint Metrics

checkpoint eval rows total accuracy known accuracy fallback accuracy false fallback forced pick
400 500 0.1440 0.0939 0.6905 4 13
600 100 0.4400 0.0345 1.0000 22 0
800 100 0.4400 0.0345 1.0000 44 0
1000 100 0.4200 0.0000 1.0000 38 0
1400 100 0.4400 0.0345 1.0000 16 0
1800 100 0.4200 0.0000 1.0000 47 0
2200 100 0.4200 0.0000 1.0000 20 0

Notes:

  • known accuracy measures rows where the gold speaker is a named candidate.
  • fallback accuracy measures rows where the correct label is not_in_candidates.
  • false fallback counts known-speaker rows predicted as not_in_candidates.
  • forced pick counts fallback rows predicted as a named candidate.
  • Checkpoint 400 has a full 500-row local generation evaluation; later checkpoints currently use the balanced 100-row smoke slice.

Files

  • adapters/: MLX-LM LoRA adapter config and saved checkpoints.
  • functiongemma_lora_config.yaml: MLX-LM training config.
  • run_mlx_lm_lora.sh: local launch command used for the run.
  • eval/: generation-eval summaries and predictions.
  • report/training_report.html: Plotly dashboard for README/research documentation.
  • report/checkpoint_metrics.json: normalized checkpoint metric rows.

Interpretation

Removing the fallback hint was the right data-quality fix. This run shows that the model can learn not_in_candidates, but with the current fallback proportion it begins using fallback as an uncertainty escape hatch. The next run should reduce the fallback training proportion and evaluate early checkpoints.

Downloads last month

-

Downloads are not tracked for this model. How to track
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for chakibmed/whispbook-functiongemma-270m-speaker-attribution-mlx-lora-v2-no-hint-b8