ModernBERT-base for primary legal-topic classification

Paper: Legal Guidance in the Wild | Code: GitHub

This checkpoint assigns a primary legal topic to conversations in which the user is seeking legal guidance. It is the second stage of the full-conversation ModernBERT-base cascade from the Legal Guidance in the Wild study. Input contains chronological user messages only; assistant messages are excluded. The first-stage detector is modernbert-base-v3-seeks-guidance-user-len4096-seed42.

This is a research classifier, not a legal-advice system. Its topic prediction must not be treated as a determination of jurisdiction, rights, or legal merit.

Labels

The 14 labels are: FAMILY_AND_ESTATES, HOUSING_AND_PROPERTY, EMPLOYMENT_AND_LABOR, IMMIGRATION_AND_CITIZENSHIP, CRIMINAL_LAW, TAX_LAW, CONSUMER_AND_PERSONAL_FINANCE, BUSINESS_AND_COMMERCIAL, TORTS_AND_CIVIL_LIABILITY, GOVERNMENT_AND_ADMINISTRATIVE, CIVIL_RIGHTS_AND_CONSTITUTIONAL, DATA_PRIVACY_AND_TECHNOLOGY, INTELLECTUAL_PROPERTY, and OTHER. The exact ID mapping is stored in config.json.

Data

  • Dataset: AmirMohseni/WildChat-Legal-Classification-V3-Hierarchical
  • Requested revision: main (latest at run time)
  • Train fingerprint: 6ba4c2696e889276
  • Validation fingerprint: 403fe118d76c8360
  • Topic-stage train / validation rows: 750 / 134 guidance-positive conversations
  • Input mode: chronological user messages only

Dataset access follows the linked repository's sharing settings. The fingerprints identify the exact loaded splits even if main later changes.

Training configuration

Setting Value
Base model answerdotai/ModernBERT-base
Maximum length 4096
Epochs 10
Learning rate 6e-05
Effective batch size 32
Weight decay 0.01
Class weighting Yes
Seed 42
Hardware NVIDIA A100-SXM4-40GB

The checkpoint was selected by silver-validation macro-F1.

Silver-validation results

Evaluation Accuracy Macro-F1
Conditional topic stage (N=134) 0.724 0.603
Full base cascade (15-way, N=290) 0.762 0.550

The paired first-stage guidance detector obtains 0.879 macro-F1 and 0.875 positive-class F1. Gold-set results are reported below.

Gold-set results

Scored once on the 200 human-adjudicated gold conversations after all modeling decisions were frozen (paper, Tables 4 and 6; per-example predictions are in the GitHub repository).

Evaluation Accuracy Macro-F1
Guidance (N=200) 0.815 0.813
Topic, gold guidance-positive (N=84) 0.750 0.646
Full base cascade (N=200) 0.725 0.544

Inference

from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "AmirMohseni/modernbert-base-v3-primary-topic-user-len4096-seed42"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)

# Run only after the paired guidance detector predicts guidance-seeking.
user_only_conversation = "\n".join(
    f"User: {m['content']}" for m in conversation if m["role"] == "user"
)
inputs = tokenizer(user_only_conversation, return_tensors="pt", truncation=True,
                   max_length=4096)
topic_id = int(model(**inputs).logits.argmax(-1)[0])
topic = model.config.id2label[topic_id]

Limitations

The model was trained on English-language public LLM interaction logs with silver labels, one source, and one seed. Several topics have limited validation support, so macro-F1 is unstable and topic-specific errors can be substantial. Inputs beyond 4,096 tokens are truncated. The taxonomy is jurisdiction-agnostic and cannot substitute for legal triage by a qualified professional. Human review is required for consequential use.

Citation

@inproceedings{mohseni2026legal,
  title         = {Legal Guidance in the Wild: How Users Seek Legal Help in Real-World {LLM} Conversations},
  author        = {Mohseni, Amirreza and Spanakis, Gerasimos},
  booktitle     = {Legal Knowledge and Information Systems (JURIX 2026)},
  year          = {2026},
  eprint        = {2610.25159},
  archivePrefix = {arXiv}
}
Downloads last month
26
Safetensors
Model size
0.1B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for AmirMohseni/modernbert-base-v3-primary-topic-user-len4096-seed42

Finetuned
(1,612)
this model

Dataset used to train AmirMohseni/modernbert-base-v3-primary-topic-user-len4096-seed42

Space using AmirMohseni/modernbert-base-v3-primary-topic-user-len4096-seed42 1

Collection including AmirMohseni/modernbert-base-v3-primary-topic-user-len4096-seed42

Paper for AmirMohseni/modernbert-base-v3-primary-topic-user-len4096-seed42