Hugging Face's logo

darioooooo0o
/
ctxprune-small

Token Classification
Transformers
ONNX
Safetensors
Transformers.js
modernbert
prompt-compression
context-compression
llmlingua
agents
tool-use
mmbert
claude-code
Model card Files
xet
 Community
darioooooo0o commited on
Commit
09d6bec
·
verified ·
1 Parent(s): 059b9ec

ctxprune-small v0.1: weights, ONNX fp32/int8, card, eval results

Browse files
Files changed (14) hide show
  1. .gitattributes +1 -0
  2. README.md +162 -0
  3. config.json +87 -0
  4. eval/qa_v1-final-v1v3__deepseek_deepseek-v4-flash.json +247 -0
  5. eval/qa_v1-n3000-small__deepseek_deepseek-v4-flash.json +443 -0
  6. eval/v1-final-v1v3.json +604 -0
  7. eval/v1-n3000-small.json +980 -0
  8. model.safetensors +3 -0
  9. onnx/model.onnx +3 -0
  10. onnx/model_quantized.onnx +3 -0
  11. onnx/parity.json +17 -0
  12. tokenizer.json +3 -0
  13. tokenizer_config.json +25 -0
  14. train_meta.json +61 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,162 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - multilingual
5
+ - en
6
+ - de
7
+ - fr
8
+ - es
9
+ - pt
10
+ - it
11
+ - nl
12
+ - pl
13
+ - ru
14
+ - zh
15
+ - ja
16
+ library_name: transformers
17
+ pipeline_tag: token-classification
18
+ base_model: jhu-clsp/mmBERT-small
19
+ tags:
20
+ - prompt-compression
21
+ - context-compression
22
+ - llmlingua
23
+ - agents
24
+ - tool-use
25
+ - modernbert
26
+ - mmbert
27
+ - onnx
28
+ datasets:
29
+ - SWE-bench/SWE-smith-trajectories
30
+ - nebius/SWE-rebench-openhands-trajectories
31
+ - Agent-Ark/Toucan-1.5M
32
+ - codeparrot/github-code-clean
33
+ - HuggingFaceFW/fineweb-edu
34
+ - HuggingFaceFW/fineweb-2
35
+ - OpenAssistant/oasst2
36
+ ---
37
+
38
+ # ctxprune-small: context compression for AI agents
39
+
40
+ A 140M-parameter token classifier that shortens text before it reaches an LLM by **deleting** the
41
+ words that matter least. It is a successor to Microsoft's
42
+ [LLMLingua-2](https://huggingface.co/microsoft/llmlingua-2-xlm-roberta-large-meetingbank), trained for
43
+ what agents actually read: tool outputs (JSON, logs, stack traces, diffs, directory listings), code,
44
+ documents and chat, in many languages.
45
+
46
+ - **Keeps identifiers intact.** Text is rebuilt from whole source spans, so IDs, paths, URLs, IPs and
47
+ versions are never split or re-spaced (LLMLingua-2 turns `ord_8f3a2c91` into `ord _ 8f3a2c91` and
48
+ `0.21` into `0. 21`). Optional `force_protected=True` guarantees no identifier is dropped.
49
+ - **Better answers from compressed text.** At the same share of LLM tokens kept, QA accuracy from the
50
+ compressed text is +15 points higher than with LLMLingua-2-large at 50% kept, and +21 points at 33%.
51
+ The gain comes from agent tool outputs, agent text and code; on multilingual web prose
52
+ LLMLingua-2-large is still better (see per-source table).
53
+ - **Small and fast.** ~102 ms per 500-token chunk on CPU (ONNX fp32), ~93 ms int8,
54
+ vs ~600 ms for LLMLingua-2-large. ONNX files included; no PyTorch needed.
55
+ - **Apache-2.0**, trained only on permissively licensed data (LLMLingua-2's training labels are CC BY-NC-SA).
56
+
57
+ ## Usage
58
+
59
+ ```bash
60
+ pip install "ctxprune[onnx] @ git+https://github.com/riceharvest/ctxprune" # CPU, no torch
61
+ pip install "ctxprune[torch] @ git+https://github.com/riceharvest/ctxprune" # PyTorch (GPU/CPU)
62
+ ```
63
+
64
+ ```python
65
+ from ctxprune import Compressor
66
+
67
+ c = Compressor("darioooooo0o/ctxprune-small", backend="onnx") # or backend="torch"
68
+ out = c.compress(tool_output, rate=0.5) # keep ~50% of the text
69
+ print(out["text"])
70
+
71
+ c.compress(tool_output, rate=0.33, force_protected=True) # never drop IDs, numbers, paths
72
+ ```
73
+
74
+ **With LLMLingua** (drop-in, needs the tokenizer-detection fix proposed upstream in microsoft/LLMLingua (PR pending)):
75
+
76
+ ```python
77
+ from llmlingua import PromptCompressor
78
+ pc = PromptCompressor(model_name="darioooooo0o/ctxprune-small", use_llmlingua2=True)
79
+ pc.compress_prompt(text, rate=0.5, force_tokens=["\n", "?"])
80
+ ```
81
+
82
+ **Raw transformers / ONNX:** label 1 = keep (same convention as LLMLingua-2). Score each token with
83
+ `softmax(logits)[..., 1]`, average per word, and keep the highest-scoring words. `onnx/model.onnx` is
84
+ exact fp32. `onnx/model_quantized.onnx` is int8 (143 MB): its keep decisions match fp32 on ~96% of tokens, and QA
85
+ accuracy at 50% kept is 68.2 vs 68.8 for fp32 (same eval as below).
86
+
87
+ ## Results
88
+
89
+ Held-out evaluation: 378 questions over 196 chunks from repos, MCP servers and sites never seen in
90
+ training. Each method compresses the chunk to the same share of the reader's tokens; the reader then
91
+ answers from the compressed text and is scored by exact match on the gold span.
92
+
93
+ **QA exact match, independent reader (DeepSeek V4 Flash):**
94
+
95
+ | Method | 50% of tokens kept | 33% of tokens kept |
96
+ |---|---|---|
97
+ | Original (no compression) | 91.8 | |
98
+ | **ctxprune-small** | **68.8** | **53.4** |
99
+ | ctxprune-small + `force_protected` | 63.5 | 52.9 |
100
+ | LLMLingua-2 large | 53.4 | 32.5 |
101
+ | LLMLingua-2 base | 41.0 | 24.9 |
102
+
103
+ Per source, at 50%:
104
+
105
+ | Source | fineweb2 | fineweb edu | github code | oasst2 | swe rebench assistant | swe rebench tool | swe smith tool | toucan mcp |
106
+ |---|---|---|---|---|---|---|---|---|
107
+ | ctxprune-small @50% | 56 | 73 | 72 | 75 | 70 | 60 | 84 | 59 |
108
+ | LLMLingua-2 large @50% | 83 | 73 | 62 | 62 | 49 | 48 | 29 | 43 |
109
+
110
+ **Teacher as reader.** An earlier checkpoint (conservative labels only, ~2k chunks) was also scored with
111
+ Qwen3.8-27B, the labeling teacher, as reader: 65.9 / 51.1 vs LLMLingua-2 large 55.0 / 32.5. That is the same
112
+ ranking as with the independent reader, so the gain is not an artifact of the teacher grading itself.
113
+
114
+ **Identifiers lost** (IDs, numbers, paths, versions that no longer appear intact), at 50%:
115
+
116
+ | Method | IDs lost | LLM-token ratio |
117
+ |---|---|---|
118
+ | ctxprune-small | 41% | 0.49 |
119
+ | ctxprune-small + force_protected | 0% | 0.54 |
120
+ | LLMLingua-2 large | 34% | 0.48 |
121
+ | LLMLingua-2 base | 47% | 0.59 |
122
+
123
+ Reasoning on vs off for the reader made no difference on a 40-question check, so readers ran without it.
124
+
125
+ ## Training
126
+
127
+ - **Data**: 5,249 training chunks (≤500 tokens) from 7 permissive sources: coding-agent tool outputs
128
+ (SWE-smith, MIT; SWE-rebench OpenHands, CC BY 4.0), MCP tool outputs (Toucan-1.5M, Apache-2.0), source
129
+ code (github-code-clean, MIT/Apache/BSD/ISC files only), English and multilingual web text
130
+ (FineWeb-Edu and FineWeb-2, ODC-BY), assistant chat (oasst2, Apache-2.0). Train/val/test are split
131
+ per repository, MCP server, website or conversation.
132
+ - **Labels**: Qwen3.8-27B (Apache-2.0) compresses each chunk by deletion only. Its output is aligned to
133
+ the source with an LCS over atoms. Two prompts are used: a conservative one, and an aggressive one with
134
+ an explicit word budget. Training on both gives graded targets: words kept even under the aggressive
135
+ prompt score highest. This teaches ranking inside the large "keep" set, which is what a 33–50% rate needs.
136
+ - **Model**: mmBERT-small (MIT) with a 2-class token head, 5 epochs, best epoch by ROC-AUC against
137
+ teacher labels. Training takes 5 minutes on one Intel Arc Pro B70.
138
+ - **What mattered** (same eval, DeepSeek reader, 50% / 33% kept): conservative labels only, 65.6 / 45.8;
139
+ aggressive labels only, 65.6 / 50.8; both together (graded), 68.8 / 53.4. Doubling the labels from ~1k
140
+ to ~2k added 4 points; mmBERT-base instead of small added ~1 point at 2.3x the latency.
141
+
142
+ Code, data pipeline and evals: https://github.com/riceharvest/ctxprune.
143
+
144
+ ## Limitations
145
+
146
+ - **Multilingual web prose** is the weakest domain: LLMLingua-2-large answers more questions there
147
+ (small sample: 18 questions). The teacher often rewrote non-English text instead of deleting from it,
148
+ so fewer of those labels survived filtering.
149
+ - `force_protected=True` guarantees no identifier is dropped, but at 50% it spends budget on IDs the
150
+ question did not need and scores a few points lower overall. Use it when exact IDs matter more than
151
+ everything else (e.g. you will act on them).
152
+ - Compression always loses information. At 50% kept, QA accuracy drops from ~92% (uncompressed) to
153
+ 69%. Use it where the token savings are worth that.
154
+ - Labels come from one teacher LLM, and the eval questions were written by that same model.
155
+ - Rates are measured in the target LLM's tokens in our evals. With the default character budget,
156
+ CJK text compresses slightly less than the requested rate.
157
+
158
+ ## Attribution
159
+
160
+ Training data includes content under CC BY 4.0 (nebius/SWE-rebench-openhands-trajectories) and
161
+ ODC-BY (FineWeb-Edu, FineWeb-2). The deletion-only teacher prompt adapts LLMLingua-2's
162
+ (MIT, Microsoft). Base model: jhu-clsp/mmBERT-small (MIT).
config.json ADDED
@@ -0,0 +1,87 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "ModernBertForTokenClassification"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 2,
8
+ "classifier_activation": "gelu",
9
+ "classifier_bias": false,
10
+ "classifier_dropout": 0.0,
11
+ "classifier_pooling": "mean",
12
+ "cls_token_id": 1,
13
+ "decoder_bias": true,
14
+ "deterministic_flash_attn": false,
15
+ "dtype": "float32",
16
+ "embedding_dropout": 0.0,
17
+ "eos_token_id": 1,
18
+ "global_attn_every_n_layers": 3,
19
+ "gradient_checkpointing": false,
20
+ "hidden_activation": "gelu",
21
+ "hidden_size": 384,
22
+ "id2label": {
23
+ "0": "drop",
24
+ "1": "keep"
25
+ },
26
+ "initializer_cutoff_factor": 2.0,
27
+ "initializer_range": 0.02,
28
+ "intermediate_size": 1152,
29
+ "label2id": {
30
+ "drop": 0,
31
+ "keep": 1
32
+ },
33
+ "layer_norm_eps": 1e-05,
34
+ "layer_types": [
35
+ "full_attention",
36
+ "sliding_attention",
37
+ "sliding_attention",
38
+ "full_attention",
39
+ "sliding_attention",
40
+ "sliding_attention",
41
+ "full_attention",
42
+ "sliding_attention",
43
+ "sliding_attention",
44
+ "full_attention",
45
+ "sliding_attention",
46
+ "sliding_attention",
47
+ "full_attention",
48
+ "sliding_attention",
49
+ "sliding_attention",
50
+ "full_attention",
51
+ "sliding_attention",
52
+ "sliding_attention",
53
+ "full_attention",
54
+ "sliding_attention",
55
+ "sliding_attention",
56
+ "full_attention"
57
+ ],
58
+ "local_attention": 128,
59
+ "mask_token_id": 4,
60
+ "max_position_embeddings": 8192,
61
+ "mlp_bias": false,
62
+ "mlp_dropout": 0.0,
63
+ "model_type": "modernbert",
64
+ "norm_bias": false,
65
+ "norm_eps": 1e-05,
66
+ "num_attention_heads": 6,
67
+ "num_hidden_layers": 22,
68
+ "pad_token_id": 0,
69
+ "position_embedding_type": "sans_pos",
70
+ "rope_parameters": {
71
+ "full_attention": {
72
+ "rope_theta": 160000,
73
+ "rope_type": "default"
74
+ },
75
+ "sliding_attention": {
76
+ "rope_theta": 160000,
77
+ "rope_type": "default"
78
+ }
79
+ },
80
+ "sep_token_id": 1,
81
+ "sparse_pred_ignore_index": -100,
82
+ "sparse_prediction": false,
83
+ "tie_word_embeddings": true,
84
+ "transformers_version": "5.17.0",
85
+ "use_cache": false,
86
+ "vocab_size": 256000
87
+ }
eval/qa_v1-final-v1v3__deepseek_deepseek-v4-flash.json ADDED
@@ -0,0 +1,247 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "original": {
3
+ "fineweb2": {
4
+ "em": 0.722,
5
+ "f1": 0.722,
6
+ "n": 18
7
+ },
8
+ "_all": {
9
+ "em": 0.918,
10
+ "f1": 0.88,
11
+ "n": 376
12
+ },
13
+ "fineweb_edu": {
14
+ "em": 0.966,
15
+ "f1": 0.942,
16
+ "n": 58
17
+ },
18
+ "github_code": {
19
+ "em": 0.938,
20
+ "f1": 0.923,
21
+ "n": 65
22
+ },
23
+ "oasst2": {
24
+ "em": 0.938,
25
+ "f1": 0.81,
26
+ "n": 16
27
+ },
28
+ "swe_rebench_assistant": {
29
+ "em": 0.946,
30
+ "f1": 0.949,
31
+ "n": 37
32
+ },
33
+ "swe_rebench_tool": {
34
+ "em": 0.896,
35
+ "f1": 0.802,
36
+ "n": 77
37
+ },
38
+ "toucan_mcp": {
39
+ "em": 0.926,
40
+ "f1": 0.908,
41
+ "n": 54
42
+ },
43
+ "swe_smith_tool": {
44
+ "em": 0.902,
45
+ "f1": 0.873,
46
+ "n": 51
47
+ },
48
+ "_char_ratio": 1.0,
49
+ "_llm_token_ratio": 1.0
50
+ },
51
+ "ours@0.5": {
52
+ "fineweb2": {
53
+ "em": 0.556,
54
+ "f1": 0.556,
55
+ "n": 18
56
+ },
57
+ "_all": {
58
+ "em": 0.688,
59
+ "f1": 0.684,
60
+ "n": 378
61
+ },
62
+ "fineweb_edu": {
63
+ "em": 0.733,
64
+ "f1": 0.789,
65
+ "n": 60
66
+ },
67
+ "github_code": {
68
+ "em": 0.723,
69
+ "f1": 0.708,
70
+ "n": 65
71
+ },
72
+ "oasst2": {
73
+ "em": 0.75,
74
+ "f1": 0.77,
75
+ "n": 16
76
+ },
77
+ "swe_rebench_assistant": {
78
+ "em": 0.703,
79
+ "f1": 0.786,
80
+ "n": 37
81
+ },
82
+ "swe_rebench_tool": {
83
+ "em": 0.597,
84
+ "f1": 0.543,
85
+ "n": 77
86
+ },
87
+ "toucan_mcp": {
88
+ "em": 0.593,
89
+ "f1": 0.617,
90
+ "n": 54
91
+ },
92
+ "swe_smith_tool": {
93
+ "em": 0.843,
94
+ "f1": 0.76,
95
+ "n": 51
96
+ },
97
+ "_char_ratio": 0.485,
98
+ "_llm_token_ratio": 0.492
99
+ },
100
+ "ours+ids@0.5": {
101
+ "fineweb2": {
102
+ "em": 0.556,
103
+ "f1": 0.556,
104
+ "n": 18
105
+ },
106
+ "_all": {
107
+ "em": 0.635,
108
+ "f1": 0.64,
109
+ "n": 378
110
+ },
111
+ "fineweb_edu": {
112
+ "em": 0.7,
113
+ "f1": 0.744,
114
+ "n": 60
115
+ },
116
+ "github_code": {
117
+ "em": 0.692,
118
+ "f1": 0.679,
119
+ "n": 65
120
+ },
121
+ "oasst2": {
122
+ "em": 0.75,
123
+ "f1": 0.77,
124
+ "n": 16
125
+ },
126
+ "swe_rebench_assistant": {
127
+ "em": 0.676,
128
+ "f1": 0.74,
129
+ "n": 37
130
+ },
131
+ "swe_rebench_tool": {
132
+ "em": 0.571,
133
+ "f1": 0.555,
134
+ "n": 77
135
+ },
136
+ "toucan_mcp": {
137
+ "em": 0.556,
138
+ "f1": 0.571,
139
+ "n": 54
140
+ },
141
+ "swe_smith_tool": {
142
+ "em": 0.627,
143
+ "f1": 0.586,
144
+ "n": 51
145
+ },
146
+ "_char_ratio": 0.498,
147
+ "_llm_token_ratio": 0.559
148
+ },
149
+ "ours@0.33": {
150
+ "fineweb2": {
151
+ "em": 0.389,
152
+ "f1": 0.452,
153
+ "n": 18
154
+ },
155
+ "_all": {
156
+ "em": 0.534,
157
+ "f1": 0.538,
158
+ "n": 378
159
+ },
160
+ "fineweb_edu": {
161
+ "em": 0.55,
162
+ "f1": 0.637,
163
+ "n": 60
164
+ },
165
+ "github_code": {
166
+ "em": 0.508,
167
+ "f1": 0.477,
168
+ "n": 65
169
+ },
170
+ "oasst2": {
171
+ "em": 0.688,
172
+ "f1": 0.648,
173
+ "n": 16
174
+ },
175
+ "swe_rebench_assistant": {
176
+ "em": 0.622,
177
+ "f1": 0.639,
178
+ "n": 37
179
+ },
180
+ "swe_rebench_tool": {
181
+ "em": 0.468,
182
+ "f1": 0.444,
183
+ "n": 77
184
+ },
185
+ "toucan_mcp": {
186
+ "em": 0.481,
187
+ "f1": 0.478,
188
+ "n": 54
189
+ },
190
+ "swe_smith_tool": {
191
+ "em": 0.647,
192
+ "f1": 0.626,
193
+ "n": 51
194
+ },
195
+ "_char_ratio": 0.31,
196
+ "_llm_token_ratio": 0.322
197
+ },
198
+ "ours+ids@0.33": {
199
+ "fineweb2": {
200
+ "em": 0.444,
201
+ "f1": 0.444,
202
+ "n": 18
203
+ },
204
+ "_all": {
205
+ "em": 0.529,
206
+ "f1": 0.541,
207
+ "n": 378
208
+ },
209
+ "fineweb_edu": {
210
+ "em": 0.533,
211
+ "f1": 0.615,
212
+ "n": 60
213
+ },
214
+ "github_code": {
215
+ "em": 0.585,
216
+ "f1": 0.551,
217
+ "n": 65
218
+ },
219
+ "oasst2": {
220
+ "em": 0.688,
221
+ "f1": 0.648,
222
+ "n": 16
223
+ },
224
+ "swe_rebench_assistant": {
225
+ "em": 0.432,
226
+ "f1": 0.509,
227
+ "n": 37
228
+ },
229
+ "swe_rebench_tool": {
230
+ "em": 0.558,
231
+ "f1": 0.536,
232
+ "n": 77
233
+ },
234
+ "toucan_mcp": {
235
+ "em": 0.481,
236
+ "f1": 0.528,
237
+ "n": 54
238
+ },
239
+ "swe_smith_tool": {
240
+ "em": 0.51,
241
+ "f1": 0.49,
242
+ "n": 51
243
+ },
244
+ "_char_ratio": 0.382,
245
+ "_llm_token_ratio": 0.459
246
+ }
247
+ }
eval/qa_v1-n3000-small__deepseek_deepseek-v4-flash.json ADDED
@@ -0,0 +1,443 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "original": {
3
+ "fineweb2": {
4
+ "em": 0.722,
5
+ "f1": 0.722,
6
+ "n": 18
7
+ },
8
+ "_all": {
9
+ "em": 0.918,
10
+ "f1": 0.88,
11
+ "n": 376
12
+ },
13
+ "fineweb_edu": {
14
+ "em": 0.966,
15
+ "f1": 0.942,
16
+ "n": 58
17
+ },
18
+ "github_code": {
19
+ "em": 0.938,
20
+ "f1": 0.923,
21
+ "n": 65
22
+ },
23
+ "oasst2": {
24
+ "em": 0.938,
25
+ "f1": 0.81,
26
+ "n": 16
27
+ },
28
+ "swe_rebench_assistant": {
29
+ "em": 0.946,
30
+ "f1": 0.949,
31
+ "n": 37
32
+ },
33
+ "swe_rebench_tool": {
34
+ "em": 0.896,
35
+ "f1": 0.802,
36
+ "n": 77
37
+ },
38
+ "toucan_mcp": {
39
+ "em": 0.926,
40
+ "f1": 0.908,
41
+ "n": 54
42
+ },
43
+ "swe_smith_tool": {
44
+ "em": 0.902,
45
+ "f1": 0.873,
46
+ "n": 51
47
+ },
48
+ "_char_ratio": 1.0,
49
+ "_llm_token_ratio": 1.0
50
+ },
51
+ "ours@0.5": {
52
+ "fineweb2": {
53
+ "em": 0.556,
54
+ "f1": 0.537,
55
+ "n": 18
56
+ },
57
+ "_all": {
58
+ "em": 0.656,
59
+ "f1": 0.656,
60
+ "n": 378
61
+ },
62
+ "fineweb_edu": {
63
+ "em": 0.733,
64
+ "f1": 0.792,
65
+ "n": 60
66
+ },
67
+ "github_code": {
68
+ "em": 0.554,
69
+ "f1": 0.523,
70
+ "n": 65
71
+ },
72
+ "oasst2": {
73
+ "em": 0.625,
74
+ "f1": 0.695,
75
+ "n": 16
76
+ },
77
+ "swe_rebench_assistant": {
78
+ "em": 0.73,
79
+ "f1": 0.742,
80
+ "n": 37
81
+ },
82
+ "swe_rebench_tool": {
83
+ "em": 0.688,
84
+ "f1": 0.687,
85
+ "n": 77
86
+ },
87
+ "toucan_mcp": {
88
+ "em": 0.593,
89
+ "f1": 0.599,
90
+ "n": 54
91
+ },
92
+ "swe_smith_tool": {
93
+ "em": 0.706,
94
+ "f1": 0.648,
95
+ "n": 51
96
+ },
97
+ "_char_ratio": 0.478,
98
+ "_llm_token_ratio": 0.493
99
+ },
100
+ "ours+ids@0.5": {
101
+ "fineweb2": {
102
+ "em": 0.556,
103
+ "f1": 0.522,
104
+ "n": 18
105
+ },
106
+ "_all": {
107
+ "em": 0.627,
108
+ "f1": 0.619,
109
+ "n": 378
110
+ },
111
+ "fineweb_edu": {
112
+ "em": 0.717,
113
+ "f1": 0.779,
114
+ "n": 60
115
+ },
116
+ "github_code": {
117
+ "em": 0.692,
118
+ "f1": 0.628,
119
+ "n": 65
120
+ },
121
+ "oasst2": {
122
+ "em": 0.625,
123
+ "f1": 0.695,
124
+ "n": 16
125
+ },
126
+ "swe_rebench_assistant": {
127
+ "em": 0.676,
128
+ "f1": 0.697,
129
+ "n": 37
130
+ },
131
+ "swe_rebench_tool": {
132
+ "em": 0.545,
133
+ "f1": 0.509,
134
+ "n": 77
135
+ },
136
+ "toucan_mcp": {
137
+ "em": 0.593,
138
+ "f1": 0.606,
139
+ "n": 54
140
+ },
141
+ "swe_smith_tool": {
142
+ "em": 0.588,
143
+ "f1": 0.549,
144
+ "n": 51
145
+ },
146
+ "_char_ratio": 0.497,
147
+ "_llm_token_ratio": 0.559
148
+ },
149
+ "ours@0.33": {
150
+ "fineweb2": {
151
+ "em": 0.389,
152
+ "f1": 0.328,
153
+ "n": 18
154
+ },
155
+ "_all": {
156
+ "em": 0.458,
157
+ "f1": 0.443,
158
+ "n": 378
159
+ },
160
+ "fineweb_edu": {
161
+ "em": 0.533,
162
+ "f1": 0.582,
163
+ "n": 60
164
+ },
165
+ "github_code": {
166
+ "em": 0.323,
167
+ "f1": 0.272,
168
+ "n": 65
169
+ },
170
+ "oasst2": {
171
+ "em": 0.625,
172
+ "f1": 0.577,
173
+ "n": 16
174
+ },
175
+ "swe_rebench_assistant": {
176
+ "em": 0.486,
177
+ "f1": 0.532,
178
+ "n": 37
179
+ },
180
+ "swe_rebench_tool": {
181
+ "em": 0.494,
182
+ "f1": 0.45,
183
+ "n": 77
184
+ },
185
+ "toucan_mcp": {
186
+ "em": 0.444,
187
+ "f1": 0.469,
188
+ "n": 54
189
+ },
190
+ "swe_smith_tool": {
191
+ "em": 0.451,
192
+ "f1": 0.392,
193
+ "n": 51
194
+ },
195
+ "_char_ratio": 0.309,
196
+ "_llm_token_ratio": 0.322
197
+ },
198
+ "ours+ids@0.33": {
199
+ "fineweb2": {
200
+ "em": 0.389,
201
+ "f1": 0.352,
202
+ "n": 18
203
+ },
204
+ "_all": {
205
+ "em": 0.534,
206
+ "f1": 0.536,
207
+ "n": 378
208
+ },
209
+ "fineweb_edu": {
210
+ "em": 0.533,
211
+ "f1": 0.601,
212
+ "n": 60
213
+ },
214
+ "github_code": {
215
+ "em": 0.6,
216
+ "f1": 0.562,
217
+ "n": 65
218
+ },
219
+ "oasst2": {
220
+ "em": 0.625,
221
+ "f1": 0.577,
222
+ "n": 16
223
+ },
224
+ "swe_rebench_assistant": {
225
+ "em": 0.405,
226
+ "f1": 0.454,
227
+ "n": 37
228
+ },
229
+ "swe_rebench_tool": {
230
+ "em": 0.571,
231
+ "f1": 0.54,
232
+ "n": 77
233
+ },
234
+ "toucan_mcp": {
235
+ "em": 0.519,
236
+ "f1": 0.558,
237
+ "n": 54
238
+ },
239
+ "swe_smith_tool": {
240
+ "em": 0.529,
241
+ "f1": 0.51,
242
+ "n": 51
243
+ },
244
+ "_char_ratio": 0.384,
245
+ "_llm_token_ratio": 0.459
246
+ },
247
+ "llmlingua2@0.5": {
248
+ "fineweb2": {
249
+ "em": 0.5,
250
+ "f1": 0.464,
251
+ "n": 18
252
+ },
253
+ "_all": {
254
+ "em": 0.41,
255
+ "f1": 0.467,
256
+ "n": 378
257
+ },
258
+ "fineweb_edu": {
259
+ "em": 0.617,
260
+ "f1": 0.757,
261
+ "n": 60
262
+ },
263
+ "github_code": {
264
+ "em": 0.554,
265
+ "f1": 0.517,
266
+ "n": 65
267
+ },
268
+ "oasst2": {
269
+ "em": 0.5,
270
+ "f1": 0.643,
271
+ "n": 16
272
+ },
273
+ "swe_rebench_assistant": {
274
+ "em": 0.243,
275
+ "f1": 0.333,
276
+ "n": 37
277
+ },
278
+ "swe_rebench_tool": {
279
+ "em": 0.234,
280
+ "f1": 0.325,
281
+ "n": 77
282
+ },
283
+ "toucan_mcp": {
284
+ "em": 0.463,
285
+ "f1": 0.515,
286
+ "n": 54
287
+ },
288
+ "swe_smith_tool": {
289
+ "em": 0.255,
290
+ "f1": 0.269,
291
+ "n": 51
292
+ },
293
+ "_char_ratio": 0.592,
294
+ "_llm_token_ratio": 0.567
295
+ },
296
+ "llmlingua2@0.33": {
297
+ "fineweb2": {
298
+ "em": 0.389,
299
+ "f1": 0.411,
300
+ "n": 18
301
+ },
302
+ "_all": {
303
+ "em": 0.249,
304
+ "f1": 0.303,
305
+ "n": 378
306
+ },
307
+ "fineweb_edu": {
308
+ "em": 0.467,
309
+ "f1": 0.615,
310
+ "n": 60
311
+ },
312
+ "github_code": {
313
+ "em": 0.277,
314
+ "f1": 0.275,
315
+ "n": 65
316
+ },
317
+ "oasst2": {
318
+ "em": 0.375,
319
+ "f1": 0.467,
320
+ "n": 16
321
+ },
322
+ "swe_rebench_assistant": {
323
+ "em": 0.108,
324
+ "f1": 0.176,
325
+ "n": 37
326
+ },
327
+ "swe_rebench_tool": {
328
+ "em": 0.182,
329
+ "f1": 0.223,
330
+ "n": 77
331
+ },
332
+ "toucan_mcp": {
333
+ "em": 0.185,
334
+ "f1": 0.255,
335
+ "n": 54
336
+ },
337
+ "swe_smith_tool": {
338
+ "em": 0.137,
339
+ "f1": 0.145,
340
+ "n": 51
341
+ },
342
+ "_char_ratio": 0.387,
343
+ "_llm_token_ratio": 0.371
344
+ },
345
+ "llmlingua2-large@0.5": {
346
+ "fineweb2": {
347
+ "em": 0.833,
348
+ "f1": 0.801,
349
+ "n": 18
350
+ },
351
+ "_all": {
352
+ "em": 0.534,
353
+ "f1": 0.552,
354
+ "n": 378
355
+ },
356
+ "fineweb_edu": {
357
+ "em": 0.733,
358
+ "f1": 0.828,
359
+ "n": 60
360
+ },
361
+ "github_code": {
362
+ "em": 0.615,
363
+ "f1": 0.56,
364
+ "n": 65
365
+ },
366
+ "oasst2": {
367
+ "em": 0.625,
368
+ "f1": 0.754,
369
+ "n": 16
370
+ },
371
+ "swe_rebench_assistant": {
372
+ "em": 0.486,
373
+ "f1": 0.579,
374
+ "n": 37
375
+ },
376
+ "swe_rebench_tool": {
377
+ "em": 0.481,
378
+ "f1": 0.422,
379
+ "n": 77
380
+ },
381
+ "toucan_mcp": {
382
+ "em": 0.426,
383
+ "f1": 0.519,
384
+ "n": 54
385
+ },
386
+ "swe_smith_tool": {
387
+ "em": 0.294,
388
+ "f1": 0.275,
389
+ "n": 51
390
+ },
391
+ "_char_ratio": 0.52,
392
+ "_llm_token_ratio": 0.499
393
+ },
394
+ "llmlingua2-large@0.33": {
395
+ "fineweb2": {
396
+ "em": 0.5,
397
+ "f1": 0.516,
398
+ "n": 18
399
+ },
400
+ "_all": {
401
+ "em": 0.325,
402
+ "f1": 0.338,
403
+ "n": 378
404
+ },
405
+ "fineweb_edu": {
406
+ "em": 0.583,
407
+ "f1": 0.704,
408
+ "n": 60
409
+ },
410
+ "github_code": {
411
+ "em": 0.308,
412
+ "f1": 0.303,
413
+ "n": 65
414
+ },
415
+ "oasst2": {
416
+ "em": 0.438,
417
+ "f1": 0.494,
418
+ "n": 16
419
+ },
420
+ "swe_rebench_assistant": {
421
+ "em": 0.243,
422
+ "f1": 0.299,
423
+ "n": 37
424
+ },
425
+ "swe_rebench_tool": {
426
+ "em": 0.325,
427
+ "f1": 0.238,
428
+ "n": 77
429
+ },
430
+ "toucan_mcp": {
431
+ "em": 0.241,
432
+ "f1": 0.28,
433
+ "n": 54
434
+ },
435
+ "swe_smith_tool": {
436
+ "em": 0.098,
437
+ "f1": 0.082,
438
+ "n": 51
439
+ },
440
+ "_char_ratio": 0.342,
441
+ "_llm_token_ratio": 0.324
442
+ }
443
+ }
eval/v1-final-v1v3.json ADDED
@@ -0,0 +1,604 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "/mnt/ssd/ctxprune-data/runs/v1-final-v1v3/final",
3
+ "split": "test",
4
+ "methods": {
5
+ "ours@0.5": {
6
+ "swe_rebench_tool": {
7
+ "n": 11,
8
+ "protected": 290,
9
+ "lost": 167,
10
+ "lost_rate": 0.5758620689655173,
11
+ "json_inputs": 0,
12
+ "json_still_valid": 0,
13
+ "mean_char_ratio": 0.46790665518158064,
14
+ "mean_llm_token_ratio": 0.454551076188952
15
+ },
16
+ "swe_smith_tool": {
17
+ "n": 6,
18
+ "protected": 209,
19
+ "lost": 116,
20
+ "lost_rate": 0.5550239234449761,
21
+ "json_inputs": 0,
22
+ "json_still_valid": 0,
23
+ "mean_char_ratio": 0.5251933817728502,
24
+ "mean_llm_token_ratio": 0.49892634141769937
25
+ },
26
+ "github_code": {
27
+ "n": 14,
28
+ "protected": 277,
29
+ "lost": 80,
30
+ "lost_rate": 0.2888086642599278,
31
+ "json_inputs": 0,
32
+ "json_still_valid": 0,
33
+ "mean_char_ratio": 0.4900923855979181,
34
+ "mean_llm_token_ratio": 0.49410920530451385
35
+ },
36
+ "oasst2": {
37
+ "n": 1,
38
+ "protected": 4,
39
+ "lost": 0,
40
+ "lost_rate": 0.0,
41
+ "json_inputs": 0,
42
+ "json_still_valid": 0,
43
+ "mean_char_ratio": 0.5233399079552926,
44
+ "mean_llm_token_ratio": 0.4981949458483754
45
+ },
46
+ "fineweb_edu": {
47
+ "n": 12,
48
+ "protected": 80,
49
+ "lost": 7,
50
+ "lost_rate": 0.0875,
51
+ "json_inputs": 0,
52
+ "json_still_valid": 0,
53
+ "mean_char_ratio": 0.4762202894804961,
54
+ "mean_llm_token_ratio": 0.4943262326551133
55
+ },
56
+ "fineweb2": {
57
+ "n": 7,
58
+ "protected": 38,
59
+ "lost": 5,
60
+ "lost_rate": 0.13157894736842105,
61
+ "json_inputs": 0,
62
+ "json_still_valid": 0,
63
+ "mean_char_ratio": 0.46665221854276134,
64
+ "mean_llm_token_ratio": 0.4980696352415177
65
+ },
66
+ "toucan_mcp": {
67
+ "n": 5,
68
+ "protected": 87,
69
+ "lost": 38,
70
+ "lost_rate": 0.4367816091954023,
71
+ "json_inputs": 1,
72
+ "json_still_valid": 0,
73
+ "mean_char_ratio": 0.531668937503145,
74
+ "mean_llm_token_ratio": 0.4917416835051349
75
+ },
76
+ "swe_rebench_assistant": {
77
+ "n": 3,
78
+ "protected": 28,
79
+ "lost": 6,
80
+ "lost_rate": 0.21428571428571427,
81
+ "json_inputs": 0,
82
+ "json_still_valid": 0,
83
+ "mean_char_ratio": 0.34696757073863704,
84
+ "mean_llm_token_ratio": 0.4981844242713808
85
+ },
86
+ "_all": {
87
+ "n": 59,
88
+ "protected": 1013,
89
+ "lost": 419,
90
+ "lost_rate": 0.4136229022704837,
91
+ "json_inputs": 1,
92
+ "json_still_valid": 0,
93
+ "mean_char_ratio": 0.4807325967331789,
94
+ "mean_llm_token_ratio": 0.48781368873878034
95
+ },
96
+ "_seconds": 5.2,
97
+ "_ms_per_chunk": 87.8
98
+ },
99
+ "ours(chars)@0.5": {
100
+ "swe_rebench_tool": {
101
+ "n": 11,
102
+ "protected": 290,
103
+ "lost": 165,
104
+ "lost_rate": 0.5689655172413793,
105
+ "json_inputs": 0,
106
+ "json_still_valid": 0,
107
+ "mean_char_ratio": 0.5031079237938365,
108
+ "mean_llm_token_ratio": 0.49231885120417124
109
+ },
110
+ "swe_smith_tool": {
111
+ "n": 6,
112
+ "protected": 209,
113
+ "lost": 133,
114
+ "lost_rate": 0.6363636363636364,
115
+ "json_inputs": 0,
116
+ "json_still_valid": 0,
117
+ "mean_char_ratio": 0.4381255138088153,
118
+ "mean_llm_token_ratio": 0.42866471440857196
119
+ },
120
+ "github_code": {
121
+ "n": 14,
122
+ "protected": 277,
123
+ "lost": 81,
124
+ "lost_rate": 0.2924187725631769,
125
+ "json_inputs": 0,
126
+ "json_still_valid": 0,
127
+ "mean_char_ratio": 0.4916920386002186,
128
+ "mean_llm_token_ratio": 0.5075418238279651
129
+ },
130
+ "oasst2": {
131
+ "n": 1,
132
+ "protected": 4,
133
+ "lost": 0,
134
+ "lost_rate": 0.0,
135
+ "json_inputs": 0,
136
+ "json_still_valid": 0,
137
+ "mean_char_ratio": 0.5016436554898094,
138
+ "mean_llm_token_ratio": 0.47653429602888087
139
+ },
140
+ "fineweb_edu": {
141
+ "n": 12,
142
+ "protected": 80,
143
+ "lost": 5,
144
+ "lost_rate": 0.0625,
145
+ "json_inputs": 0,
146
+ "json_still_valid": 0,
147
+ "mean_char_ratio": 0.5067393191897068,
148
+ "mean_llm_token_ratio": 0.5236473269669588
149
+ },
150
+ "fineweb2": {
151
+ "n": 7,
152
+ "protected": 38,
153
+ "lost": 4,
154
+ "lost_rate": 0.10526315789473684,
155
+ "json_inputs": 0,
156
+ "json_still_valid": 0,
157
+ "mean_char_ratio": 0.5362221881998749,
158
+ "mean_llm_token_ratio": 0.5748280755051555
159
+ },
160
+ "toucan_mcp": {
161
+ "n": 5,
162
+ "protected": 87,
163
+ "lost": 45,
164
+ "lost_rate": 0.5172413793103449,
165
+ "json_inputs": 1,
166
+ "json_still_valid": 0,
167
+ "mean_char_ratio": 0.4654126478489207,
168
+ "mean_llm_token_ratio": 0.4294138779235846
169
+ },
170
+ "swe_rebench_assistant": {
171
+ "n": 3,
172
+ "protected": 28,
173
+ "lost": 4,
174
+ "lost_rate": 0.14285714285714285,
175
+ "json_inputs": 0,
176
+ "json_still_valid": 0,
177
+ "mean_char_ratio": 0.5077163620401056,
178
+ "mean_llm_token_ratio": 0.6181099697404046
179
+ },
180
+ "_all": {
181
+ "n": 59,
182
+ "protected": 1013,
183
+ "lost": 437,
184
+ "lost_rate": 0.43139190523198423,
185
+ "json_inputs": 1,
186
+ "json_still_valid": 0,
187
+ "mean_char_ratio": 0.4954730832799744,
188
+ "mean_llm_token_ratio": 0.5064165971236685
189
+ },
190
+ "_seconds": 5.2,
191
+ "_ms_per_chunk": 88.9
192
+ },
193
+ "ours+ids@0.5": {
194
+ "swe_rebench_tool": {
195
+ "n": 11,
196
+ "protected": 290,
197
+ "lost": 0,
198
+ "lost_rate": 0.0,
199
+ "json_inputs": 0,
200
+ "json_still_valid": 0,
201
+ "mean_char_ratio": 0.5848708565017198,
202
+ "mean_llm_token_ratio": 0.6607594850156845
203
+ },
204
+ "swe_smith_tool": {
205
+ "n": 6,
206
+ "protected": 209,
207
+ "lost": 0,
208
+ "lost_rate": 0.0,
209
+ "json_inputs": 0,
210
+ "json_still_valid": 0,
211
+ "mean_char_ratio": 0.43344472653878213,
212
+ "mean_llm_token_ratio": 0.5282570081147547
213
+ },
214
+ "github_code": {
215
+ "n": 14,
216
+ "protected": 277,
217
+ "lost": 0,
218
+ "lost_rate": 0.0,
219
+ "json_inputs": 0,
220
+ "json_still_valid": 0,
221
+ "mean_char_ratio": 0.49802537688259274,
222
+ "mean_llm_token_ratio": 0.5062940001561812
223
+ },
224
+ "oasst2": {
225
+ "n": 1,
226
+ "protected": 4,
227
+ "lost": 0,
228
+ "lost_rate": 0.0,
229
+ "json_inputs": 0,
230
+ "json_still_valid": 0,
231
+ "mean_char_ratio": 0.5233399079552926,
232
+ "mean_llm_token_ratio": 0.4981949458483754
233
+ },
234
+ "fineweb_edu": {
235
+ "n": 12,
236
+ "protected": 80,
237
+ "lost": 0,
238
+ "lost_rate": 0.0,
239
+ "json_inputs": 0,
240
+ "json_still_valid": 0,
241
+ "mean_char_ratio": 0.4647579279628507,
242
+ "mean_llm_token_ratio": 0.4987588567685885
243
+ },
244
+ "fineweb2": {
245
+ "n": 7,
246
+ "protected": 38,
247
+ "lost": 0,
248
+ "lost_rate": 0.0,
249
+ "json_inputs": 0,
250
+ "json_still_valid": 0,
251
+ "mean_char_ratio": 0.4646904859777982,
252
+ "mean_llm_token_ratio": 0.4980696352415177
253
+ },
254
+ "toucan_mcp": {
255
+ "n": 5,
256
+ "protected": 87,
257
+ "lost": 0,
258
+ "lost_rate": 0.0,
259
+ "json_inputs": 1,
260
+ "json_still_valid": 0,
261
+ "mean_char_ratio": 0.4868151814862343,
262
+ "mean_llm_token_ratio": 0.5379034319648817
263
+ },
264
+ "swe_rebench_assistant": {
265
+ "n": 3,
266
+ "protected": 28,
267
+ "lost": 0,
268
+ "lost_rate": 0.0,
269
+ "json_inputs": 0,
270
+ "json_still_valid": 0,
271
+ "mean_char_ratio": 0.33618142678510043,
272
+ "mean_llm_token_ratio": 0.4981844242713808
273
+ },
274
+ "_all": {
275
+ "n": 59,
276
+ "protected": 1013,
277
+ "lost": 0,
278
+ "lost_rate": 0.0,
279
+ "json_inputs": 1,
280
+ "json_still_valid": 0,
281
+ "mean_char_ratio": 0.48817782525844866,
282
+ "mean_llm_token_ratio": 0.5369469744482747
283
+ },
284
+ "_seconds": 5.7,
285
+ "_ms_per_chunk": 96.0
286
+ },
287
+ "ours@0.33": {
288
+ "swe_rebench_tool": {
289
+ "n": 11,
290
+ "protected": 290,
291
+ "lost": 209,
292
+ "lost_rate": 0.7206896551724138,
293
+ "json_inputs": 0,
294
+ "json_still_valid": 0,
295
+ "mean_char_ratio": 0.327296715791273,
296
+ "mean_llm_token_ratio": 0.31459708951298865
297
+ },
298
+ "swe_smith_tool": {
299
+ "n": 6,
300
+ "protected": 209,
301
+ "lost": 151,
302
+ "lost_rate": 0.722488038277512,
303
+ "json_inputs": 0,
304
+ "json_still_valid": 0,
305
+ "mean_char_ratio": 0.3492269554168825,
306
+ "mean_llm_token_ratio": 0.3270323309890272
307
+ },
308
+ "github_code": {
309
+ "n": 14,
310
+ "protected": 277,
311
+ "lost": 134,
312
+ "lost_rate": 0.48375451263537905,
313
+ "json_inputs": 0,
314
+ "json_still_valid": 0,
315
+ "mean_char_ratio": 0.3234588557857195,
316
+ "mean_llm_token_ratio": 0.32284723564881385
317
+ },
318
+ "oasst2": {
319
+ "n": 1,
320
+ "protected": 4,
321
+ "lost": 0,
322
+ "lost_rate": 0.0,
323
+ "json_inputs": 0,
324
+ "json_still_valid": 0,
325
+ "mean_char_ratio": 0.33596318211702825,
326
+ "mean_llm_token_ratio": 0.3285198555956679
327
+ },
328
+ "fineweb_edu": {
329
+ "n": 12,
330
+ "protected": 80,
331
+ "lost": 18,
332
+ "lost_rate": 0.225,
333
+ "json_inputs": 0,
334
+ "json_still_valid": 0,
335
+ "mean_char_ratio": 0.2877951029262656,
336
+ "mean_llm_token_ratio": 0.3277054998143117
337
+ },
338
+ "fineweb2": {
339
+ "n": 7,
340
+ "protected": 38,
341
+ "lost": 12,
342
+ "lost_rate": 0.3157894736842105,
343
+ "json_inputs": 0,
344
+ "json_still_valid": 0,
345
+ "mean_char_ratio": 0.30053077932409866,
346
+ "mean_llm_token_ratio": 0.3264086608045815
347
+ },
348
+ "toucan_mcp": {
349
+ "n": 5,
350
+ "protected": 87,
351
+ "lost": 52,
352
+ "lost_rate": 0.5977011494252874,
353
+ "json_inputs": 1,
354
+ "json_still_valid": 0,
355
+ "mean_char_ratio": 0.34743765737380344,
356
+ "mean_llm_token_ratio": 0.2995299597179061
357
+ },
358
+ "swe_rebench_assistant": {
359
+ "n": 3,
360
+ "protected": 28,
361
+ "lost": 10,
362
+ "lost_rate": 0.35714285714285715,
363
+ "json_inputs": 0,
364
+ "json_still_valid": 0,
365
+ "mean_char_ratio": 0.18068367830835266,
366
+ "mean_llm_token_ratio": 0.3247372193024367
367
+ },
368
+ "_all": {
369
+ "n": 59,
370
+ "protected": 1013,
371
+ "lost": 586,
372
+ "lost_rate": 0.5784797630799605,
373
+ "json_inputs": 1,
374
+ "json_still_valid": 0,
375
+ "mean_char_ratio": 0.3118052335847517,
376
+ "mean_llm_token_ratio": 0.32136154585011445
377
+ },
378
+ "_seconds": 5.6,
379
+ "_ms_per_chunk": 94.8
380
+ },
381
+ "ours(chars)@0.33": {
382
+ "swe_rebench_tool": {
383
+ "n": 11,
384
+ "protected": 290,
385
+ "lost": 210,
386
+ "lost_rate": 0.7241379310344828,
387
+ "json_inputs": 0,
388
+ "json_still_valid": 0,
389
+ "mean_char_ratio": 0.3439364522856813,
390
+ "mean_llm_token_ratio": 0.3314479793413248
391
+ },
392
+ "swe_smith_tool": {
393
+ "n": 6,
394
+ "protected": 209,
395
+ "lost": 162,
396
+ "lost_rate": 0.7751196172248804,
397
+ "json_inputs": 0,
398
+ "json_still_valid": 0,
399
+ "mean_char_ratio": 0.28580805596060765,
400
+ "mean_llm_token_ratio": 0.27243181669152455
401
+ },
402
+ "github_code": {
403
+ "n": 14,
404
+ "protected": 277,
405
+ "lost": 138,
406
+ "lost_rate": 0.4981949458483754,
407
+ "json_inputs": 0,
408
+ "json_still_valid": 0,
409
+ "mean_char_ratio": 0.31775308471956165,
410
+ "mean_llm_token_ratio": 0.31525592307107037
411
+ },
412
+ "oasst2": {
413
+ "n": 1,
414
+ "protected": 4,
415
+ "lost": 0,
416
+ "lost_rate": 0.0,
417
+ "json_inputs": 0,
418
+ "json_still_valid": 0,
419
+ "mean_char_ratio": 0.3333333333333333,
420
+ "mean_llm_token_ratio": 0.3249097472924188
421
+ },
422
+ "fineweb_edu": {
423
+ "n": 12,
424
+ "protected": 80,
425
+ "lost": 10,
426
+ "lost_rate": 0.125,
427
+ "json_inputs": 0,
428
+ "json_still_valid": 0,
429
+ "mean_char_ratio": 0.3365119094824531,
430
+ "mean_llm_token_ratio": 0.37627862799399553
431
+ },
432
+ "fineweb2": {
433
+ "n": 7,
434
+ "protected": 38,
435
+ "lost": 11,
436
+ "lost_rate": 0.2894736842105263,
437
+ "json_inputs": 0,
438
+ "json_still_valid": 0,
439
+ "mean_char_ratio": 0.3573804579329673,
440
+ "mean_llm_token_ratio": 0.3856092246572405
441
+ },
442
+ "toucan_mcp": {
443
+ "n": 5,
444
+ "protected": 87,
445
+ "lost": 54,
446
+ "lost_rate": 0.6206896551724138,
447
+ "json_inputs": 1,
448
+ "json_still_valid": 0,
449
+ "mean_char_ratio": 0.30707900248035375,
450
+ "mean_llm_token_ratio": 0.27636029308647636
451
+ },
452
+ "swe_rebench_assistant": {
453
+ "n": 3,
454
+ "protected": 28,
455
+ "lost": 6,
456
+ "lost_rate": 0.21428571428571427,
457
+ "json_inputs": 0,
458
+ "json_still_valid": 0,
459
+ "mean_char_ratio": 0.3374447399733389,
460
+ "mean_llm_token_ratio": 0.49111124382863514
461
+ },
462
+ "_all": {
463
+ "n": 59,
464
+ "protected": 1013,
465
+ "lost": 591,
466
+ "lost_rate": 0.5834155972359328,
467
+ "json_inputs": 1,
468
+ "json_still_valid": 0,
469
+ "mean_char_ratio": 0.3282637488467005,
470
+ "mean_llm_token_ratio": 0.3404873669260685
471
+ },
472
+ "_seconds": 5.0,
473
+ "_ms_per_chunk": 84.4
474
+ },
475
+ "ours+ids@0.33": {
476
+ "swe_rebench_tool": {
477
+ "n": 11,
478
+ "protected": 290,
479
+ "lost": 0,
480
+ "lost_rate": 0.0,
481
+ "json_inputs": 0,
482
+ "json_still_valid": 0,
483
+ "mean_char_ratio": 0.5750178444782263,
484
+ "mean_llm_token_ratio": 0.6529765472791863
485
+ },
486
+ "swe_smith_tool": {
487
+ "n": 6,
488
+ "protected": 209,
489
+ "lost": 0,
490
+ "lost_rate": 0.0,
491
+ "json_inputs": 0,
492
+ "json_still_valid": 0,
493
+ "mean_char_ratio": 0.3742236154767515,
494
+ "mean_llm_token_ratio": 0.47296373915503437
495
+ },
496
+ "github_code": {
497
+ "n": 14,
498
+ "protected": 277,
499
+ "lost": 0,
500
+ "lost_rate": 0.0,
501
+ "json_inputs": 0,
502
+ "json_still_valid": 0,
503
+ "mean_char_ratio": 0.3753987414580236,
504
+ "mean_llm_token_ratio": 0.38185252441714607
505
+ },
506
+ "oasst2": {
507
+ "n": 1,
508
+ "protected": 4,
509
+ "lost": 0,
510
+ "lost_rate": 0.0,
511
+ "json_inputs": 0,
512
+ "json_still_valid": 0,
513
+ "mean_char_ratio": 0.33596318211702825,
514
+ "mean_llm_token_ratio": 0.3285198555956679
515
+ },
516
+ "fineweb_edu": {
517
+ "n": 12,
518
+ "protected": 80,
519
+ "lost": 0,
520
+ "lost_rate": 0.0,
521
+ "json_inputs": 0,
522
+ "json_still_valid": 0,
523
+ "mean_char_ratio": 0.27070723428966886,
524
+ "mean_llm_token_ratio": 0.3296230704501357
525
+ },
526
+ "fineweb2": {
527
+ "n": 7,
528
+ "protected": 38,
529
+ "lost": 0,
530
+ "lost_rate": 0.0,
531
+ "json_inputs": 0,
532
+ "json_still_valid": 0,
533
+ "mean_char_ratio": 0.2918360105579523,
534
+ "mean_llm_token_ratio": 0.3264086608045815
535
+ },
536
+ "toucan_mcp": {
537
+ "n": 5,
538
+ "protected": 87,
539
+ "lost": 0,
540
+ "lost_rate": 0.0,
541
+ "json_inputs": 1,
542
+ "json_still_valid": 0,
543
+ "mean_char_ratio": 0.41575461058811414,
544
+ "mean_llm_token_ratio": 0.48518356616090375
545
+ },
546
+ "swe_rebench_assistant": {
547
+ "n": 3,
548
+ "protected": 28,
549
+ "lost": 0,
550
+ "lost_rate": 0.0,
551
+ "json_inputs": 0,
552
+ "json_still_valid": 0,
553
+ "mean_char_ratio": 0.17654440314872719,
554
+ "mean_llm_token_ratio": 0.3341654722089505
555
+ },
556
+ "_all": {
557
+ "n": 59,
558
+ "protected": 1013,
559
+ "lost": 0,
560
+ "lost_rate": 0.0,
561
+ "json_inputs": 1,
562
+ "json_still_valid": 0,
563
+ "mean_char_ratio": 0.3739292998715051,
564
+ "mean_llm_token_ratio": 0.42989375204918706
565
+ },
566
+ "_seconds": 5.7,
567
+ "_ms_per_chunk": 97.0
568
+ }
569
+ },
570
+ "agreement": {
571
+ "swe_rebench_tool": {
572
+ "keep_f1": 0.789,
573
+ "acc": 0.697
574
+ },
575
+ "swe_smith_tool": {
576
+ "keep_f1": 0.834,
577
+ "acc": 0.746
578
+ },
579
+ "github_code": {
580
+ "keep_f1": 0.932,
581
+ "acc": 0.921
582
+ },
583
+ "oasst2": {
584
+ "keep_f1": 0.802,
585
+ "acc": 0.847
586
+ },
587
+ "fineweb_edu": {
588
+ "keep_f1": 0.873,
589
+ "acc": 0.795
590
+ },
591
+ "fineweb2": {
592
+ "keep_f1": 0.762,
593
+ "acc": 0.624
594
+ },
595
+ "toucan_mcp": {
596
+ "keep_f1": 0.684,
597
+ "acc": 0.592
598
+ },
599
+ "swe_rebench_assistant": {
600
+ "keep_f1": 0.833,
601
+ "acc": 0.731
602
+ }
603
+ }
604
+ }
eval/v1-n3000-small.json ADDED
@@ -0,0 +1,980 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "/mnt/ssd/ctxprune-data/runs/v1-n3000-small/final",
3
+ "split": "test",
4
+ "methods": {
5
+ "ours@0.5": {
6
+ "swe_rebench_tool": {
7
+ "n": 11,
8
+ "protected": 290,
9
+ "lost": 144,
10
+ "lost_rate": 0.496551724137931,
11
+ "json_inputs": 0,
12
+ "json_still_valid": 0,
13
+ "mean_char_ratio": 0.48423768432288145,
14
+ "mean_llm_token_ratio": 0.47301808510243837
15
+ },
16
+ "swe_smith_tool": {
17
+ "n": 6,
18
+ "protected": 209,
19
+ "lost": 113,
20
+ "lost_rate": 0.5406698564593302,
21
+ "json_inputs": 0,
22
+ "json_still_valid": 0,
23
+ "mean_char_ratio": 0.5045351789343399,
24
+ "mean_llm_token_ratio": 0.4958170055781004
25
+ },
26
+ "github_code": {
27
+ "n": 14,
28
+ "protected": 277,
29
+ "lost": 86,
30
+ "lost_rate": 0.3104693140794224,
31
+ "json_inputs": 0,
32
+ "json_still_valid": 0,
33
+ "mean_char_ratio": 0.4940215098011816,
34
+ "mean_llm_token_ratio": 0.49412901256934294
35
+ },
36
+ "oasst2": {
37
+ "n": 1,
38
+ "protected": 4,
39
+ "lost": 0,
40
+ "lost_rate": 0.0,
41
+ "json_inputs": 0,
42
+ "json_still_valid": 0,
43
+ "mean_char_ratio": 0.5470085470085471,
44
+ "mean_llm_token_ratio": 0.4981949458483754
45
+ },
46
+ "fineweb_edu": {
47
+ "n": 12,
48
+ "protected": 80,
49
+ "lost": 5,
50
+ "lost_rate": 0.0625,
51
+ "json_inputs": 0,
52
+ "json_still_valid": 0,
53
+ "mean_char_ratio": 0.47453594159266865,
54
+ "mean_llm_token_ratio": 0.49865726487076095
55
+ },
56
+ "fineweb2": {
57
+ "n": 7,
58
+ "protected": 38,
59
+ "lost": 5,
60
+ "lost_rate": 0.13157894736842105,
61
+ "json_inputs": 0,
62
+ "json_still_valid": 0,
63
+ "mean_char_ratio": 0.4638766596639581,
64
+ "mean_llm_token_ratio": 0.4973468624901391
65
+ },
66
+ "toucan_mcp": {
67
+ "n": 5,
68
+ "protected": 87,
69
+ "lost": 18,
70
+ "lost_rate": 0.20689655172413793,
71
+ "json_inputs": 1,
72
+ "json_still_valid": 0,
73
+ "mean_char_ratio": 0.4772902876699033,
74
+ "mean_llm_token_ratio": 0.46184248650945336
75
+ },
76
+ "swe_rebench_assistant": {
77
+ "n": 3,
78
+ "protected": 28,
79
+ "lost": 5,
80
+ "lost_rate": 0.17857142857142858,
81
+ "json_inputs": 0,
82
+ "json_still_valid": 0,
83
+ "mean_char_ratio": 0.3414247920356499,
84
+ "mean_llm_token_ratio": 0.4974518235387801
85
+ },
86
+ "_all": {
87
+ "n": 59,
88
+ "protected": 1013,
89
+ "lost": 376,
90
+ "lost_rate": 0.3711747285291214,
91
+ "json_inputs": 1,
92
+ "json_still_valid": 0,
93
+ "mean_char_ratio": 0.47744794943388175,
94
+ "mean_llm_token_ratio": 0.4891692408552257
95
+ },
96
+ "_seconds": 6.9,
97
+ "_ms_per_chunk": 116.3
98
+ },
99
+ "ours(chars)@0.5": {
100
+ "swe_rebench_tool": {
101
+ "n": 11,
102
+ "protected": 290,
103
+ "lost": 146,
104
+ "lost_rate": 0.503448275862069,
105
+ "json_inputs": 0,
106
+ "json_still_valid": 0,
107
+ "mean_char_ratio": 0.50514117726754,
108
+ "mean_llm_token_ratio": 0.4932422422339848
109
+ },
110
+ "swe_smith_tool": {
111
+ "n": 6,
112
+ "protected": 209,
113
+ "lost": 122,
114
+ "lost_rate": 0.583732057416268,
115
+ "json_inputs": 0,
116
+ "json_still_valid": 0,
117
+ "mean_char_ratio": 0.44408396808198775,
118
+ "mean_llm_token_ratio": 0.4358757119521421
119
+ },
120
+ "github_code": {
121
+ "n": 14,
122
+ "protected": 277,
123
+ "lost": 89,
124
+ "lost_rate": 0.3212996389891697,
125
+ "json_inputs": 0,
126
+ "json_still_valid": 0,
127
+ "mean_char_ratio": 0.4847587082377705,
128
+ "mean_llm_token_ratio": 0.4879555289996183
129
+ },
130
+ "oasst2": {
131
+ "n": 1,
132
+ "protected": 4,
133
+ "lost": 0,
134
+ "lost_rate": 0.0,
135
+ "json_inputs": 0,
136
+ "json_still_valid": 0,
137
+ "mean_char_ratio": 0.5055884286653517,
138
+ "mean_llm_token_ratio": 0.4693140794223827
139
+ },
140
+ "fineweb_edu": {
141
+ "n": 12,
142
+ "protected": 80,
143
+ "lost": 4,
144
+ "lost_rate": 0.05,
145
+ "json_inputs": 0,
146
+ "json_still_valid": 0,
147
+ "mean_char_ratio": 0.5105836670519339,
148
+ "mean_llm_token_ratio": 0.5317948991842393
149
+ },
150
+ "fineweb2": {
151
+ "n": 7,
152
+ "protected": 38,
153
+ "lost": 3,
154
+ "lost_rate": 0.07894736842105263,
155
+ "json_inputs": 0,
156
+ "json_still_valid": 0,
157
+ "mean_char_ratio": 0.5476579453546034,
158
+ "mean_llm_token_ratio": 0.5851437803482594
159
+ },
160
+ "toucan_mcp": {
161
+ "n": 5,
162
+ "protected": 87,
163
+ "lost": 17,
164
+ "lost_rate": 0.19540229885057472,
165
+ "json_inputs": 1,
166
+ "json_still_valid": 0,
167
+ "mean_char_ratio": 0.48779160972796537,
168
+ "mean_llm_token_ratio": 0.4897364920998493
169
+ },
170
+ "swe_rebench_assistant": {
171
+ "n": 3,
172
+ "protected": 28,
173
+ "lost": 2,
174
+ "lost_rate": 0.07142857142857142,
175
+ "json_inputs": 0,
176
+ "json_still_valid": 0,
177
+ "mean_char_ratio": 0.5104579306427771,
178
+ "mean_llm_token_ratio": 0.6134396400700748
179
+ },
180
+ "_all": {
181
+ "n": 59,
182
+ "protected": 1013,
183
+ "lost": 383,
184
+ "lost_rate": 0.3780848963474827,
185
+ "json_inputs": 1,
186
+ "json_still_valid": 0,
187
+ "mean_char_ratio": 0.499054382459366,
188
+ "mean_llm_token_ratio": 0.5103077466959641
189
+ },
190
+ "_seconds": 5.3,
191
+ "_ms_per_chunk": 89.5
192
+ },
193
+ "ours+ids@0.5": {
194
+ "swe_rebench_tool": {
195
+ "n": 11,
196
+ "protected": 290,
197
+ "lost": 0,
198
+ "lost_rate": 0.0,
199
+ "json_inputs": 0,
200
+ "json_still_valid": 0,
201
+ "mean_char_ratio": 0.5864942331250964,
202
+ "mean_llm_token_ratio": 0.6607594850156845
203
+ },
204
+ "swe_smith_tool": {
205
+ "n": 6,
206
+ "protected": 209,
207
+ "lost": 0,
208
+ "lost_rate": 0.0,
209
+ "json_inputs": 0,
210
+ "json_still_valid": 0,
211
+ "mean_char_ratio": 0.4296961730913889,
212
+ "mean_llm_token_ratio": 0.5287682555585176
213
+ },
214
+ "github_code": {
215
+ "n": 14,
216
+ "protected": 277,
217
+ "lost": 0,
218
+ "lost_rate": 0.0,
219
+ "json_inputs": 0,
220
+ "json_still_valid": 0,
221
+ "mean_char_ratio": 0.5125688993269367,
222
+ "mean_llm_token_ratio": 0.5055027797480987
223
+ },
224
+ "oasst2": {
225
+ "n": 1,
226
+ "protected": 4,
227
+ "lost": 0,
228
+ "lost_rate": 0.0,
229
+ "json_inputs": 0,
230
+ "json_still_valid": 0,
231
+ "mean_char_ratio": 0.5470085470085471,
232
+ "mean_llm_token_ratio": 0.4981949458483754
233
+ },
234
+ "fineweb_edu": {
235
+ "n": 12,
236
+ "protected": 80,
237
+ "lost": 0,
238
+ "lost_rate": 0.0,
239
+ "json_inputs": 0,
240
+ "json_still_valid": 0,
241
+ "mean_char_ratio": 0.464029585344431,
242
+ "mean_llm_token_ratio": 0.49865726487076095
243
+ },
244
+ "fineweb2": {
245
+ "n": 7,
246
+ "protected": 38,
247
+ "lost": 0,
248
+ "lost_rate": 0.0,
249
+ "json_inputs": 0,
250
+ "json_still_valid": 0,
251
+ "mean_char_ratio": 0.4598986286108202,
252
+ "mean_llm_token_ratio": 0.4980696352415177
253
+ },
254
+ "toucan_mcp": {
255
+ "n": 5,
256
+ "protected": 87,
257
+ "lost": 0,
258
+ "lost_rate": 0.0,
259
+ "json_inputs": 1,
260
+ "json_still_valid": 0,
261
+ "mean_char_ratio": 0.47559215990350046,
262
+ "mean_llm_token_ratio": 0.5379034319648817
263
+ },
264
+ "swe_rebench_assistant": {
265
+ "n": 3,
266
+ "protected": 28,
267
+ "lost": 0,
268
+ "lost_rate": 0.0,
269
+ "json_inputs": 0,
270
+ "json_still_valid": 0,
271
+ "mean_char_ratio": 0.3414247920356499,
272
+ "mean_llm_token_ratio": 0.4981844242713808
273
+ },
274
+ "_all": {
275
+ "n": 59,
276
+ "protected": 1013,
277
+ "lost": 0,
278
+ "lost_rate": 0.0,
279
+ "json_inputs": 1,
280
+ "json_still_valid": 0,
281
+ "mean_char_ratio": 0.4905502939075156,
282
+ "mean_llm_token_ratio": 0.5367905557393846
283
+ },
284
+ "_seconds": 5.9,
285
+ "_ms_per_chunk": 100.8
286
+ },
287
+ "ours@0.33": {
288
+ "swe_rebench_tool": {
289
+ "n": 11,
290
+ "protected": 290,
291
+ "lost": 197,
292
+ "lost_rate": 0.6793103448275862,
293
+ "json_inputs": 0,
294
+ "json_still_valid": 0,
295
+ "mean_char_ratio": 0.3147373748073354,
296
+ "mean_llm_token_ratio": 0.3031697600710548
297
+ },
298
+ "swe_smith_tool": {
299
+ "n": 6,
300
+ "protected": 209,
301
+ "lost": 150,
302
+ "lost_rate": 0.7177033492822966,
303
+ "json_inputs": 0,
304
+ "json_still_valid": 0,
305
+ "mean_char_ratio": 0.3495610928822163,
306
+ "mean_llm_token_ratio": 0.3256947048838342
307
+ },
308
+ "github_code": {
309
+ "n": 14,
310
+ "protected": 277,
311
+ "lost": 137,
312
+ "lost_rate": 0.49458483754512633,
313
+ "json_inputs": 0,
314
+ "json_still_valid": 0,
315
+ "mean_char_ratio": 0.3337368310757179,
316
+ "mean_llm_token_ratio": 0.3250242003716078
317
+ },
318
+ "oasst2": {
319
+ "n": 1,
320
+ "protected": 4,
321
+ "lost": 0,
322
+ "lost_rate": 0.0,
323
+ "json_inputs": 0,
324
+ "json_still_valid": 0,
325
+ "mean_char_ratio": 0.3254437869822485,
326
+ "mean_llm_token_ratio": 0.3285198555956679
327
+ },
328
+ "fineweb_edu": {
329
+ "n": 12,
330
+ "protected": 80,
331
+ "lost": 13,
332
+ "lost_rate": 0.1625,
333
+ "json_inputs": 0,
334
+ "json_still_valid": 0,
335
+ "mean_char_ratio": 0.28687166627630484,
336
+ "mean_llm_token_ratio": 0.31721398253047167
337
+ },
338
+ "fineweb2": {
339
+ "n": 7,
340
+ "protected": 38,
341
+ "lost": 13,
342
+ "lost_rate": 0.34210526315789475,
343
+ "json_inputs": 0,
344
+ "json_still_valid": 0,
345
+ "mean_char_ratio": 0.2915544664318485,
346
+ "mean_llm_token_ratio": 0.32604307329293963
347
+ },
348
+ "toucan_mcp": {
349
+ "n": 5,
350
+ "protected": 87,
351
+ "lost": 26,
352
+ "lost_rate": 0.2988505747126437,
353
+ "json_inputs": 1,
354
+ "json_still_valid": 0,
355
+ "mean_char_ratio": 0.3389784519761102,
356
+ "mean_llm_token_ratio": 0.30767336131301837
357
+ },
358
+ "swe_rebench_assistant": {
359
+ "n": 3,
360
+ "protected": 28,
361
+ "lost": 8,
362
+ "lost_rate": 0.2857142857142857,
363
+ "json_inputs": 0,
364
+ "json_still_valid": 0,
365
+ "mean_char_ratio": 0.17224531982003863,
366
+ "mean_llm_token_ratio": 0.2935857620640229
367
+ },
368
+ "_all": {
369
+ "n": 59,
370
+ "protected": 1013,
371
+ "lost": 544,
372
+ "lost_rate": 0.5370187561697927,
373
+ "json_inputs": 1,
374
+ "json_still_valid": 0,
375
+ "mean_char_ratio": 0.30935943359145,
376
+ "mean_llm_token_ratio": 0.31654046859417256
377
+ },
378
+ "_seconds": 6.2,
379
+ "_ms_per_chunk": 105.9
380
+ },
381
+ "ours(chars)@0.33": {
382
+ "swe_rebench_tool": {
383
+ "n": 11,
384
+ "protected": 290,
385
+ "lost": 200,
386
+ "lost_rate": 0.6896551724137931,
387
+ "json_inputs": 0,
388
+ "json_still_valid": 0,
389
+ "mean_char_ratio": 0.34712354434240505,
390
+ "mean_llm_token_ratio": 0.33936415907512835
391
+ },
392
+ "swe_smith_tool": {
393
+ "n": 6,
394
+ "protected": 209,
395
+ "lost": 164,
396
+ "lost_rate": 0.784688995215311,
397
+ "json_inputs": 0,
398
+ "json_still_valid": 0,
399
+ "mean_char_ratio": 0.28845387196121053,
400
+ "mean_llm_token_ratio": 0.27615503415370324
401
+ },
402
+ "github_code": {
403
+ "n": 14,
404
+ "protected": 277,
405
+ "lost": 140,
406
+ "lost_rate": 0.5054151624548736,
407
+ "json_inputs": 0,
408
+ "json_still_valid": 0,
409
+ "mean_char_ratio": 0.31909759484598904,
410
+ "mean_llm_token_ratio": 0.3109828118463588
411
+ },
412
+ "oasst2": {
413
+ "n": 1,
414
+ "protected": 4,
415
+ "lost": 0,
416
+ "lost_rate": 0.0,
417
+ "json_inputs": 0,
418
+ "json_still_valid": 0,
419
+ "mean_char_ratio": 0.3326758711374096,
420
+ "mean_llm_token_ratio": 0.33212996389891697
421
+ },
422
+ "fineweb_edu": {
423
+ "n": 12,
424
+ "protected": 80,
425
+ "lost": 9,
426
+ "lost_rate": 0.1125,
427
+ "json_inputs": 0,
428
+ "json_still_valid": 0,
429
+ "mean_char_ratio": 0.3375446856730053,
430
+ "mean_llm_token_ratio": 0.36797164170699337
431
+ },
432
+ "fineweb2": {
433
+ "n": 7,
434
+ "protected": 38,
435
+ "lost": 11,
436
+ "lost_rate": 0.2894736842105263,
437
+ "json_inputs": 0,
438
+ "json_still_valid": 0,
439
+ "mean_char_ratio": 0.372994124832598,
440
+ "mean_llm_token_ratio": 0.416826266445812
441
+ },
442
+ "toucan_mcp": {
443
+ "n": 5,
444
+ "protected": 87,
445
+ "lost": 31,
446
+ "lost_rate": 0.3563218390804598,
447
+ "json_inputs": 1,
448
+ "json_still_valid": 0,
449
+ "mean_char_ratio": 0.3301455391887981,
450
+ "mean_llm_token_ratio": 0.29778120189050133
451
+ },
452
+ "swe_rebench_assistant": {
453
+ "n": 3,
454
+ "protected": 28,
455
+ "lost": 4,
456
+ "lost_rate": 0.14285714285714285,
457
+ "json_inputs": 0,
458
+ "json_still_valid": 0,
459
+ "mean_char_ratio": 0.33194010436629257,
460
+ "mean_llm_token_ratio": 0.4887661251791687
461
+ },
462
+ "_all": {
463
+ "n": 59,
464
+ "protected": 1013,
465
+ "lost": 559,
466
+ "lost_rate": 0.5518262586377097,
467
+ "json_inputs": 1,
468
+ "json_still_valid": 0,
469
+ "mean_char_ratio": 0.3331723310078321,
470
+ "mean_llm_token_ratio": 0.3451605633066303
471
+ },
472
+ "_seconds": 5.7,
473
+ "_ms_per_chunk": 96.7
474
+ },
475
+ "ours+ids@0.33": {
476
+ "swe_rebench_tool": {
477
+ "n": 11,
478
+ "protected": 290,
479
+ "lost": 0,
480
+ "lost_rate": 0.0,
481
+ "json_inputs": 0,
482
+ "json_still_valid": 0,
483
+ "mean_char_ratio": 0.5750178444782263,
484
+ "mean_llm_token_ratio": 0.6529765472791863
485
+ },
486
+ "swe_smith_tool": {
487
+ "n": 6,
488
+ "protected": 209,
489
+ "lost": 0,
490
+ "lost_rate": 0.0,
491
+ "json_inputs": 0,
492
+ "json_still_valid": 0,
493
+ "mean_char_ratio": 0.37701232938751267,
494
+ "mean_llm_token_ratio": 0.47296373915503437
495
+ },
496
+ "github_code": {
497
+ "n": 14,
498
+ "protected": 277,
499
+ "lost": 0,
500
+ "lost_rate": 0.0,
501
+ "json_inputs": 0,
502
+ "json_still_valid": 0,
503
+ "mean_char_ratio": 0.3780213667901772,
504
+ "mean_llm_token_ratio": 0.38185252441714607
505
+ },
506
+ "oasst2": {
507
+ "n": 1,
508
+ "protected": 4,
509
+ "lost": 0,
510
+ "lost_rate": 0.0,
511
+ "json_inputs": 0,
512
+ "json_still_valid": 0,
513
+ "mean_char_ratio": 0.3254437869822485,
514
+ "mean_llm_token_ratio": 0.3285198555956679
515
+ },
516
+ "fineweb_edu": {
517
+ "n": 12,
518
+ "protected": 80,
519
+ "lost": 0,
520
+ "lost_rate": 0.0,
521
+ "json_inputs": 0,
522
+ "json_still_valid": 0,
523
+ "mean_char_ratio": 0.28171062653699325,
524
+ "mean_llm_token_ratio": 0.3305404524458229
525
+ },
526
+ "fineweb2": {
527
+ "n": 7,
528
+ "protected": 38,
529
+ "lost": 0,
530
+ "lost_rate": 0.0,
531
+ "json_inputs": 0,
532
+ "json_still_valid": 0,
533
+ "mean_char_ratio": 0.27900554381268056,
534
+ "mean_llm_token_ratio": 0.3262027295623831
535
+ },
536
+ "toucan_mcp": {
537
+ "n": 5,
538
+ "protected": 87,
539
+ "lost": 0,
540
+ "lost_rate": 0.0,
541
+ "json_inputs": 1,
542
+ "json_still_valid": 0,
543
+ "mean_char_ratio": 0.4161847181149958,
544
+ "mean_llm_token_ratio": 0.48518356616090375
545
+ },
546
+ "swe_rebench_assistant": {
547
+ "n": 3,
548
+ "protected": 28,
549
+ "lost": 0,
550
+ "lost_rate": 0.0,
551
+ "json_inputs": 0,
552
+ "json_still_valid": 0,
553
+ "mean_char_ratio": 0.18484746678022312,
554
+ "mean_llm_token_ratio": 0.3341654722089505
555
+ },
556
+ "_all": {
557
+ "n": 59,
558
+ "protected": 1013,
559
+ "lost": 0,
560
+ "lost_rate": 0.0,
561
+ "json_inputs": 1,
562
+ "json_still_valid": 0,
563
+ "mean_char_ratio": 0.37583127972336655,
564
+ "mean_llm_token_ratio": 0.43005590569754065
565
+ },
566
+ "_seconds": 5.9,
567
+ "_ms_per_chunk": 99.6
568
+ },
569
+ "llmlingua2@0.5": {
570
+ "swe_rebench_tool": {
571
+ "n": 11,
572
+ "protected": 290,
573
+ "lost": 177,
574
+ "lost_rate": 0.6103448275862069,
575
+ "json_inputs": 0,
576
+ "json_still_valid": 0,
577
+ "mean_char_ratio": 0.6691956196057945,
578
+ "mean_llm_token_ratio": 0.6151253760386801
579
+ },
580
+ "swe_smith_tool": {
581
+ "n": 6,
582
+ "protected": 209,
583
+ "lost": 69,
584
+ "lost_rate": 0.33014354066985646,
585
+ "json_inputs": 0,
586
+ "json_still_valid": 0,
587
+ "mean_char_ratio": 0.5833648327533828,
588
+ "mean_llm_token_ratio": 0.5938419310574503
589
+ },
590
+ "github_code": {
591
+ "n": 14,
592
+ "protected": 277,
593
+ "lost": 157,
594
+ "lost_rate": 0.5667870036101083,
595
+ "json_inputs": 0,
596
+ "json_still_valid": 0,
597
+ "mean_char_ratio": 0.6100950049930705,
598
+ "mean_llm_token_ratio": 0.5404820682262972
599
+ },
600
+ "oasst2": {
601
+ "n": 1,
602
+ "protected": 4,
603
+ "lost": 0,
604
+ "lost_rate": 0.0,
605
+ "json_inputs": 0,
606
+ "json_still_valid": 0,
607
+ "mean_char_ratio": 0.5785667324128863,
608
+ "mean_llm_token_ratio": 0.44765342960288806
609
+ },
610
+ "fineweb_edu": {
611
+ "n": 12,
612
+ "protected": 80,
613
+ "lost": 9,
614
+ "lost_rate": 0.1125,
615
+ "json_inputs": 0,
616
+ "json_still_valid": 0,
617
+ "mean_char_ratio": 0.5360388866928232,
618
+ "mean_llm_token_ratio": 0.48237259084432965
619
+ },
620
+ "fineweb2": {
621
+ "n": 7,
622
+ "protected": 38,
623
+ "lost": 11,
624
+ "lost_rate": 0.2894736842105263,
625
+ "json_inputs": 0,
626
+ "json_still_valid": 0,
627
+ "mean_char_ratio": 0.6865074668533124,
628
+ "mean_llm_token_ratio": 0.897635687187918
629
+ },
630
+ "toucan_mcp": {
631
+ "n": 5,
632
+ "protected": 87,
633
+ "lost": 37,
634
+ "lost_rate": 0.42528735632183906,
635
+ "json_inputs": 1,
636
+ "json_still_valid": 0,
637
+ "mean_char_ratio": 0.6356355690804469,
638
+ "mean_llm_token_ratio": 0.5623541129880764
639
+ },
640
+ "swe_rebench_assistant": {
641
+ "n": 3,
642
+ "protected": 28,
643
+ "lost": 18,
644
+ "lost_rate": 0.6428571428571429,
645
+ "json_inputs": 0,
646
+ "json_still_valid": 0,
647
+ "mean_char_ratio": 0.5794979969770179,
648
+ "mean_llm_token_ratio": 0.5970297818123905
649
+ },
650
+ "_all": {
651
+ "n": 59,
652
+ "protected": 1013,
653
+ "lost": 478,
654
+ "lost_rate": 0.47186574531095754,
655
+ "json_inputs": 1,
656
+ "json_still_valid": 0,
657
+ "mean_char_ratio": 0.6124733620189875,
658
+ "mean_llm_token_ratio": 0.5935358291248503
659
+ },
660
+ "_seconds": 12.4,
661
+ "_ms_per_chunk": 210.6
662
+ },
663
+ "llmlingua2@0.33": {
664
+ "swe_rebench_tool": {
665
+ "n": 11,
666
+ "protected": 290,
667
+ "lost": 199,
668
+ "lost_rate": 0.6862068965517242,
669
+ "json_inputs": 0,
670
+ "json_still_valid": 0,
671
+ "mean_char_ratio": 0.44745477331072575,
672
+ "mean_llm_token_ratio": 0.4046752440325844
673
+ },
674
+ "swe_smith_tool": {
675
+ "n": 6,
676
+ "protected": 209,
677
+ "lost": 116,
678
+ "lost_rate": 0.5550239234449761,
679
+ "json_inputs": 0,
680
+ "json_still_valid": 0,
681
+ "mean_char_ratio": 0.37855999939473106,
682
+ "mean_llm_token_ratio": 0.37122538889428897
683
+ },
684
+ "github_code": {
685
+ "n": 14,
686
+ "protected": 277,
687
+ "lost": 178,
688
+ "lost_rate": 0.6425992779783394,
689
+ "json_inputs": 0,
690
+ "json_still_valid": 0,
691
+ "mean_char_ratio": 0.37766550004700583,
692
+ "mean_llm_token_ratio": 0.3367045773069327
693
+ },
694
+ "oasst2": {
695
+ "n": 1,
696
+ "protected": 4,
697
+ "lost": 4,
698
+ "lost_rate": 1.0,
699
+ "json_inputs": 0,
700
+ "json_still_valid": 0,
701
+ "mean_char_ratio": 0.36752136752136755,
702
+ "mean_llm_token_ratio": 0.27075812274368233
703
+ },
704
+ "fineweb_edu": {
705
+ "n": 12,
706
+ "protected": 80,
707
+ "lost": 29,
708
+ "lost_rate": 0.3625,
709
+ "json_inputs": 0,
710
+ "json_still_valid": 0,
711
+ "mean_char_ratio": 0.3407403459255646,
712
+ "mean_llm_token_ratio": 0.31395025784196623
713
+ },
714
+ "fineweb2": {
715
+ "n": 7,
716
+ "protected": 38,
717
+ "lost": 14,
718
+ "lost_rate": 0.3684210526315789,
719
+ "json_inputs": 0,
720
+ "json_still_valid": 0,
721
+ "mean_char_ratio": 0.45192975999299223,
722
+ "mean_llm_token_ratio": 0.605414245714466
723
+ },
724
+ "toucan_mcp": {
725
+ "n": 5,
726
+ "protected": 87,
727
+ "lost": 59,
728
+ "lost_rate": 0.6781609195402298,
729
+ "json_inputs": 1,
730
+ "json_still_valid": 0,
731
+ "mean_char_ratio": 0.423237618553439,
732
+ "mean_llm_token_ratio": 0.34902219144411106
733
+ },
734
+ "swe_rebench_assistant": {
735
+ "n": 3,
736
+ "protected": 28,
737
+ "lost": 19,
738
+ "lost_rate": 0.6785714285714286,
739
+ "json_inputs": 0,
740
+ "json_still_valid": 0,
741
+ "mean_char_ratio": 0.427098006395479,
742
+ "mean_llm_token_ratio": 0.3630892658066571
743
+ },
744
+ "_all": {
745
+ "n": 59,
746
+ "protected": 1013,
747
+ "lost": 618,
748
+ "lost_rate": 0.6100691016781836,
749
+ "json_inputs": 1,
750
+ "json_still_valid": 0,
751
+ "mean_char_ratio": 0.398272465321647,
752
+ "mean_llm_token_ratio": 0.3814081998561065
753
+ },
754
+ "_seconds": 12.4,
755
+ "_ms_per_chunk": 209.4
756
+ },
757
+ "llmlingua2-large@0.5": {
758
+ "swe_rebench_tool": {
759
+ "n": 11,
760
+ "protected": 290,
761
+ "lost": 163,
762
+ "lost_rate": 0.5620689655172414,
763
+ "json_inputs": 0,
764
+ "json_still_valid": 0,
765
+ "mean_char_ratio": 0.4862299048654031,
766
+ "mean_llm_token_ratio": 0.46746874412517836
767
+ },
768
+ "swe_smith_tool": {
769
+ "n": 6,
770
+ "protected": 209,
771
+ "lost": 44,
772
+ "lost_rate": 0.21052631578947367,
773
+ "json_inputs": 0,
774
+ "json_still_valid": 0,
775
+ "mean_char_ratio": 0.46634522232007836,
776
+ "mean_llm_token_ratio": 0.5330424195197456
777
+ },
778
+ "github_code": {
779
+ "n": 14,
780
+ "protected": 277,
781
+ "lost": 81,
782
+ "lost_rate": 0.2924187725631769,
783
+ "json_inputs": 0,
784
+ "json_still_valid": 0,
785
+ "mean_char_ratio": 0.540475012477371,
786
+ "mean_llm_token_ratio": 0.4740879291770878
787
+ },
788
+ "oasst2": {
789
+ "n": 1,
790
+ "protected": 4,
791
+ "lost": 0,
792
+ "lost_rate": 0.0,
793
+ "json_inputs": 0,
794
+ "json_still_valid": 0,
795
+ "mean_char_ratio": 0.5779092702169625,
796
+ "mean_llm_token_ratio": 0.4657039711191336
797
+ },
798
+ "fineweb_edu": {
799
+ "n": 12,
800
+ "protected": 80,
801
+ "lost": 5,
802
+ "lost_rate": 0.0625,
803
+ "json_inputs": 0,
804
+ "json_still_valid": 0,
805
+ "mean_char_ratio": 0.5524875855166003,
806
+ "mean_llm_token_ratio": 0.49699727481502626
807
+ },
808
+ "fineweb2": {
809
+ "n": 7,
810
+ "protected": 38,
811
+ "lost": 6,
812
+ "lost_rate": 0.15789473684210525,
813
+ "json_inputs": 0,
814
+ "json_still_valid": 0,
815
+ "mean_char_ratio": 0.4768851583633436,
816
+ "mean_llm_token_ratio": 0.4689590342088241
817
+ },
818
+ "toucan_mcp": {
819
+ "n": 5,
820
+ "protected": 87,
821
+ "lost": 33,
822
+ "lost_rate": 0.3793103448275862,
823
+ "json_inputs": 1,
824
+ "json_still_valid": 0,
825
+ "mean_char_ratio": 0.5035957891832148,
826
+ "mean_llm_token_ratio": 0.4585054497428723
827
+ },
828
+ "swe_rebench_assistant": {
829
+ "n": 3,
830
+ "protected": 28,
831
+ "lost": 9,
832
+ "lost_rate": 0.32142857142857145,
833
+ "json_inputs": 0,
834
+ "json_still_valid": 0,
835
+ "mean_char_ratio": 0.5488299090397311,
836
+ "mean_llm_token_ratio": 0.5219003822264692
837
+ },
838
+ "_all": {
839
+ "n": 59,
840
+ "protected": 1013,
841
+ "lost": 341,
842
+ "lost_rate": 0.3366238894373149,
843
+ "json_inputs": 1,
844
+ "json_still_valid": 0,
845
+ "mean_char_ratio": 0.5156555176291175,
846
+ "mean_llm_token_ratio": 0.4838687222835534
847
+ },
848
+ "_seconds": 39.4,
849
+ "_ms_per_chunk": 667.1
850
+ },
851
+ "llmlingua2-large@0.33": {
852
+ "swe_rebench_tool": {
853
+ "n": 11,
854
+ "protected": 290,
855
+ "lost": 232,
856
+ "lost_rate": 0.8,
857
+ "json_inputs": 0,
858
+ "json_still_valid": 0,
859
+ "mean_char_ratio": 0.3212607339625909,
860
+ "mean_llm_token_ratio": 0.288410022984389
861
+ },
862
+ "swe_smith_tool": {
863
+ "n": 6,
864
+ "protected": 209,
865
+ "lost": 111,
866
+ "lost_rate": 0.5311004784688995,
867
+ "json_inputs": 0,
868
+ "json_still_valid": 0,
869
+ "mean_char_ratio": 0.31311735908907634,
870
+ "mean_llm_token_ratio": 0.3429220606121219
871
+ },
872
+ "github_code": {
873
+ "n": 14,
874
+ "protected": 277,
875
+ "lost": 153,
876
+ "lost_rate": 0.5523465703971119,
877
+ "json_inputs": 0,
878
+ "json_still_valid": 0,
879
+ "mean_char_ratio": 0.34945597580550364,
880
+ "mean_llm_token_ratio": 0.28937061972477196
881
+ },
882
+ "oasst2": {
883
+ "n": 1,
884
+ "protected": 4,
885
+ "lost": 1,
886
+ "lost_rate": 0.25,
887
+ "json_inputs": 0,
888
+ "json_still_valid": 0,
889
+ "mean_char_ratio": 0.38461538461538464,
890
+ "mean_llm_token_ratio": 0.2996389891696751
891
+ },
892
+ "fineweb_edu": {
893
+ "n": 12,
894
+ "protected": 80,
895
+ "lost": 19,
896
+ "lost_rate": 0.2375,
897
+ "json_inputs": 0,
898
+ "json_still_valid": 0,
899
+ "mean_char_ratio": 0.36325656006131846,
900
+ "mean_llm_token_ratio": 0.33728651747536625
901
+ },
902
+ "fineweb2": {
903
+ "n": 7,
904
+ "protected": 38,
905
+ "lost": 10,
906
+ "lost_rate": 0.2631578947368421,
907
+ "json_inputs": 0,
908
+ "json_still_valid": 0,
909
+ "mean_char_ratio": 0.2933769356046527,
910
+ "mean_llm_token_ratio": 0.2973858511093756
911
+ },
912
+ "toucan_mcp": {
913
+ "n": 5,
914
+ "protected": 87,
915
+ "lost": 56,
916
+ "lost_rate": 0.6436781609195402,
917
+ "json_inputs": 1,
918
+ "json_still_valid": 0,
919
+ "mean_char_ratio": 0.3086746186794157,
920
+ "mean_llm_token_ratio": 0.27550261465399106
921
+ },
922
+ "swe_rebench_assistant": {
923
+ "n": 3,
924
+ "protected": 28,
925
+ "lost": 15,
926
+ "lost_rate": 0.5357142857142857,
927
+ "json_inputs": 0,
928
+ "json_still_valid": 0,
929
+ "mean_char_ratio": 0.3886568343090659,
930
+ "mean_llm_token_ratio": 0.2838569039655996
931
+ },
932
+ "_all": {
933
+ "n": 59,
934
+ "protected": 1013,
935
+ "lost": 597,
936
+ "lost_rate": 0.5893385982230998,
937
+ "json_inputs": 1,
938
+ "json_still_valid": 0,
939
+ "mean_char_ratio": 0.33579037525945865,
940
+ "mean_llm_token_ratio": 0.3040524277026148
941
+ },
942
+ "_seconds": 35.1,
943
+ "_ms_per_chunk": 595.8
944
+ }
945
+ },
946
+ "agreement": {
947
+ "swe_rebench_tool": {
948
+ "keep_f1": 0.959,
949
+ "acc": 0.928
950
+ },
951
+ "swe_smith_tool": {
952
+ "keep_f1": 0.949,
953
+ "acc": 0.906
954
+ },
955
+ "github_code": {
956
+ "keep_f1": 0.994,
957
+ "acc": 0.993
958
+ },
959
+ "oasst2": {
960
+ "keep_f1": 0.793,
961
+ "acc": 0.832
962
+ },
963
+ "fineweb_edu": {
964
+ "keep_f1": 0.883,
965
+ "acc": 0.803
966
+ },
967
+ "fineweb2": {
968
+ "keep_f1": 0.81,
969
+ "acc": 0.69
970
+ },
971
+ "toucan_mcp": {
972
+ "keep_f1": 0.956,
973
+ "acc": 0.927
974
+ },
975
+ "swe_rebench_assistant": {
976
+ "keep_f1": 0.946,
977
+ "acc": 0.902
978
+ }
979
+ }
980
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:671befd923d19025dbd05e342d35b7b9467ca022f8e2a1b3bb06274045fd8f21
3
+ size 562583392
onnx/model.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe605d35ebe4219c4adff23b6ae6ce139993a67148e4ba78f60cd677c4f20036
3
+ size 563098131
onnx/model_quantized.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a00b6154721359f3786367d07be50e235a072d59887496eb462334c0b599ac48
3
+ size 142501996
onnx/parity.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "sizes_mb": {
3
+ "fp32": 563.1,
4
+ "int8": 142.5
5
+ },
6
+ "fp32": {
7
+ "mean_abs_prob_diff": 0.0,
8
+ "keep_decision_agreement": 1.0,
9
+ "ms_per_chunk": 101.5
10
+ },
11
+ "int8": {
12
+ "mean_abs_prob_diff": 0.03948,
13
+ "keep_decision_agreement": 0.9586,
14
+ "ms_per_chunk": 93.4
15
+ },
16
+ "torch_ms_per_chunk": 120.2
17
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f
3
+ size 34363188
tokenizer_config.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "bos_token": "<bos>",
4
+ "clean_up_tokenization_spaces": false,
5
+ "cls_token": "<bos>",
6
+ "eos_token": "<eos>",
7
+ "extra_special_tokens": [
8
+ "<start_of_turn>",
9
+ "<end_of_turn>"
10
+ ],
11
+ "is_local": false,
12
+ "local_files_only": false,
13
+ "mask_token": "<mask>",
14
+ "model_input_names": [
15
+ "input_ids",
16
+ "attention_mask"
17
+ ],
18
+ "model_max_length": 8192,
19
+ "pad_token": "<pad>",
20
+ "padding_side": "right",
21
+ "sep_token": "<eos>",
22
+ "spaces_between_special_tokens": false,
23
+ "tokenizer_class": "TokenizersBackend",
24
+ "unk_token": "<unk>"
25
+ }
train_meta.json ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "args": {
3
+ "data": "[PosixPath('/mnt/ssd/ctxprune-data/v1')]",
4
+ "labels": "['qwen38_v1set', 'qwen38_v3']",
5
+ "base": "jhu-clsp/mmBERT-small",
6
+ "init_from": "None",
7
+ "out": "/mnt/ssd/ctxprune-data/runs/v1-final-v1v3",
8
+ "resume": "False",
9
+ "epochs": "5.0",
10
+ "lr": "5e-05",
11
+ "batch_size": "32",
12
+ "grad_accum": "1",
13
+ "max_len": "512",
14
+ "no_deletion_frac": "0.5",
15
+ "max_train": "None",
16
+ "no_balance": "False",
17
+ "train_frac": "1.0",
18
+ "device": "None",
19
+ "seed": "0"
20
+ },
21
+ "data_stats": {
22
+ "rows": 6417,
23
+ "filtered": 381,
24
+ "no_deletion_dropped": 587,
25
+ "balanced": {
26
+ "before": 5249,
27
+ "after": 5249,
28
+ "per_source": {
29
+ "fineweb2": 627,
30
+ "fineweb_edu": 708,
31
+ "github_code": 837,
32
+ "oasst2": 512,
33
+ "swe_rebench_assistant": 264,
34
+ "swe_rebench_tool": 743,
35
+ "swe_smith_tool": 829,
36
+ "toucan_mcp": 729
37
+ }
38
+ }
39
+ },
40
+ "train_rows": 5249,
41
+ "val_rows": 94,
42
+ "train_seconds": 305.6,
43
+ "val_metrics": {
44
+ "eval_loss": 0.4691639244556427,
45
+ "eval_accuracy": 0.7759162303664922,
46
+ "eval_keep_precision": 0.7907237403928267,
47
+ "eval_keep_recall": 0.8886156429942419,
48
+ "eval_keep_f1": 0.8368165386353368,
49
+ "eval_drop_precision": 0.7365583770747623,
50
+ "eval_drop_recall": 0.5697355426314057,
51
+ "eval_drop_f1": 0.6424947407499072,
52
+ "eval_macro_f1": 0.7396556396926219,
53
+ "eval_auc": 0.8372793755487984,
54
+ "eval_keep_rate_pred": 0.726624006205158,
55
+ "eval_keep_rate_true": 0.6465774675198759,
56
+ "eval_runtime": 0.5768,
57
+ "eval_samples_per_second": 162.969,
58
+ "eval_steps_per_second": 3.467,
59
+ "epoch": 5.0
60
+ }
61
+ }