MultiBLiMP v2 - Language Overview

Subject-Auxiliary agreement prediction through decision trees. Click a language row (or the Results button) to open its create_pairs results below. The small chevron after N Keep (or the Diagnostics button) is separate: it opens just the bucket breakdown, inline, right where those items dropped out of the pipeline.

Overview Entropy Type:

Languages shown in green have categorical agreement throughout — all samples share the label Yes.

The following languages were omitted for having too few or uninformative agreement labels (label distribution shown): Abkhaz (35), Afrikaans (1672), Akuntsu (12), Albanian (89), Alemannic (666), Amharic (331), Armenian (1500), Aromanian (4), Assamese (7), Azerbaijani (17), Bambara (947), Basque (5336), Bavarian (577), Bengali (5), Bokota (150), Brahui (8), Buryat (103), Cantonese (188), Cappadocian (53), Catalan (13336), Central Kurdish (1), Chinese (4897), Chintang (18), Chukchi (24), Classical Armenian (1212), Classical Chinese (2681), Coptic (718), Danish (3627), Dutch (10411), English (14834), Erzya (199), Esperanto (65), Estonian (6811), Faroese (846), Finnish (5085), Galician (540), Georgian (1088), German (76851), Gheg (218), Greek (1113), Gujarati (89), Haitian Creole (3560), Highland Puebla Nahuatl (51), Hittite (5), Hungarian (126), Ika (125), Indonesian (1945), Italian (14187), Japanese (5631), Javanese (165), Kaapor (6), Kadiweu (2), Karelian (26), Karo (42), Kazakh (186), Khoekhoe (2447), Kiche (80), Komi Permyak (13), Komi Zyrian (96), Korean (3239), Kyrgyz (42), Latgalian (3), Ligurian (180), Livvi (15), Low Saxon (868), Luxembourgish (17), Madi (8), Makurap (1), Malayalam (27), Maltese (395), Manx (34), Middle French (2383), Moksha (30), Munduruku (8), Naga (88), Naija (6371), Neapolitan (6), Nenets (3), Nepali (3), Nheengatu (268), North Sami (1022), Northern Kurdish (255), Northwest Gbaya (3), Norwegian Bokmaal (10133), Norwegian Nynorsk (9935), Occitan (273), Odia (9), Old English (8), Old French (5299), Old Georgian (50), Old Irish (7), Old Occitan (646), Old Turkish (4), Ottoman Turkish (304), Paumari (7), Persian (11837), Pesh (14), Phrygian (15), Portuguese (17655), Romanian (5660), Ruuli (64), Scottish Gaelic (140), Shanghainese (182), Sicilian (146), Sinhala (19), Skolt Sami (44), Slovenian (6959), South Levantine Arabic (5), Spanish (12004), Swedish (6767), Tagalog (2), Tatar (18), Teko (8), Thai (2253), Tswana (8), Turkish (4029), Umbrian (9), Upper Sorbian (187), Uyghur (496), Uzbek (104), Veps (18), Vietnamese (471), Warlpiri (46), Welsh (546), Western Armenian (2373), Western Sierra Puebla Nahuatl (141), Wolof (2118), Xavante (52), Xibe (30), Yakut (1), Yiddish (2290), Yoruba (217), Zaar (264).

Keep threshold: entropy < 0.12

Coverage 0%100%

Item loss 0%100%

Language

Distribution

Base Entropy

Reduced Entropy

Δ Entropy

DT Acc%

N RAW

N KEEP

N PAIRS

Table columns

Distribution: The language's full Yes / No / unk label mix, before the decision tree drops unk rows. Always ordered Yes → No → unk, coloured to match the tree pages. Hover a bar for exact percentages.
Base Entropy: Entropy of the target label distribution before any splitting. The uninformed baseline. Coloured on the item-loss scale (teal = low = good), relative to 1 bit or the highest base entropy in this table, whichever is larger.
Reduced Entropy: Entropy remaining after fitting the decision tree; lower means the tree explains more of the outcome. Coloured on the item-loss scale (teal = low = good), on the same 1-bit-or-highest-base-entropy scale as Base Entropy above, so the two columns are directly comparable -- teal here means a language landed on a meaningfully low absolute entropy, not just a low entropy relative to its own (possibly already-low) starting point.
Δ Entropy: Base Entropy minus Reduced Entropy. How much uncertainty the tree removes. Coloured on the coverage scale (teal = good), by what fraction of the base entropy that delta represents, not the raw number. 0.3 off a base of 0.4 is a much bigger win than 0.3 off a base of 2.0.
DT Acc%: The decision tree's accuracy scored on its own training data. Coloured on the coverage scale (teal = high = good).
N RAW: Candidate rows before the entropy/leaf-confidence filter is applied. Not coloured. There's no cheap per-language reference point (like treebank size) to scale it against, and not informative enough to be worth adding one for.
N KEEP: Of those, the rows kept after the leaf-confidence filter (entropy < 0.12). The ones actually attempted. Coloured on the coverage scale, as a fraction of N RAW — faded (with a hover tooltip) when N RAW is below 10, since that fraction is too noisy to mean much at that size (2/2 reads as a perfect 100% off pure small-sample luck). The colour still shows, just dimmed, rather than disappearing outright.
N PAIRS: Rows that ended up as a correctly re-inflected minimal pair. Coloured on the coverage scale, as a fraction of N KEEP, with the same under-10 fade as N KEEP above (checked against N KEEP this time, not N RAW). Clickable when example rows are available — opens the same examples modal as the bucket breakdown.

Results (below-row panel)

# UD candidates (raw): Same value as N RAW above. Candidate rows before the entropy/leaf-confidence filter.
# UD candidates (kept): Same value as N KEEP above. The rows kept after that filter.
# forms of interest: Distinct Aux/nsubj forms eligible for this prediction target at all, whether or not they ended up labelled "Yes".
% covered by UM: Share of those forms found directly in UniMorph's inflection tables.
% covered by UM+UD: Share covered once the UD-derived fallback is added on top of UniMorph.
# UM lemmas / forms: Size of the UniMorph inflection table available for this language.
# valid from multi: Minimal pairs that came from a row with more than one candidate swap form, as a share of all pairs. Clickable when example rows are available.
Feature distribution: Tally of attempted feature-value swaps, as "FROM → TO: count", across every attempt that reached the comparison step.
P(value|value): Of the attempts that started at that value, the fraction where re-inflecting still landed back on the same value. I.e. how often the inflector failed to actually change it.

Diagnostics (bucket breakdown)

Any bucket with at least one row (here, and N PAIRS / "valid from multi" in the Results panel above) has an examples link showing real sample rows — reconstructed sentences, feature values, and a treebank link — for that language and outcome.

no match: No candidate swap form was found for this row at all.
no inflection: A swap form was targeted, but the inflector had nothing to offer for it.
same inflection: The re-inflected form came back identical to the original surface form.
same feature: The re-inflected form's feature value still overlapped the original. The swap didn't actually change it.
undefined feature: The target feature was undefined or missing for this form.
ambiguous subject: The row's subject couldn't be uniquely resolved.

Numerator / denominator reference

What each percentage above is actually a fraction of. "Row" scope means once per candidate row; "item" scope means once per attempted swap_form. A single row can produce more than one item, so row- and item-scoped percentages use different denominators. extra_pairs isn't shown as a percentage in the table, but shares the same item-scoped pool as the four buckets above.

Column	Numerator	Denominator	Denominator comes from	Scope	Why
% covered by UM	`num_covered_um`	# forms of interest	`num_forms_of_interest`	forms	Distinct Aux/nsubj forms found in UniMorph alone.
% covered by UM+UD	`num_covered_um_ud`	# forms of interest	`num_forms_of_interest`	forms	Distinct Aux/nsubj forms found via UniMorph or the UD-derived fallback.
% no match	# no match (`no_candidates`)	# UD candidates (kept)	`n_keep`	row	Fires once per candidate row, in place of the swap_form loop. The two are mutually exclusive.
% no inflection	# no inflection (`no_inflections`)	# UD candidates (kept)	`n_keep`	row	Same as "no match": fires instead of entering the swap_form loop.
% same inflection	# same inflection (`same_forms`)	swap_items	`items_seen − no_match − no_inflection`	item	Fires once per attempted swap_form; a row can yield more than one item.
% same feature	# same feature (`same_features`)	swap_items	`items_seen − no_match − no_inflection`	item	Fires once per attempted swap_form.
% undefined feature	# undefined feature (`undefined_features`)	swap_items	`items_seen − no_match − no_inflection`	item	Fires once per attempted swap_form.
% ambiguous subject	# ambiguous subject (`ambiguous_subjects`)	swap_items	`items_seen − no_match − no_inflection`	item	Fires once per attempted swap_form.
(not shown) extra_pairs	`extra_pairs`	swap_items	`items_seen − no_match − no_inflection`	item	Would collect otherwise-valid swaps (same eligibility as correct_swaps) that get capped once their FROM→TO combination hits the max_num_of_pairs limit. Always 0 today since create_pairs is never called with that limit set.

Colour scales

Coverage: Vermilion (red) at 0%, violet around the middle, teal at 100%. Used where a high percentage is good (UM/UM+UD coverage, DT Acc%, N KEEP as a fraction of N RAW, N PAIRS as a fraction of N KEEP, and Δ Entropy as a fraction of Base Entropy).
Item loss: Teal at 0%, violet around the middle, vermilion at 100%, same scale mirrored. Used in the bucket breakdown (a high percentage means more rows dropped out at that step) and for Base/Reduced Entropy, scaled against the highest base entropy shown in this table rather than a fixed 0–100 range.