MultiBLiMP 2.0 - Subject-Verb Agreement Overview

Subject-Verb agreement prediction through decision trees. Click a language row (or the Results button) to open its create_pairs results below. The small chevron after N Keep (or the Diagnostics button) is separate: it opens just the bucket breakdown, inline, right where those items dropped out of the pipeline.

Main Page Overview

1 language with too sparse a joint-tagging rate to be confident — click to expand

The following languages have enough raw "Yes" samples (≥ 10) but too sparse a joint-tagging rate relative to the full dataset to be statistically confident this isn't incidental tagging (Wilson lower bound below 1%; label distribution shown):

Bulgarian (unk: 3141; no: 24; yes: 18)

Keep threshold: leaf accuracy > 0.95

Coverage 0%100%

Item loss 0%100%

Language

Distribution

Root Acc

Train Acc

Δ Acc

Test Acc

N RAW

N KEEP

N PAIRS

Table columns

Distribution: The language's full Yes / No / unk label mix, before the decision tree drops unk rows. Always ordered Yes → No → unk, coloured to match the tree pages. Hover a bar for exact percentages.
Root Acc: Smoothed accuracy (+0.5 per class) of always predicting the majority class before any splitting — the root node’s own accuracy. The uninformed baseline. Coloured on the coverage scale between 50% and 100% (teal = high).
Train Acc: The decision tree’s accuracy on its own training data: (sum of each leaf’s majority count + 0.5) / (N + 1). Smoothed once for the whole dataset, exactly like Root Acc, so the two are directly comparable and extra leaves are not penalised one by one (the per-leaf acc shown on the tree page is smoothed per leaf, for the keep test, and so does not average to this). Same colour scale as Root Acc.
Δ Acc: Train Acc minus Root Acc, in percentage points; never negative. Coloured on the coverage scale (teal = good) by the share of the root’s remaining error the tree removed, i.e. gain ÷ (100% − Root Acc), clamped to 0–100%. +1 point off a root accuracy of 98% is a much bigger win than +1 point off a root accuracy of 60%.
Test Acc: Raw accuracy of the same tree on the held-out split it was not fit on. Compare with Train Acc to spot overfitting: a large gap means the tree learned its training rows more than the pattern. Coloured on the same 50–100% scale, and faded (with a tooltip) when the test split has fewer than 10 rows, since that figure is then too noisy to mean much. Blank when the model has no recorded test split.
N RAW: Candidate rows before the leaf-confidence filter is applied. Not coloured. There's no cheap per-language reference point (like treebank size) to scale it against, and not informative enough to be worth adding one for. A language with more than zero but fewer than 10 is left out of this table entirely and listed instead, with its full label distribution, in the "too few or uninformative agreement labels" note above — still real data, just too little of it for this language & condition to draw much of a conclusion from.
N KEEP: Of those, the rows kept after the leaf-confidence filter (leaf accuracy > 0.95). The ones actually attempted. Coloured on the coverage scale, as a fraction of N RAW — faded (with a hover tooltip) when N RAW is below 10, since that fraction is too noisy to mean much at that size (2/2 reads as a perfect 100% off pure small-sample luck). The colour still shows, just dimmed, rather than disappearing outright. A language whose strict pass alone wasn't enough and got a laxer, depth-aware retry (sva_trees.second_chance) gets a "retried" tag under its name — N KEEP then reflects that retry's own per-leaf thresholds, whether or not it produced any pairs.
N PAIRS: Rows that ended up as a correctly re-inflected minimal pair. Coloured on the coverage scale, as a fraction of N KEEP, with the same under-10 fade as N KEEP above (checked against N KEEP this time, not N RAW). Clickable when example rows are available — opens the same examples modal as the bucket breakdown.

Results (below-row panel)

# UD candidates (raw): Same value as N RAW above. Candidate rows before the leaf-confidence filter.
# UD candidates (kept): Same value as N KEEP above. The rows kept after that filter.
# forms of interest: Distinct Aux/nsubj forms eligible for this prediction target at all, whether or not they ended up labelled "Yes".
% covered by UM: Share of those forms found directly in UniMorph's inflection tables.
% covered by UM+UD: Share covered once the UD-derived fallback is added on top of UniMorph.
# UM lemmas / forms: Size of the UniMorph inflection table available for this language.
# valid from multi: Minimal pairs that came from a row with more than one candidate swap form, as a share of all pairs. Clickable when example rows are available.
Feature distribution: Tally of attempted feature-value swaps, as "FROM → TO: count", across every attempt that reached the comparison step.
P(value|value): Of the attempts that started at that value, the fraction where re-inflecting still landed back on the same value. I.e. how often the inflector failed to actually change it.

Diagnostics (bucket breakdown)

Any bucket with at least one row (here, and N PAIRS / "valid from multi" in the Results panel above) has an examples link showing real sample rows — reconstructed sentences, feature values, and a treebank link — for that language and outcome.

no match: No candidate swap form was found for this row at all.
no inflection: A swap form was targeted, but the inflector had nothing to offer for it.
same inflection: The re-inflected form came back identical to the original surface form.
same feature: The re-inflected form's feature value still overlapped the original. The swap didn't actually change it.
undefined feature: The target feature was undefined or missing for this form.
ambiguous subject: The row's subject couldn't be uniquely resolved.
conflicting features: The reinflected form directly opposes the original on another feature (or adds an argument slot or feature dimension), so it isn't a minimal pair.

Numerator / denominator reference

What each percentage above is actually a fraction of. "Row" scope means once per candidate row; "item" scope means once per attempted swap_form. A single row can produce more than one item, so row- and item-scoped percentages use different denominators. extra_pairs isn't shown as a percentage in the table, but shares the same item-scoped pool as the four buckets above.

Column	Numerator	Denominator	Denominator comes from	Scope	Why
% covered by UM	`num_covered_um`	# forms of interest	`num_forms_of_interest`	forms	Distinct Aux/nsubj forms found in UniMorph alone.
% covered by UM+UD	`num_covered_um_ud`	# forms of interest	`num_forms_of_interest`	forms	Distinct Aux/nsubj forms found via UniMorph or the UD-derived fallback.
% no match	# no match (`no_candidates`)	# UD candidates (kept)	`n_keep`	row	Fires once per candidate row, in place of the swap_form loop. The two are mutually exclusive.
% no inflection	# no inflection (`no_inflections`)	# UD candidates (kept)	`n_keep`	row	Same as "no match": fires instead of entering the swap_form loop.
% same inflection	# same inflection (`same_forms`)	swap_items	`items_seen − no_match − no_inflection`	item	Fires once per attempted swap_form; a row can yield more than one item.
% same feature	# same feature (`same_features`)	swap_items	`items_seen − no_match − no_inflection`	item	Fires once per attempted swap_form.
% undefined feature	# undefined feature (`undefined_features`)	swap_items	`items_seen − no_match − no_inflection`	item	Fires once per attempted swap_form.
% conflicting features	# conflicting features (`conflicting_features`)	swap_items	`items_seen − no_match − no_inflection`	item	Fires once per attempted swap_form. Always 0 for NPA.
% ambiguous subject	# ambiguous subject (`ambiguous_subjects`)	swap_items	`items_seen − no_match − no_inflection`	item	Fires once per attempted swap_form.
(not shown) extra_pairs	`extra_pairs`	swap_items	`items_seen − no_match − no_inflection`	item	Would collect otherwise-valid swaps (same eligibility as correct_swaps) that get capped once their FROM→TO combination hits the max_num_of_pairs limit. Always 0 today since create_pairs is never called with that limit set.

Colour scales

Coverage: Vermilion (red) at 0%, violet around the middle, teal at 100%. Used where a high percentage is good (UM/UM+UD coverage, Root/Train Accuracy between 50% and 100%, N KEEP as a fraction of N RAW, N PAIRS as a fraction of N KEEP, and Δ Accuracy as a share of the root’s remaining error).
Item loss: Teal at 0%, violet around the middle, vermilion at 100%, same scale mirrored. Used in the bucket breakdown (a high percentage means more rows dropped out at that step).