MultiBLiMP 2.0 - Subject-Verb Number Agreement Overview

Table columns

Distribution

The language's full Yes / No / unk label mix, before the decision tree drops unk rows. Always ordered Yes → No → unk, coloured to match the tree pages. Hover a bar for exact percentages.

Root Acc

Smoothed accuracy (+0.5 per class) of always predicting the majority class before any splitting — the root node’s own accuracy. The uninformed baseline. Coloured on the coverage scale between 50% and 100% (teal = high).

Train Acc

The decision tree’s accuracy on its own training data: (sum of each leaf’s majority count + 0.5) / (N + 1). Smoothed once for the whole dataset, exactly like Root Acc, so the two are directly comparable and extra leaves are not penalised one by one (the per-leaf acc shown on the tree page is smoothed per leaf, for the keep test, and so does not average to this). Same colour scale as Root Acc.

Δ Acc

Train Acc minus Root Acc, in percentage points; never negative. Coloured on the coverage scale (teal = good) by the share of the root’s remaining error the tree removed, i.e. gain ÷ (100% − Root Acc), clamped to 0–100%. +1 point off a root accuracy of 98% is a much bigger win than +1 point off a root accuracy of 60%.

Test Acc

Raw accuracy of the same tree on the held-out split it was not fit on. Compare with Train Acc to spot overfitting: a large gap means the tree learned its training rows more than the pattern. Coloured on the same 50–100% scale, and faded (with a tooltip) when the test split has fewer than 10 rows, since that figure is then too noisy to mean much. Blank when the model has no recorded test split.

N RAW

Candidate rows before the leaf-confidence filter is applied. Not coloured. There's no cheap per-language reference point (like treebank size) to scale it against, and not informative enough to be worth adding one for. A language with more than zero but fewer than 10 is left out of this table entirely and listed instead, with its full label distribution, in the "too few or uninformative agreement labels" note above — still real data, just too little of it for this language & condition to draw much of a conclusion from.

N KEEP

Of those, the rows kept after the leaf-confidence filter (leaf accuracy > 0.95). The ones actually attempted. Coloured on the coverage scale, as a fraction of N RAW — faded (with a hover tooltip) when N RAW is below 10, since that fraction is too noisy to mean much at that size (2/2 reads as a perfect 100% off pure small-sample luck). The colour still shows, just dimmed, rather than disappearing outright. A language whose strict pass alone wasn't enough and got a laxer, depth-aware retry (sva_trees.second_chance) gets a "retried" tag under its name — N KEEP then reflects that retry's own per-leaf thresholds, whether or not it produced any pairs.

N PAIRS

Rows that ended up as a correctly re-inflected minimal pair. Coloured on the coverage scale, as a fraction of N KEEP, with the same under-10 fade as N KEEP above (checked against N KEEP this time, not N RAW). Clickable when example rows are available — opens the same examples modal as the bucket breakdown.

Results (below-row panel)

# UD candidates (raw)

Same value as N RAW above. Candidate rows before the leaf-confidence filter.

# UD candidates (kept)

Same value as N KEEP above. The rows kept after that filter.

# forms of interest

Distinct Verb/nsubj forms eligible for this prediction target at all, whether or not they ended up labelled "Yes".

% covered by UM

Share of those forms found directly in UniMorph's inflection tables.

% covered by UM+UD

Share covered once the UD-derived fallback is added on top of UniMorph.

# UM lemmas / forms

Size of the UniMorph inflection table available for this language.

# valid from multi

Minimal pairs that came from a row with more than one candidate swap form, as a share of all pairs. Clickable when example rows are available.

Feature distribution

Tally of attempted feature-value swaps, as "FROM → TO: count", across every attempt that reached the comparison step.

P(value|value)

Of the attempts that started at that value, the fraction where re-inflecting still landed back on the same value. I.e. how often the inflector failed to actually change it.

Diagnostics (bucket breakdown)

Any bucket with at least one row (here, and N PAIRS / "valid from multi" in the Results panel above) has an examples link showing real sample rows — reconstructed sentences, feature values, and a treebank link — for that language and outcome.

no match

No candidate swap form was found for this row at all.

no inflection

A swap form was targeted, but the inflector had nothing to offer for it.

same inflection

The re-inflected form came back identical to the original surface form.

same feature

The re-inflected form's feature value still overlapped the original. The swap didn't actually change it.

undefined feature

The target feature was undefined or missing for this form.

ambiguous subject

The row's subject couldn't be uniquely resolved.

conflicting features

The reinflected form directly opposes the original on another feature (or adds an argument slot or feature dimension), so it isn't a minimal pair.

Numerator / denominator reference

What each percentage above is actually a fraction of. "Row" scope means once per candidate row; "item" scope means once per attempted swap_form. A single row can produce more than one item, so row- and item-scoped percentages use different denominators. extra_pairs isn't shown as a percentage in the table, but shares the same item-scoped pool as the four buckets above.

Colour scales

Coverage

Vermilion (red) at 0%, violet around the middle, teal at 100%. Used where a high percentage is good (UM/UM+UD coverage, Root/Train Accuracy between 50% and 100%, N KEEP as a fraction of N RAW, N PAIRS as a fraction of N KEEP, and Δ Accuracy as a share of the root’s remaining error).

Item loss

Teal at 0%, violet around the middle, vermilion at 100%, same scale mirrored. Used in the bucket breakdown (a high percentage means more rows dropped out at that step).