Skip to content

SkillRater: Untangling Capabilities in Multimodal Data

Decomposing data quality into independent capability dimensions

Useful for what?

The input is shared. What teaches each rater is not.

Shared training examples · same SigLIP features to every rater
Its own validation objectiveSEED-Bench · A-OKVQA · VSR
Its own validation objectiveTextVQA · ChartQA · RD-Tablebench
Its own validation objectiveM3Exam · MathVista · AI2D
Visual understanding validation teaches this rater. The other objectives stay separate.
Select a rater to trace its feedback connection. Conceptual overview: shared inputs, independently learned scoring functions. Their selections are combined later—not their validation objectives.

A rater rϕr_\phi assigns each training example zz a scalar weight w=rϕ(z)∈[0,1]w = r_\phi(z) \in [0,1]. The goal is to find weights that improve performance on a held-out validation set. DataRater frames this as a bilevel optimization:

The model learns from data. The rater learns from the result.

Rater

Scores examples

Fixed

Model

Learns from weighted data

Updating

Validation

Tests one capability

Fixed

The validation signal returns to the rater.

The rater weights the examples. Those weighted losses update the model; the rater stays fixed.

One schematic cycle, not an experiment replay. Repeat independently for visual understanding, OCR and STEM; each rater receives only its own validation signal.
The bilevel objective
min⁡ϕ  Lval ⁣(θ⋆(ϕ)⏟post-inner model)⏟validation objectives.t.θ⋆(ϕ)=arg⁡min⁡θ  Ez∼Dtrain[rϕ(z)⏟sample weight  ℓ(θ;z)⏞per-sample loss]⏟weighted training loss\min_{\colorbox{#e5eeff}{$\scriptstyle\phi$}}\; \underbrace{ \colorbox{#f2e9ff}{$\mathcal{L}_{\mathrm{val}}$}\!\Bigl( \underbrace{\colorbox{#e6f4f5}{$\theta^{\star}(\phi)$}}_{\color{#2e60ad}\text{post-inner model}} \Bigr) }_{\color{#7c45b4}\text{validation objective}} \quad\mathrm{s.t.}\quad \colorbox{#e6f4f5}{$\theta^{\star}(\phi)$} =\arg\min_{\colorbox{#e6f4f5}{$\scriptstyle\theta$}}\; \underbrace{ \mathbb{E}_{z\sim\colorbox{#e9f4e6}{$\scriptstyle\mathcal{D}_{\mathrm{train}}$}} \Bigl[ \underbrace{\colorbox{#fcf1cf}{$r_\phi(z)$}}_{\color{#cd6f2c}\text{sample weight}} \;\overbrace{\colorbox{#fbfadf}{$\ell(\theta;z)$}}^{\color{#986e23}\text{per-sample loss}} \Bigr] }_{\color{#5b8841}\text{weighted training loss}}

Scroll to read the full equation.

The outer objective minimizes validation loss on the model state θ⋆(ϕ)\colorbox{#e2f5f5}{$\theta^{\star}(\phi)$} that results from training with rater-weighted losses. In practice, θ⋆(ϕ)\colorbox{#e2f5f5}{$\theta^{\star}(\phi)$} is approximated by running a finite number of inner-loop SGD steps.

In the inner loop, we initialize the base model parameters θ(0)=θ\theta^{(0)} = \theta and run SS weighted SGD steps. For a minibatch B\mathcal{B} and learning rate α\alpha:

Inner loop · train the model
θ(s+1)=θ(s)−α⏟inner step size  ∇θ(s)(1∣B∣∑z∈Brϕ(z) ℓ(θ(s);z))⏟weighted microbatch gradient,s∈{0,…,S−1}\colorbox{#e6f4f5}{$\theta^{(s+1)}$} =\colorbox{#e6f4f5}{$\theta^{(s)}$} -\underbrace{\colorbox{#f5efff}{$\alpha$}}_{\color{#7c45b4}\text{inner step size}} \;\underbrace{ \nabla_{\colorbox{#e6f4f5}{$\scriptstyle\theta^{(s)}$}} \Biggl( \frac{1}{\lvert\colorbox{#e9f4e6}{$\mathcal{B}$}\rvert} \sum_{z\in\colorbox{#e9f4e6}{$\scriptstyle\mathcal{B}$}} \colorbox{#fcf1cf}{$r_\phi(z)$}\, \colorbox{#fbfadf}{$\ell(\theta^{(s)};z)$} \Biggr) }_{\color{#5b8841}\text{weighted microbatch gradient}}, \quad s\in\{0,\ldots,S-1\}

Scroll to read the full equation.

Each gradient step is weighted by the rater's scores — samples the rater considers high-value contribute more to the update. The post-inner-loop state is θ(S)\colorbox{#e2f5f5}{$\theta^{(S)}$}.

The rater parameters ϕ\colorbox{#e2eeff}{$\phi$} are then updated to improve held-out performance on this post-inner-loop state:

Outer loop · improve the rater
ϕ←ϕ−η⏟outer step size  ∇ϕLval(θ(S)(ϕ))⏟meta-gradient on rater\colorbox{#e5eeff}{$\phi$} \leftarrow\colorbox{#e5eeff}{$\phi$} -\underbrace{\colorbox{#f5efff}{$\eta$}}_{\color{#7c45b4}\text{outer step size}} \;\underbrace{ \nabla_{\colorbox{#e5eeff}{$\scriptstyle\phi$}} \colorbox{#f2e9ff}{$\mathcal{L}_{\mathrm{val}}$} \Bigl(\colorbox{#e6f4f5}{$\theta^{(S)}(\phi)$}\Bigr) }_{\color{#2e60ad}\text{meta-gradient on rater}}

Scroll to read the full equation.

This is the meta-gradient: it tells the rater how to adjust its scores so that the resulting weighted training produces a model that does better on validation.

SkillRater's key extension is running this bilevel loop independently for each capability c∈{1,…,C}c \in \{1, \dots, C\}:

One validation objective per capability
min⁡ϕc  Lval(c)(θ(S)(ϕc))s.t.θ(S)(ϕc)=InnerLoop ⁣(θ, rϕc, Dtrain)\min_{\colorbox{#e5eeff}{$\scriptstyle\phi_c$}}\; \colorbox{#f2e9ff}{$\mathcal{L}_{\mathrm{val}}^{(c)}$} \Bigl(\colorbox{#e6f4f5}{$\theta^{(S)}(\phi_c)$}\Bigr) \quad\mathrm{s.t.}\quad \colorbox{#e6f4f5}{$\theta^{(S)}(\phi_c)$} =\texttt{InnerLoop}\!\bigl( \theta,\,r_{\phi_c},\, \colorbox{#e9f4e6}{$\mathcal{D}_{\mathrm{train}}$} \bigr)

Scroll to read the full equation.

where Lval(c)\colorbox{#f2e9ff}{$\mathcal{L}_{\mathrm{val}}^{(c)}$} is a capability-specific validation objective and rϕc\colorbox{#ffeed1}{$r_{\phi_c}$} is the corresponding rater. The inner loop uses the same shared training data, but each rater's outer-loop gradients come only from its own capability benchmarks. This is the structural difference from monolithic scoring: the Visual Understanding rater never sees OCR validation signal, and vice versa, so the raters are free to learn genuinely independent notions of quality.

Many selections. One copy.

Illustrative samples: a dot means selected. Keep a sample if any rater selects it.

Tightening selection
Broad earlyMore selective later
Sample
Union
8 of 9 retained
Visual
OCR
STEM
D

Kept once

Sample D is selected by Visual and OCR and STEM. It enters the union once, even when more than one rater selects it.

Schematic selections from independent rankings—not measured scores. Tightening is illustrative, not the paper’s exact thresholds or quadratic schedule.

Held-out capability benchmarks

2B parameters · Mammoth-10M · higher is better

Capability benchmark percentages. SkillRater gains are percentage-point differences from DataRater.
CategoryMammoth (Baseline)DataRaterSkillRater CombinedSkillRater
20.1%20.9%22.4%23.6%+2.7 pp
48.1%48.5%47.3%50.1%+1.6 pp
65.9%68.3%71.4%71.5%+3.2 pp
44.7%45.9%47.0%48.4%+2.5 pp

OCR

Inspect a row to compare its exact scores

DataRater

48.5%

+1.6 pp

SkillRater

50.1%

All source values retained. Bold marks the best score in each row. Gains are absolute percentage-point differences (pp) vs. DataRater. Swipe the table to compare every method.

Learn at 1B. Transfer to 2B.

Reconstructed charts · marker values estimated from the source image

CurriculumBaseline

1B Model

0.420.440.460.480.500.5225k50k75k100kTraining StepsVQA Average

Curriculum ≈0.510Baseline ≈0.476

2B Model

0.610.620.630.640.650.660.6725k50k75k100kTraining StepsVQA Average

Curriculum ≈0.671Baseline ≈0.648

Hover, tap or use arrow keys on either plot to compare the same checkpoint. The y-scales differ. The curriculum is below baseline at the first two checkpoints and above it at the last two. Values marked ≈ are image estimates, not original measurements. Brackets show Curriculum minus Baseline in the plotted score units.
Estimated values & source

Estimated from the visible marker centers in the original SkillRater scale-transfer figure. Rounded to 0.001; no hidden checkpoints or intermediate values were reconstructed.

ModelStepsCurriculum ≈Baseline ≈
1B25k0.4130.464
1B50k0.4710.493
1B75k0.5200.456
1B100k0.5100.476
2B25k0.6100.619
2B50k0.6240.633
2B75k0.6600.632
2B100k0.6710.648

Different raters find different signal.

Estimated scatter positions · exact printed correlations and PCA percentages

Select a rater to emphasize it; select again to restore all.

Top-10% selections in PCA space

-0.3-0.3-0.2-0.2-0.1-0.10.00.00.10.10.20.20.30.3VQAOCRSTEMPC1 (36.6% var.)PC2 (33.8% var.)

Hover, tap or use arrow keys to inspect a reconstructed point.

Pearson correlation matrix

Exact printed pairwise Pearson correlations for the full corpus. Blank diagonal values were not printed.
VQAOCRSTEM
VQANot printed
OCRNot printed
STEMNot printed

Select a pair to locate both raters in the scatter. Values are exact.

Full corpus

PCA variance

051015202530354035.5%PC132.9%PC231.6%PC333.3%Var. Explained (%)

Three components, each close to one-third. These are not rater categories.

eff. dim = 2.99

The original scatter depicts a random 50k subsample. This reconstruction estimates only discernible dot centers; overlapping or very faint points may be omitted. It is not the original sample dataset, and no statistics are computed from it. Correlations, variance percentages and effective dimensionality reproduce the exact printed values. Axis and bar percentages remain as printed in the source.
Reconstruction method & estimated point coordinates

Colored dot centers were located in the source image, calibrated against the PC1/PC2 axis ticks and rounded to 0.001. Legend dots, text and directional arrows were excluded. The faint background reproduces only distinguishable centers—no random points were added. Arrow directions were traced from the same image.

Image-estimated point positions, not original data.
PointPC1 ≈PC2 ≈
VQA-1−0.157−0.009
VQA-2−0.131−0.029
VQA-3−0.138−0.042
VQA-4−0.165−0.067
VQA-5−0.118−0.079
VQA-6−0.098−0.083
VQA-7−0.038−0.086
VQA-8−0.258−0.093
VQA-9−0.189−0.097
VQA-10−0.123−0.098
VQA-11−0.037−0.100
VQA-12−0.341−0.102
VQA-13−0.010−0.106
VQA-14−0.108−0.109
VQA-15−0.174−0.109
VQA-16−0.053−0.130
VQA-17−0.070−0.158
VQA-18−0.105−0.178
VQA-19−0.032−0.198
OCR-10.1710.002
OCR-20.137−0.024
OCR-30.196−0.026
OCR-40.215−0.037
OCR-50.079−0.057
OCR-60.109−0.062
OCR-70.087−0.096
OCR-80.084−0.106
OCR-90.053−0.146
OCR-100.067−0.151
OCR-110.104−0.198
OCR-120.085−0.226
STEM-1−0.0690.189
STEM-20.0420.179
STEM-30.1580.178
STEM-4−0.1010.173
STEM-5−0.0130.141
STEM-60.0310.139
STEM-70.0730.136
STEM-8−0.1210.136
STEM-90.0020.116
STEM-10−0.0270.106
STEM-11−0.0490.094
STEM-12−0.0550.091
STEM-130.0010.086
STEM-14−0.0080.073
STEM-150.0250.071
STEM-160.0530.067
STEM-17−0.0170.065
STEM-18−0.1150.056
STEM-190.0250.045
STEM-200.0100.033
STEM-210.0350.029
STEM-220.0900.019
STEM-230.0310.017
STEM-240.042−0.004
STEM-25−0.042−0.014

SkillRater Quality Examples

SkillRater Quality Examples · original

Original comparison of retained and filtered samples for STEM, OCR and Visual raters, including all six source questions, answer excerpts and rater scores.

Enlarge to read the original labels; scroll to inspect the full image.

Original comparison of retained and filtered samples for STEM, OCR and Visual raters, including all six source questions, answer excerpts and rater scores.

STEM Rater

Reason about physical properties

RetainedHigh Quality+0.515
A person wearing a thick brown coat.

Q: Considering the coat, what season is it most appropriate for, and what characteristics make it suitable?

A: The coat is most appropriate for winter. Its thick fabric is ideal for insulation...

FilteredLow Quality−0.425
An English-language speaking exercise on an evaluation slide.

Q: What educational concept is being assessed by the speaking exercise in the image?

A: The concept appears to be conversational skills in English, the ability to respond...

OCR Rater

Read embedded text in context

RetainedHigh Quality+0.479
A presentation slide headed Galdos in „Madrid“, with a photograph of a memorial plaque.

Q: What does the use of quotation marks around "Madrid" imply about the presenter's intention?

A: The quotation marks suggest Madrid is significant in the life of Benito Pérez Galdós...

FilteredLow Quality−0.612
A digitally edited blue face with one large eye.

Q: Considering the digital manipulation, what can we infer about the skills required?

A: We infer high skill in photo editing. Such a composite requires color matching...

Visual Rater

Ground reasoning in the scene

RetainedHigh Quality+0.408
An aerial view of a harbor with a lighthouse at the end of a breakwater.

Q: Considering the design and location of the lighthouse, what might be its historical significance and its role in maritime navigation?

A: The lighthouse on the breakwater likely serves as a navigational aid for vessels. Its historical significance may stem from ensuring maritime...

FilteredLow Quality−0.581
A round wooden table with outward-angled legs and central cross support.

Q: Considering the table's design, what factors contribute to its stability?

A: Primary factors include the outward leg angle and central cross support...

Compare retained and filtered within one capability. All six original photos, scores and truncated answers are retained. Scores are reproduced as shown, not probabilities. Swipe or choose a capability above.
@article{sahi2026skillrater,
  title={SkillRater: Untangling Capabilities in Multimodal Data},
  author={Sahi, Naveen and Dohmann, Jeremy and Aghajanyan, Armen and Shrivastava, Akshat},
  year={2026}
}

We believe in building in the open and empowering the community. You can start creating with Isaac today.