Teaching VLMs to think visually

Abstract

The success of reinforcement learning on verifiable rewards (RLVR) is constrained by the reasoning behaviors already present in the base model, yet it remains poorly understood which initial reasoning strategies, which we call "reasoning priors", make an effective starting point, particularly for vision-language models whose priors are inherited from a text-only pre-training phase. We study this question directly: we teach a 9B VLM to play the card game "Set!" by supervised fine-tuning on programmatically constructed chains of thought that isolate individual reasoning strategies (textual vs. visually grounded reasoning, with and without backtracking), then apply RL on outcome-level rewards to each variant.

We find that (i) backtracking/verification in the SFT prior is essential for stable RL convergence; (ii) visually grounded reasoning trains more stably, prevents hallucinations in the chain of thought, and measurably shifts attention mass toward image tokens; and (iii) checkpoint ability is a poor gating criterion for RL readiness — the grounded-backtracking checkpoint enters RL at 18.4% out of distribution against 92.2% for its textual counterpart, yet erases the deficit within ten RL steps and finishes ahead on every evaluation.

1. Introduction

Around a year and a half ago, with the release of "reasoning" models like o1 [1] and DeepSeek's R1 [2], LLMs started to get really good at complex tasks. Reasoning behaviors are largely already present in pre-trained models [3], but they can be further refined via a pipelined approach of (optional) SFT on demonstrations of step-by-step problem solving, followed by RL on outcome-level rewards in which the model is allowed to explore how to combine reasoning strategies in novel ways to solve problems [4,5].

Because rollouts in RL are sampled from the model directly, the success of an RL training run is constrained by the set of base reasoning capabilities already present in the model. That said, it is not perfectly well understood which sets of initial reasoning strategies, which we call "reasoning priors", serve as an optimal starting point for reinforcement learning of reasoning models.

Recent work by Gandhi et al. [6] demonstrated that model quality alone is not a primary determinant of downstream performance post-RL; rather, the presence or absence of certain effective reasoning behaviors determines the final outcome of RL. In particular, they identify backtracking and verification as the two most important behaviors that make a base model an effective starting point for RL on text-based math problem solving.

2. Reasoning for VLMs

Since most reasoning behaviors arise from the text pre-training phase, and VLMs are typically trained by combining a pre-trained text-only transformer with a pre-trained vision transformer [7], VLMs are at a disadvantage in creating visually faithful reasoning chains when reasoning over image-based problems. Two challenges are distinctive to the visual setting:

Priors. Most reasoning priors come from the pre-training corpus, but a VLM's language backbone was pre-trained without images — pre-training corpora may simply not contain good priors for visual problem solving.
Attention. Chain-of-thought is performed in text tokens, and a common attention failure mode is image↔image and text↔text attention — as the chain grows, the reasoning becomes unmoored from the image.

DeepSeek-AI [8] demonstrated through synthetically constructed datasets that visual grounding can be elicited during ordinary VQA problem solving — interleaving spatial references (points, boxes, paths) with the textual chain — though it remained unclear what benefits this process yields. Our experiments address that gap by measuring the downstream effect of grounding on RL training dynamics and attention behavior.

Concretely, we ask three questions. (1) Which reasoning strategies make effective SFT baselines for RL? (2) Does grounding the reasoning process improve outcomes? (3) Is model ability the right criterion for selecting a checkpoint to begin RL from?

3. Experimental setup

We teach our 9B VLM base model to play the popular card game "Set!" by reasoning through the visual stimulus via programmatically constructed chains of thought. In Set!, each card varies along four attributes (count, color, shading, shape); a valid set is three cards in which every attribute is either all-same or all-different. The task is a natural probe for visual problem solving: it requires fine-grained perception, combinatorial search, and verification — and validity is programmatically checkable, making it ideal for RLVR (Figure 1).

Test any three cards

3 valid Sets among 220 possible triples

numberall same

3 · 3 · 3

shapeall same

squiggle · squiggle · squiggle

colorall different

green · red · purple

shadingall different

empty · striped · solid

Valid Set — every property is all same or all different

Tap or press Enter on cards to choose a triple; a fourth pick starts a new one. The hand is the one in the original Figure 1; verdicts are computed from its cards.

Figure 1 — A generated 4×3 Set! hand; valid sets outlined.

Test any three cards

3 valid Sets among 220 possible triples

numberall same

3 · 3 · 3

shapeall same

squiggle · squiggle · squiggle

colorall different

green · red · purple

shadingall different

empty · striped · solid

Valid Set — every property is all same or all different

Tap or press Enter on cards to choose a triple; a fourth pick starts a new one. The hand is the one in the original Figure 1; verdicts are computed from its cards.

Figure 1 — A generated 4×3 Set! hand; valid sets outlined.

3.1 Data generation and SFT

We programmatically generate 20,000 random 4×3 Set! hands, each paired with generated chain-of-thought rationales in a variety of strategies. Rationale variants are seed-matched — the same hands appear in every dataset — so that datasets differ only in reasoning strategy: textual reasoning (card attributes named in prose) vs. grounded reasoning (explicit references to card positions in the image), each with or without backtracking (proposing a candidate set, verifying it attribute-by-attribute, and abandoning failed candidates). We SFT the base VLM separately on each dataset, producing one checkpoint per reasoning strategy (Figure 3).

One hand, four ways to reason about it

Variant e · Grounded + backtrackingset_game_train_grounded_with_backtracking

References

Answer

every reference carries image coordinates17 anchors

Search path2·3·74·10·114·6·86·7·117·9·12

There are cards arranged in a grid. Let me point to and describe each card.

- Card 1 (row 1, col 1):

2 purple diamonds, empty(26,26) (255,335)

- Card 2 (row 1, col 2):

2 purple diamonds, solid(293,26) (522,335)

- Card 3 (row 1, col 3):

1 red oval, solid(560,26) (789,335)

- Card 4 (row 1, col 4):

1 green diamond, solid(827,26) (1056,335)

- Card 5 (row 2, col 1):

3 red ovals, solid(26,367) (255,676)

- Card 6 (row 2, col 2):

3 green diamonds, striped(293,367) (522,676)

- Card 7 (row 2, col 3):

3 green squiggles, empty(560,367) (789,676)

- Card 8 (row 2, col 4):

1 green diamond, striped(827,367) (1056,676)

- Card 9 (row 3, col 1):

3 red squiggles, striped(26,708) (255,1017)

- Card 10 (row 3, col 2):

2 green squiggles, solid(293,708) (522,1017)

- Card 11 (row 3, col 3):

3 green ovals, solid(560,708) (789,1017)

- Card 12 (row 3, col 4):

3 purple squiggles, solid(827,708) (1056,1017)

I'll consider a few candidate triple(s) and verify each:

- Candidate 1: candidate set Card 2Card 3Card 7

Number: 2, 1, 3 (all different)

Shape: diamond, oval, squiggle (all different)

Color: purple, red, green (all different)

Shading: solid, solid, empty (mixed)

The shading property is mixed (neither all-same nor all-different) → INVALID — not a Set. I'll reject this candidate.

- Candidate 2: candidate set Card 4Card 10Card 11

Number: 1, 2, 3 (all different)

Shape: diamond, squiggle, oval (all different)

Color: green, green, green (all same)

Shading: solid, solid, solid (all same)

All four properties are all-same or all-different → VALID Set.

- Candidate 3: candidate set Card 4Card 6Card 8

Number: 1, 3, 1 (mixed)

Shape: diamond, diamond, diamond (all same)

Color: green, green, green (all same)

Shading: solid, striped, striped (mixed)

The number, shading property is mixed (neither all-same nor all-different) → INVALID — not a Set. I'll reject this candidate.

- Candidate 4: candidate set Card 6Card 7Card 11

Number: 3, 3, 3 (all same)

Shape: diamond, squiggle, oval (all different)

Color: green, green, green (all same)

Shading: striped, empty, solid (all different)

All four properties are all-same or all-different → VALID Set.

- Candidate 5: candidate set Card 7Card 9Card 12

Number: 3, 3, 3 (all same)

Shape: squiggle, squiggle, squiggle (all same)

Color: green, red, purple (all different)

Shading: empty, striped, solid (all different)

All four properties are all-same or all-different → VALID Set.

Among the valid candidates I'll commit to Candidate 5 (cards 7, 9, 12) as my answer.

45/45

Textual rationales name cards in words; grounded ones attach a box to every card they mention (a point_box: two corner pixels in the image). Backtracking proposes a triple, checks each property and drops it if any is mixed. Switch either strategy at any point; the hand and candidates stay the same. Hover a tag to find its box. Rationales reconstructed from the training templates for the Figure 1 hand, with verdicts computed from the cards; not a model rollout.

Figure 2 — Seed-matched rationale variants for one hand — textual vs. visually grounded reasoning, each with and without backtracking. One strategy per SFT dataset.

Training pipeline

Five runs · one variable

Isaac — 9B base

image / video grounding + QA · reasoning data · no grounded reasoning

Generate 5 SFT datasets · seed-matched Set! hands · only the rationale style differs

No CoT
Textual
Grounded
Textual + backtracking
Grounded + backtracking

SFT — 200 steps · ×5 models

95% Set! data · 5% original pretraining mixture

Evaluate

in-domain: 4×3 Set! hands · OOD: 3×3 Set! hands

results → Figure 5

RL — DAPO, 60 steps · ×5 models

each model trained on a separate, held-out set of Set! hands

Evaluate

in-domain: 4×3 Set! hands · OOD: 3×3 Set! hands

results → Figures 7–8

Each coloured track is one model. Hue marks the rationale format (grey none, blue textual, orange grounded); dashed tracks have no backtracking, solid tracks do. Hover, focus or tap a dataset to follow its run.

Figure 3 — Training pipeline. One dataset is generated per reasoning strategy; the base VLM is SFT’d on each; RL is then applied to every SFT checkpoint under an identical reward.

One hand, four ways to reason about it

Variant e · Grounded + backtrackingset_game_train_grounded_with_backtracking

References

Answer

every reference carries image coordinates17 anchors

Search path2·3·74·10·114·6·86·7·117·9·12

There are cards arranged in a grid. Let me point to and describe each card.

- Card 1 (row 1, col 1):

2 purple diamonds, empty(26,26) (255,335)

- Card 2 (row 1, col 2):

2 purple diamonds, solid(293,26) (522,335)

- Card 3 (row 1, col 3):

1 red oval, solid(560,26) (789,335)

- Card 4 (row 1, col 4):

1 green diamond, solid(827,26) (1056,335)

- Card 5 (row 2, col 1):

3 red ovals, solid(26,367) (255,676)

- Card 6 (row 2, col 2):

3 green diamonds, striped(293,367) (522,676)

- Card 7 (row 2, col 3):

3 green squiggles, empty(560,367) (789,676)

- Card 8 (row 2, col 4):

1 green diamond, striped(827,367) (1056,676)

- Card 9 (row 3, col 1):

3 red squiggles, striped(26,708) (255,1017)

- Card 10 (row 3, col 2):

2 green squiggles, solid(293,708) (522,1017)

- Card 11 (row 3, col 3):

3 green ovals, solid(560,708) (789,1017)

- Card 12 (row 3, col 4):

3 purple squiggles, solid(827,708) (1056,1017)

I'll consider a few candidate triple(s) and verify each:

- Candidate 1: candidate set Card 2Card 3Card 7

Number: 2, 1, 3 (all different)

Shape: diamond, oval, squiggle (all different)

Color: purple, red, green (all different)

Shading: solid, solid, empty (mixed)

The shading property is mixed (neither all-same nor all-different) → INVALID — not a Set. I'll reject this candidate.

- Candidate 2: candidate set Card 4Card 10Card 11

Number: 1, 2, 3 (all different)

Shape: diamond, squiggle, oval (all different)

Color: green, green, green (all same)

Shading: solid, solid, solid (all same)

All four properties are all-same or all-different → VALID Set.

- Candidate 3: candidate set Card 4Card 6Card 8

Number: 1, 3, 1 (mixed)

Shape: diamond, diamond, diamond (all same)

Color: green, green, green (all same)

Shading: solid, striped, striped (mixed)

The number, shading property is mixed (neither all-same nor all-different) → INVALID — not a Set. I'll reject this candidate.

- Candidate 4: candidate set Card 6Card 7Card 11

Number: 3, 3, 3 (all same)

Shape: diamond, squiggle, oval (all different)

Color: green, green, green (all same)

Shading: striped, empty, solid (all different)

All four properties are all-same or all-different → VALID Set.

- Candidate 5: candidate set Card 7Card 9Card 12

Number: 3, 3, 3 (all same)

Shape: squiggle, squiggle, squiggle (all same)

Color: green, red, purple (all different)

Shading: empty, striped, solid (all different)

All four properties are all-same or all-different → VALID Set.

Among the valid candidates I'll commit to Candidate 5 (cards 7, 9, 12) as my answer.

45/45

Figure 2 — Rationale variants for one hand — textual vs. visually grounded reasoning, each with and without backtracking. One strategy per SFT dataset.

Training pipeline

Five runs · one variable

Isaac — 9B base

image / video grounding + QA · reasoning data · no grounded reasoning

Generate 5 SFT datasets · seed-matched Set! hands · only the rationale style differs

No CoT
Textual
Grounded
Textual + backtracking
Grounded + backtracking

SFT — 200 steps · ×5 models

95% Set! data · 5% original pretraining mixture

Evaluate

in-domain: 4×3 Set! hands · OOD: 3×3 Set! hands

results → Figure 5

RL — DAPO, 60 steps · ×5 models

each model trained on a separate, held-out set of Set! hands

Evaluate

in-domain: 4×3 Set! hands · OOD: 3×3 Set! hands

results → Figures 7–8

Figure 3 — Training pipeline. One dataset is generated per reasoning strategy; the base VLM is SFT’d on each; RL is then applied to every SFT checkpoint under an identical reward.

3.2 RL and reward design

We post-train each SFT checkpoint with RL on outcome-level rewards [9]. Reward design proved delicate: pure exact-match rewards are too sparse for early training, while pure dense rewards admit reward hacking via partially correct answers. We explored a number of reward formulations, and only one led to convergence — dense partial credit with an all-or-nothing completeness bonus:

R = \frac{1}{N} \sum_{i = 1}^{N} max_{j} IoU (ŝ_{i}, s_{j}) + λ \cdot 1 [every valid set recovered exactly]

R = \frac{1}{N} \sum_{i = 1}^{N} max_{j} IoU (ŝ_{i}, s_{j})

+ λ \cdot 1 [every valid set recovered exactly]

Dense partial credit · mean best IoU of each mentioned candidateCompleteness bonus · all or nothing

where ŝᵢ are the N candidate sets mentioned in the response, sⱼ the ground-truth sets, and the bonus λ is awarded only on exact recovery of all valid sets.

Reward anatomy

threshold 0.5 · em_weight 0.3

GT1IoU 0.78

GT2IoU 0.56

GT3IoU 0

Select a dashed box to move, resize or refit it.

Add a guess on

1Hungarian matching · IoU > 0.5 to count

GT1 (card 7) ⇔ P1 · IoU 0.782 ✓ matched
GT2 (card 9) ⇔ P2 · IoU 0.557 ✓ matched (barely)
GT3 (card 12) ⇔ ∅ · IoU 0.000 × unmatched
P3 (card 11) ⇔ ∅ · spurious × extra pred

2Dense term · partial credit

num = 0.782 + 0.557 = 1.339

denom = max(M=3, N=3) = 3

base = num / denom = 0.446

3Exact-match bonus

complete = (all GT matched) AND (M = N)

= False → bonus forfeited

4Reward

r = 0.7 × 0.446 + 0.3 × 0

0.312

dense, up to 0.7bonus, 0.3 or 0

Matched boxes still earn partial credit, which gives early training a signal. The 0.3 bonus waits for an exact, complete answer.

IoU (intersection over union) measures how well two boxes overlap, from 0 (apart) to 1 (identical); matching pairs each true card with at most one guess. Drag a dashed box or its corners, tap a card to add a guess, or select a box and use the arrow keys (+/− resizes, Delete removes). Scored live with the Figure 4 worked-example rules: ground truth is the Set on cards 7, 9 and 12, matching counts a pair only when IoU > 0.5, base = Σ matched IoU / max(M, N), and r = 0.7 × base + 0.3 × complete. The worked example applies the equation's λ as this em_weight of 0.3.

Figure 4 — Reward anatomy: a worked example. Each mentioned candidate is scored by IoU against its best-matching ground-truth set; the mean provides dense signal, and the completeness bonus preserves pressure toward exact solutions.

Reward anatomy

threshold 0.5 · em_weight 0.3

GT1IoU 0.78

GT2IoU 0.56

GT3IoU 0

Select a dashed box to move, resize or refit it.

Add a guess on

1Hungarian matching · IoU > 0.5 to count

GT1 (card 7) ⇔ P1 · IoU 0.782 ✓ matched
GT2 (card 9) ⇔ P2 · IoU 0.557 ✓ matched (barely)
GT3 (card 12) ⇔ ∅ · IoU 0.000 × unmatched
P3 (card 11) ⇔ ∅ · spurious × extra pred

2Dense term · partial credit

num = 0.782 + 0.557 = 1.339

denom = max(M=3, N=3) = 3

base = num / denom = 0.446

3Exact-match bonus

complete = (all GT matched) AND (M = N)

= False → bonus forfeited

4Reward

r = 0.7 × 0.446 + 0.3 × 0

0.312

dense, up to 0.7bonus, 0.3 or 0

Matched boxes still earn partial credit, which gives early training a signal. The 0.3 bonus waits for an exact, complete answer.

4. Results

4.1 SFT: backtracking is essential

After SFT alone, backtracking is the difference between competence and failure: with it, the model reaches 93.5% (textual) and 94.6% (grounded) in-distribution, versus at most 39.9% for any variant without it. At SFT time the strongest overall prior is textual backtracking, which leads across both evaluations — 93.5% in-distribution and 92.2% on out-of-distribution 3×3 hands — while grounded backtracking, despite matching it in-distribution, collapses to 18.4% OOD.

SFT eval results — Set! 9B (iou_exact_match)

thinking_effort=HIGH · variants match panels (a–e)

Hover, focus or tap a row to read it in plain terms.

SFT eval results, Set! 9B, iou_exact_match, thinking_effort=HIGH. Columns sort by value.
	Change from in-distribution to OOD, percentage points
0.0%	±0.0 points	0.0%
0.5%	+0.5 points	1.0%
36.8%	+16.8 points	53.6%
39.9%	+16.2 points	56.1%
93.5%	−1.3 points	92.2%
94.6%	−76.2 points	18.4%

Backtracking dominates either modality alone (d, e ≥ 92% in-dist). Grounded + backtracking (e) collapses on 3×3 OOD: 94.6% → 18.4%.

Hover, focus or tap a row; sort by either column, or open Chart to see both splits on one axis. In-distribution: 4×3 hands, like the training data. OOD: 3×3 hands, a layout not used in training. iou_exact_match is the exact-match score used throughout; higher is better. Values as published.

Figure 5 — SFT evaluation (9B, iou_exact_match). Backtracking dominates either modality alone; textual backtracking leads across both settings, while grounded backtracking collapses on 3×3 OOD (94.6% in-distribution → 18.4%).

Qualitative analysis of the errors reveals a sharper contrast than the scores suggest. Without backtracking, both modalities mislabel invalid sets as valid. With backtracking, the failures diverge by modality: the textual model fails by attempting to form sets out of cards that do not exist — hallucinating a 12th card in a 9-card hand — whereas the grounded model remains faithful to the image.

How each prior fails out of distribution

Set! 3×3 OOD

No backtracking

Misreads while checking

check failscommit sound

Textual

Reads the numbers 3, 3, 1 as “all different”, so an invalid triple passes.

Number3, 3, 1 · model says “all different” — actually mixed
Shapesquiggle, diamond, oval · all different
Colorgreen, red, purple · all different
Shadingstriped, striped, striped · all same

passes the check, so it is committed

AnswerSet 1 (cards 2, 5, 6)not a Set

Grounded

Reads the numbers 3, 2, 2 as “all different”, so an invalid triple passes.

Number3, 2, 2 · model says “all different” — actually mixed
Other attributes elided in the excerpt

passes the check, so it is committed

AnswerSet 1 (cards 2, 5, 8)not a Set

With backtracking

Checks correctly, then ignores the verdict

check soundcommit fails

Textual + backtracking

Names a card 12 in a nine-card hand, rejects the candidate, then commits to it.

Proposes cards 2, 6 and 12. A 3×3 hand has no card 12.

Its own check:Candidate 3 rejectedINVALID, rejected

commits to it anyway

AnswerCandidate 3just rejected

Grounded + backtracking

Describes the cards cleanly and rejects every candidate, then commits to one anyway.

Its own checks:

Candidate 1 rejected (elided in the excerpt)Candidate 2 rejected (elided in the excerpt)Candidate 3 rejected

no candidate passed — commits anyway

AnswerCandidate 3 (cards 1, 4, 7)

Excerpts from OOD (3×3) eval transcripts; hands differ across panels and “…” marks elided text. Red ✗ marks and the summaries above are annotations, not model output. The slot grid is a schematic; the transcript's actual hand isn't shown.

Figure 6 — Post-SFT failure modes on out-of-distribution 3×3 hands. Top: without backtracking, verification errors. Bottom: with backtracking, the commit step ignores verification — textual hallucinates a nonexistent card; grounded keeps perception clean but commits to a rejected candidate.

SFT eval results — Set! 9B (iou_exact_match)

thinking_effort=HIGH · variants match panels (a–e)

Hover, focus or tap a row to read it in plain terms.

SFT eval results, Set! 9B, iou_exact_match, thinking_effort=HIGH. Columns sort by value.
	Change from in-distribution to OOD, percentage points
0.0%	±0.0 points	0.0%
0.5%	+0.5 points	1.0%
36.8%	+16.8 points	53.6%
39.9%	+16.2 points	56.1%
93.5%	−1.3 points	92.2%
94.6%	−76.2 points	18.4%

Backtracking dominates either modality alone (d, e ≥ 92% in-dist). Grounded + backtracking (e) collapses on 3×3 OOD: 94.6% → 18.4%.

How each prior fails out of distribution

Set! 3×3 OOD

No backtracking

Misreads while checking

check failscommit sound

Textual

Reads the numbers 3, 3, 1 as “all different”, so an invalid triple passes.

Number3, 3, 1 · model says “all different” — actually mixed
Shapesquiggle, diamond, oval · all different
Colorgreen, red, purple · all different
Shadingstriped, striped, striped · all same

passes the check, so it is committed

AnswerSet 1 (cards 2, 5, 6)not a Set

Grounded

Reads the numbers 3, 2, 2 as “all different”, so an invalid triple passes.

Number3, 2, 2 · model says “all different” — actually mixed
Other attributes elided in the excerpt

passes the check, so it is committed

AnswerSet 1 (cards 2, 5, 8)not a Set

With backtracking

Checks correctly, then ignores the verdict

check soundcommit fails

Textual + backtracking

Names a card 12 in a nine-card hand, rejects the candidate, then commits to it.

Proposes cards 2, 6 and 12. A 3×3 hand has no card 12.

Its own check:Candidate 3 rejectedINVALID, rejected

commits to it anyway

AnswerCandidate 3just rejected

Grounded + backtracking

Describes the cards cleanly and rejects every candidate, then commits to one anyway.

Its own checks:

Candidate 1 rejected (elided in the excerpt)Candidate 2 rejected (elided in the excerpt)Candidate 3 rejected

no candidate passed — commits anyway

AnswerCandidate 3 (cards 1, 4, 7)

4.2 RL: the grounded deficit reverses

RL on checkpoints without backtracking blows up: reward collapses, completion lengths spike, and entropy degenerates. Among the backtracking checkpoints, grounded backtracking enters RL with a severe out-of-distribution deficit — 18.4% vs. 92.2% for textual backtracking — and erases it almost immediately: within ten RL steps it draws level OOD, and it finishes the run ahead on both evaluations while textual backtracking plateaus. The checkpoint with the worst overall performance after SFT was thus able to overcome its deficits after RL and eventually become the best-performing checkpoint — its reasoning priors were simply better suited to effective exploration during the RL phase.

Held-out evaluation during RL

above-random iou_exact_match, %

In distribution

set_game · HIGH

Out of distribution

set_game_3×3 · HIGH

Show the values as a table

Figure 8 values (digitized) · set_game · HIGH (above-random iou_exact_match, %)
Step	d · Textual + backtracking	b · Textual	c · Grounded	e · Grounded + backtracking
0	≈92.7	35.0	38.4	95.0
10	≈94.9	35.0	12.5	93.0
20	≈95.0	35.5	7.0	95.0
30	93.8	32.5	2.0	96.5
40	93.0	17.5	1.5	98.0
50	93.5	≈1.0	1.0	98.0
60	92.0	≈1.0	1.0	98.0

Figure 8 values (digitized) · set_game_3×3 · HIGH (above-random iou_exact_match, %)
Step	d · Textual + backtracking	b · Textual	c · Grounded	e · Grounded + backtracking
0	92.0	52.2	54.9	16.6
10	≈93.0	54.0	21.0	93.0
20	≈93.5	50.1	14.5	95.6
30	93.8	48.5	4.0	96.5
40	93.7	42.9	0.2	96.5
50	92.5	3.5	0.2	96.5
60	91.5	≈2.9	0.2	96.9

In distribution means 4×3 hands like the training data; out of distribution means 3×3 hands, a layout the models never trained on. Hover or drag across a chart, or focus it and use the arrow keys, to read every variant at one step; choose a legend entry or group to trace it. Values are digitized from the published plot (marker centres, ≈±1% of the axis); hollow markers and ≈ mark points that sit under another series in the source. Metric: above-random iou_exact_match, which differs from the raw percentages in the SFT table.

Figure 8 — RL evaluation (iou_exact_match above random). Out of distribution (bottom), grounded backtracking recovers from 18.4% to parity within ten steps and finishes ahead, while no-backtracking variants collapse.

RL training dynamics by SFT prior

EMA α = 0.15, as published

train/reward (mean)

mean

iou_exact_match (mean across rollouts)

mean, 0–1

completions/mean_length

tokens

entropy

nats

Show the values as a table

Figure 7 values (digitized) · train/reward (mean) (train/reward)
Step	d · Textual + backtracking	b · Textual	c · Grounded	e · Grounded + backtracking
1	0.32	0.22	0.24	0.80
5	0.33	≈0.23	0.23	0.79
10	0.34	≈0.22	0.22	0.78
15	0.35	≈0.22	0.22	0.80
20	0.36	≈0.22	0.22	0.82
25	0.37	≈0.22	0.22	0.83
30	0.37	≈0.22	0.22	0.85
35	0.36	≈0.20	0.21	0.85
40	0.35	0.19	0.21	0.84
45	0.33	0.15	0.22	0.85
50	0.30	0.11	0.23	0.86
55	0.29	0.09	0.24	0.85
60	0.29	0.08	0.24	0.86

Figure 7 values (digitized) · iou_exact_match (mean across rollouts) (iou_exact_match)
Step	d · Textual + backtracking	b · Textual	c · Grounded	e · Grounded + backtracking
1	0.170	≈0.021	0.021	0.660
5	0.170	≈0.034	0.021	0.650
10	0.180	0.041	0.021	0.640
15	0.181	≈0.039	0.021	0.660
20	0.191	≈0.034	0.020	0.680
25	0.191	≈0.034	0.021	0.701
30	0.191	≈0.020	0.020	0.729
35	0.180	≈0.015	0.015	0.739
40	0.160	≈0.010	0.010	0.740
45	0.131	≈0.008	0.008	0.730
50	0.102	≈0.006	0.006	0.730
55	0.091	≈0.005	0.005	0.739
60	0.090	≈0.006	0.006	0.740

Figure 7 values (digitized) · completions/mean_length (mean completion length, tokens)
Step	d · Textual + backtracking	b · Textual	c · Grounded	e · Grounded + backtracking
1	1,199	650	561	1,598
5	1,230	650	566	1,620
10	1,259	650	574	1,651
15	1,300	650	581	1,660
20	1,350	650	586	1,680
25	1,400	≈612	585	1,701
30	1,408	≈585	585	1,720
35	1,420	≈588	588	1,740
40	1,451	≈583	583	1,751
45	1,501	799	585	1,761
50	≈1,509	≈1,500	585	1,769
55	1,509	2,298	586	1,769
60	1,490	2,802	585	1,761

Figure 7 values (digitized) · entropy (entropy, nats)
Step	d · Textual + backtracking	b · Textual	c · Grounded	e · Grounded + backtracking
1	≈0.032	≈0.033	0.066	0.014
5	0.033	0.035	0.069	0.016
10	0.032	0.036	0.070	0.016
15	0.031	0.034	0.070	0.015
20	≈0.031	≈0.032	0.068	0.014
25	≈0.030	≈0.031	0.063	0.013
30	≈0.029	≈0.030	0.055	0.013
35	0.028	≈0.028	0.046	0.012
40	≈0.027	0.027	0.038	0.012
45	≈0.026	≈0.031	0.031	0.012
50	0.025	≈0.031	0.029	0.011
55	0.025	≈0.027	0.028	0.011
60	≈0.024	≈0.025	0.027	0.011

Reward is what RL optimizes; completion length is response length in tokens; entropy measures how spread out the model's token choices are. A run that blows up shows reward collapsing, lengths spiking and entropy degenerating. Hover or drag across any chart, or focus the charts and use the arrow keys, to read all four metrics at one step; choose a legend entry or group to trace it through every panel. Values are digitized from the published plot (marker centres, ≈±1% of each axis); hollow markers and ≈ mark points that sit under another series in the source.

Figure 7 — RL training dynamics by SFT prior (reward, IoU, completion length, entropy). Variants without backtracking destabilize and collapse.

RL training dynamics by SFT prior

EMA α = 0.15, as published

train/reward (mean)

mean

iou_exact_match (mean across rollouts)

mean, 0–1

completions/mean_length

tokens

entropy

nats

Show the values as a table

Figure 7 values (digitized) · train/reward (mean) (train/reward)
Step	d · Textual + backtracking	b · Textual	c · Grounded	e · Grounded + backtracking
1	0.32	0.22	0.24	0.80
5	0.33	≈0.23	0.23	0.79
10	0.34	≈0.22	0.22	0.78
15	0.35	≈0.22	0.22	0.80
20	0.36	≈0.22	0.22	0.82
25	0.37	≈0.22	0.22	0.83
30	0.37	≈0.22	0.22	0.85
35	0.36	≈0.20	0.21	0.85
40	0.35	0.19	0.21	0.84
45	0.33	0.15	0.22	0.85
50	0.30	0.11	0.23	0.86
55	0.29	0.09	0.24	0.85
60	0.29	0.08	0.24	0.86

Figure 7 values (digitized) · iou_exact_match (mean across rollouts) (iou_exact_match)
Step	d · Textual + backtracking	b · Textual	c · Grounded	e · Grounded + backtracking
1	0.170	≈0.021	0.021	0.660
5	0.170	≈0.034	0.021	0.650
10	0.180	0.041	0.021	0.640
15	0.181	≈0.039	0.021	0.660
20	0.191	≈0.034	0.020	0.680
25	0.191	≈0.034	0.021	0.701
30	0.191	≈0.020	0.020	0.729
35	0.180	≈0.015	0.015	0.739
40	0.160	≈0.010	0.010	0.740
45	0.131	≈0.008	0.008	0.730
50	0.102	≈0.006	0.006	0.730
55	0.091	≈0.005	0.005	0.739
60	0.090	≈0.006	0.006	0.740

Figure 7 values (digitized) · completions/mean_length (mean completion length, tokens)
Step	d · Textual + backtracking	b · Textual	c · Grounded	e · Grounded + backtracking
1	1,199	650	561	1,598
5	1,230	650	566	1,620
10	1,259	650	574	1,651
15	1,300	650	581	1,660
20	1,350	650	586	1,680
25	1,400	≈612	585	1,701
30	1,408	≈585	585	1,720
35	1,420	≈588	588	1,740
40	1,451	≈583	583	1,751
45	1,501	799	585	1,761
50	≈1,509	≈1,500	585	1,769
55	1,509	2,298	586	1,769
60	1,490	2,802	585	1,761

Figure 7 values (digitized) · entropy (entropy, nats)
Step	d · Textual + backtracking	b · Textual	c · Grounded	e · Grounded + backtracking
1	≈0.032	≈0.033	0.066	0.014
5	0.033	0.035	0.069	0.016
10	0.032	0.036	0.070	0.016
15	0.031	0.034	0.070	0.015
20	≈0.031	≈0.032	0.068	0.014
25	≈0.030	≈0.031	0.063	0.013
30	≈0.029	≈0.030	0.055	0.013
35	0.028	≈0.028	0.046	0.012
40	≈0.027	0.027	0.038	0.012
45	≈0.026	≈0.031	0.031	0.012
50	0.025	≈0.031	0.029	0.011
55	0.025	≈0.027	0.028	0.011
60	≈0.024	≈0.025	0.027	0.011

Figure 7 — RL training dynamics by SFT prior (reward, IoU, completion length, entropy). Variants without backtracking destabilize and collapse.

Held-out evaluation during RL

above-random iou_exact_match, %

In distribution

set_game · HIGH

Out of distribution

set_game_3×3 · HIGH

Show the values as a table

Figure 8 values (digitized) · set_game · HIGH (above-random iou_exact_match, %)
Step	d · Textual + backtracking	b · Textual	c · Grounded	e · Grounded + backtracking
0	≈92.7	35.0	38.4	95.0
10	≈94.9	35.0	12.5	93.0
20	≈95.0	35.5	7.0	95.0
30	93.8	32.5	2.0	96.5
40	93.0	17.5	1.5	98.0
50	93.5	≈1.0	1.0	98.0
60	92.0	≈1.0	1.0	98.0

Figure 8 values (digitized) · set_game_3×3 · HIGH (above-random iou_exact_match, %)
Step	d · Textual + backtracking	b · Textual	c · Grounded	e · Grounded + backtracking
0	92.0	52.2	54.9	16.6
10	≈93.0	54.0	21.0	93.0
20	≈93.5	50.1	14.5	95.6
30	93.8	48.5	4.0	96.5
40	93.7	42.9	0.2	96.5
50	92.5	3.5	0.2	96.5
60	91.5	≈2.9	0.2	96.9

4.3 Attention: grounding moves mass back onto the image

To probe why grounding helps, we measure cross-segment attention during generation. Text-only reasoning progressively shifts attention mass away from image tokens as the chain grows — consistent with the detachment failure mode of §2 — whereas grounded reasoning shifts attention mass toward the image tokens it references (Figure 9).

9B Set! 3×3 — cross-segment attention

Each row is a group of tokens doing the looking; each column is what they look at. A cell is the share of that row's attention landing on that column. The orange-outlined cells are attention from the reasoning and the answer onto the image.

Pretrained

Attention mass for Pretrained. Rows attend from (query), columns are attended to (key).
	prompt	image	CoT	answer
prompt	not shown	not shown	not shown	not shown
image		not shown	not shown	not shown
CoT				not shown
answer

SFT+RL (textual w/ backtracking)

Attention mass for SFT+RL (textual w/ backtracking). Rows attend from (query), columns are attended to (key).
	prompt	image	CoT	answer
prompt	not shown	not shown	not shown	not shown
image		not shown	not shown	not shown
CoT				not shown
answer

SFT+RL (grounded w/ backtracking)

Attention mass for SFT+RL (grounded w/ backtracking). Rows attend from (query), columns are attended to (key).
	prompt	image	CoT	answer
prompt	not shown	not shown	not shown	not shown
image		not shown	not shown	not shown
CoT				not shown
answer

0.00.85attention mass (mean over heads + full-attn layers)

CoT tokens looking at image tokens: 0.053 pretrained → 0.039 textual (−0.013) → 0.056 grounded (+0.003).

Attention on the image, side by side

The orange-outlined cells as bars on one axis from zero: small shares, so read the change, not the size.

Hover, tap or use the arrow keys on a cell to compare it across the three models; Tab moves between them. Values as published: mean over heads and full-attention layers; dashed cells are not shown in the source. Changes marked ≈ are computed from the rounded values; the image-column changes are the published ones.

Figure 9 — Cross-segment attention (9B, 3×3 Set!). Grounded reasoning allocates substantially more attention mass to image tokens during the chain of thought than textual reasoning.

9B Set! 3×3 — cross-segment attention

Pretrained

Attention mass for Pretrained. Rows attend from (query), columns are attended to (key).
	prompt	image	CoT	answer
prompt	not shown	not shown	not shown	not shown
image		not shown	not shown	not shown
CoT				not shown
answer

SFT+RL (textual w/ backtracking)

Attention mass for SFT+RL (textual w/ backtracking). Rows attend from (query), columns are attended to (key).
	prompt	image	CoT	answer
prompt	not shown	not shown	not shown	not shown
image		not shown	not shown	not shown
CoT				not shown
answer

SFT+RL (grounded w/ backtracking)

Attention mass for SFT+RL (grounded w/ backtracking). Rows attend from (query), columns are attended to (key).
	prompt	image	CoT	answer
prompt	not shown	not shown	not shown	not shown
image		not shown	not shown	not shown
CoT				not shown
answer

0.00.85attention mass (mean over heads + full-attn layers)

CoT tokens looking at image tokens: 0.053 pretrained → 0.039 textual (−0.013) → 0.056 grounded (+0.003).

Attention on the image, side by side

The orange-outlined cells as bars on one axis from zero: small shares, so read the change, not the size.

Figure 9 — Cross-segment attention (9B, 3×3 Set!). Grounded reasoning allocates substantially more attention mass to image tokens during the chain of thought than textual reasoning.

5. Conclusion

Returning to the three questions that motivated the study:

What reasoning strategies make effective SFT baselines for RL? Backtracking/verification is essential for convergence — its absence is not a mild handicap but a failure mode, mirroring the text-domain findings of Gandhi et al. in the visual domain.
Does grounding in the reasoning process improve outcomes? Yes — but only RL reveals it. After SFT alone, grounding looks strictly worse out of distribution; under RL it trains stably for longer, keeps perception anchored to the image, and finishes ahead on every evaluation.
Is model ability the right gating criterion for when to begin RL? No. The grounded-backtracking checkpoint trailed its textual counterpart by 74 points out of distribution after SFT, yet overtook every other variant after RL. Teams selecting checkpoints for RL post-training should audit reasoning behaviors, not just benchmark scores.

Limitations. Set! is a single synthetic task with programmatically verifiable rewards; the attribute vocabulary is small and the grounding format (grid positions) is simpler than natural-image references. Whether these findings transfer to generalist visual problem solving remains open.

Future work. Four directions follow naturally: (i) scaling to generalist visual problem solving over real-world data and many tasks; (ii) mixing multiple reasoning strategies per task to study the exploration–exploitation trade-off in RL-ability; (iii) using grounded reasoning to obtain less noisy process-reward supervision over chains of thought; and (iv) using grounded reasoning to scale vision compute at inference time, i.e. via crop-and-resize operations during the chain.

Points at each card

6. References

OpenAI. Learning to Reason with LLMs. September 2024. openai.com/index/learning-to-reason-with-llms
J. Wei, X. Wang, D. Schuurmans, et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. arXiv:2201.11903
DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948
E. Zelikman, Y. Wu, J. Mu, N. D. Goodman. STaR: Bootstrapping Reasoning With Reasoning. NeurIPS 2022. arXiv:2203.14465
N. Lambert, J. Morrison, V. Pyatkin, et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. 2024. arXiv:2411.15124
K. Gandhi, A. Chakravarthy, A. Singh, et al. Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs. 2025. arXiv:2503.01307
S. Bai, Y. Cai, R. Chen, et al. Qwen3-VL Technical Report. 2025. arXiv:2511.21631
DeepSeek-AI. Thinking with Visual Primitives. 2026.
Z. Shao, P. Wang, Q. Zhu, et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024. arXiv:2402.03300

@article{perceptron2026visualthinking,
  author = {Jeremy Dohmann and PerceptronAI},
  title = {Teaching VLMs to Think Visually},
  journal = {PerceptronAI Research Blog},
  year = {2026},
  note = {https://perceptron.inc/blog/teaching-vlms-to-think-visually},
}