GroundBench.
Multi-resolution polygon grounding exposes the geometry gap in vision–language models.
Bounding-box scores on RefCOCO-family grounding leave little room to separate frontier systems — and boxes discard shape. We re-target the same 1,500 image–expression–referent triples to exact-N polygons at five vertex budgets, then audit filled-region IoU and legal-polygon completion under one fixed denominator.
Boxes localise. Polygons discriminate.
A four-number rectangle carries no shape, follows no boundary, and credits background interior. Dense masks restore shape but need a specialist decoder. Exact-N polygons sit in between — expressive enough to expose contour, compact enough to be emitted as text through the same chat interface used for everything else.
Four coordinates, one box
Limited discrimination · coarse region geometry · fixed output budget · low shape fidelity.
2N coordinates, one contour
Enhanced discrimination · boundary-aware geometry · controlled vertex budget · fine-grained shape fidelity.
Matched control. Because headline box IoU is scored against the ground-truth box and contour IoU against GN, the two are not directly comparable. Rasterising each predicted box into an exact-N polygon against the same contour target gives 57.7 (direct) vs 57.3 (box-derived) when pooled — the gap is an output-contract gap, not a scoring artefact.
The task
Given an image I, a referring expression r, and an announced budget N, the model returns exactly N normalised points. One line, 2N integers in {0…999}. No tools, no mask head, no task-specific training.
A picture, a phrase, a number
One image, one referring expression such as “the zebra behind the fence”, and the announced vertex budget N.
One line of 2N numbers
Exactly N points walked around the object’s edge, each coordinate an integer from 0 to 999. Nothing else — no tools, no mask head, no drawing canvas.
Filled-region utility
Does the emitted sequence still cover a useful region? Finite coordinates are mapped to pixels and their clipped cyclic fill is rasterised; IoU against the canonical target GN.
Legal-polygon completion
Distinct in-range integer vertices forming a simple, positive-area polygon. Duplicates, zero area, and self-intersection score zero — with the same fixed denominator.
Start point and traversal direction are unrestricted — filled-mask IoU is invariant to cyclic shift and reversal. Output allowances are non-binding: the longest legal answer is about 520 characters against a tightest allowance of 1,280 tokens.
Canonical targets
Five deterministic targets per referent, N ∈ {8, 16, 24, 32, 64}, derived from one selected annotation contour and frozen before evaluation. Image, phrase, and referent stay constant — only the answer surface changes. All five files share the same 1,500 UIDs, so every cross-budget claim is paired on exact question intersections.
Split the longest edge
Find the longest straight run along the outline and drop a new point at its midpoint. Repeat. Detail is added where the outline is currently coarsest.
Drop the flattest corner
Find the vertex whose neighbours form the smallest triangle — the corner that bends the least — and remove it. Repeat. The shape loses its least informative point first.
Both rules are deterministic and ties take the lowest index, so the same annotation always yields the same five targets. Watch it run on the left.
Mean IoU between the exact-N target and the unreduced source part. Higher means less approximation loss — which is why the collapse at N = 64 cannot be blamed on a worse reference.
One part, deterministically
When an annotation has several polygon parts, construction takes the part with the most listed vertices. This affects 148 / 1,500 UIDs; within them the selected part covers 71.76% of the all-parts union on average and is the largest-area part 85.14% of the time.
Eight starting offsets
Targets are reconstructed after shifting the sampling origin through eight offsets spaced at 12.5% of the contour. Configuration ordering and the high-budget reversal both survive.
Alternate masks, alternate backend
The same predictions are re-scored against the unreduced part, the all-parts union, and a continuous-area integrator. Five-budget means shift by at most 1.1 points without changing the ordering.
The cohort is not an average
1,500 unique image–referent pairs, one per image, 500 each from RefCOCO, RefCOCO+, and RefCOCOg. Every question ships with frozen metadata — distractor count, contour complexity, relative scale, semantic group, phrase length, and a reproducible composite difficulty label — so results can be sliced instead of summarised.
Pooled fixed-IoU (Table 5). Difficulty, D, Q, and RS use IoU²; semantic groups use the best-effort view. Every configuration declines monotonically from easy to hard and from low to high contour complexity — and improves with target scale.
Results
Four hosted configurations answer all 1,500 questions at all five budgets, with Qwen thinking-off as a same-endpoint ablation. Headline IoU² locks the first parseable output over one primary call plus two stateless retries. Every cell is fixed-denominator — an unparseable response scores zero — paired with its parseable-only counterpart.
▸ Each cell reads fixed / parseable-only: the solid bar is the fixed score, the faint one behind it is parseable-only, and the gap is what parse failures cost. Bold marks best, underline second, within the selected budget. Open the full explorer →
A large, unresolved gap
Gemini leads, then Qwen-max, Doubao, Kimi. Thresholds separate Kimi’s low-budget object finding from Gemini’s stronger conditional geometry — both lose high-N reliability.
Collapse at the densest budget
Scores peak at moderate budgets and fall at N = 64 despite increasing reference fidelity, and the decline persists when extra vertices only subdivide edges without changing the filled target. At N = 64 Gemini’s filled IoU stays 37.7 while legal-polygon coverage and IoU fall to 7.5 / 4.2.
One threshold underdetermines quality
Acc@.5 preserves a single ordering across all four configurations and all five budgets; fixed IoU has one Qwen–Gemini inversion at N = 64. Lower thresholds induce further cross-budget changes.
Get it & cite it
Five JSONL files share the same 1,500 UIDs. The harness, the frozen per-question metadata, the alternate reference masks, and the official scoring code all live in the repository — that is where to look to reproduce or extend anything on this page.
@misc{groundbench2026,
title = {GroundBench: Multi-Resolution Polygon Grounding Exposes
the Geometry Gap in Vision-Language Models},
author = {Bian, Zhonghan and Wang, Zhenran and Li, Jinsong and Qi, Zhangyang},
year = {2026},
url = {https://github.com/co-minder/Groundbench-codebase},
note = {Preprint. Correspondence: zhenran.w.1103@gmail.com}
}