Overview

The benchmark used here is WiserUI-Bench [1], a collection of 300 real UI pairs with verified A/B-test winners. The important part is not that we found a much stronger model. Most of the gain came from fixing how the task is measured. Multimodal models often prefer whichever design appears in a particular screen position. If that bias is not controlled, a model can look accurate without actually judging the designs well.

Our main conclusion is simple: counterbalance the comparison, measure consistency, and use the prediction for choosing what to test, not for steering a test that is already running.

Headline result

• Chance: 25%
• Best published: 39.00% [1]
• This work: 57.28%
• Perfect: 100%

Chance is 25%, not 50%, because a pair only counts as correct when the model chooses the same winning design in both display orders. The 57.28% result has not been significance-tested on the cleaned 103-pair corpus, so it should be read as a point estimate.

1. Why this problem matters

A/B tests cost traffic and time. While a test is running, some visitors see the worse version, and most teams have more ideas than they can realistically test. A useful prediction system would not replace A/B testing. It would help decide which ideas deserve traffic first.

The task is difficult because a screenshot does not contain all the information that affects conversion. Audience, traffic source, season, device mix, page speed, and the rest of the funnel can all matter. Small but real lifts can also be visually invisible.

The strongest published WiserUI-Bench result we use for comparison is 39.00% [1]. Our result is higher, but the main reason is the scoring method rather than a claim that our model understands design dramatically better.

2. The measurement problem

If you show a model two screenshots, ask which is better, then swap their positions and ask again, it will often reverse its answer. Across our tests, models chose the design shown second more often than the design shown first, with later-slot preference ranging from 51.8% to 83.3%.

This creates a serious evaluation problem. On a balanced dataset, a model that always chooses the second option can score close to 50% even if it understands nothing about the designs.

Counterbalancing

We therefore run every pair twice:

• Order 1: Design X, then Design Y
• Order 2: Design Y, then Design X

A pair is counted as correct only if both runs choose the same real-world winner. We call this Consistent Accuracy.

For example, if Y is the verified winner and the model chooses Y in both orders, the pair is correct. If it simply chooses whichever design appears second, the pair is wrong.

This does not remove position bias. It prevents that bias from earning credit. Under this rule, a pure slot-following model scores 0% instead of looking like a 50% system.

The bias remains large

• GPT-5.6-Sol: 47.90% when the winner is in slot A, 65.55% in slot B
• Opus 5: 51.26% in slot A, 74.79% in slot B
• Sonnet 5: 49.58% in slot A, 73.11% in slot B
• Three-model ensemble: 47.90% in slot A, 74.79% in slot B

The ensemble made the asymmetry worse instead of cancelling it. We do not know why the bias occurs, but it appears across models, providers, datasets, and vote counts.

3. The system that worked

The final setup is intentionally simple. A single model compares the two designs, both slot orders are tested, and the model is sampled several times. The final answer is decided by majority vote, with early stopping once the result can no longer change.

On the cleaned 103-pair benchmark slice, Sonnet 5 with nine votes reaches 57.28% Consistent Accuracy. On an older 119-pair slate, Sonnet 5 at nine votes reached 56.30%, while Opus 5 at three votes reached 59.66%. Those two older results were statistically indistinguishable from each other.

Cost on the 119-pair slate

• Sonnet 5, N=9: 56.30% accuracy; $0.065 per pair; $0.092 per correct pair
• Opus 5, N=3: 59.66% accuracy; $0.103 per pair; $0.172 per correct pair

In that comparison, nine cheaper votes produced roughly the same accuracy as three more expensive ones for about half the cost per correct pair.

4. What did not help

We tested many obvious improvements. Most did not work because they failed to remove the shared position bias or because they added complexity without adding useful information.

Persona panels

Five prompted “personas” on one model reduced accuracy from 42.86% to 29.41%. The personas agreed with one another 84.29% of the time, so they behaved more like copies of the same judge than independent viewpoints. Their majority vote amplified shared mistakes.

Multiple models

Combining three models from two providers reached 47.06%, compared with 51.26% for the best single model. The models shared the same preference for the later slot, so majority voting reinforced the bias.

Confidence filtering

Unanimous or high-margin votes were not reliably more accurate. Agreement often meant the models shared the same bias, not that they had found stronger evidence.

More reasoning

More reasoning did not improve performance. On one sweep, Medium reasoning reached 51.26%, High fell to 44.54%, and XHigh reached 47.90%. This pattern is consistent with research showing that long chains of reasoning can sometimes pull multimodal models away from visual evidence and toward textual priors [3], [4], [5], [6], [7].

More detailed prompts

A six-field evidence-and-risk format performed about 13 percentage points worse than a simpler two-field format. Extra structure appeared to strengthen the position preference rather than improve judgment.

Scoring designs separately

Scoring each design on its own also performed worse. Accuracy fell from 52.43% to 47.57%, while ties increased. The useful signal appears to be in the comparison between two designs, not in an absolute score assigned to either one alone.

Image tools

Zoom and crop tools underperformed the no-tools baseline. Many meaningful differences in these tests are global, such as layout, hierarchy, offer, and emphasis. Magnifying small regions can remove that broader context.

5. More votes help, but there is no magic number

Early in the project, separate experiments seemed to show an ideal number of votes. That turned out to be noise from running each vote count as a different batch.

When we instead took one batch of calls and re-used the same calls to compare different vote counts, the curve became smooth:

• N = 1: 46.22%
• N = 3: 52.10%
• N = 5: 52.94%
• N = 7: 53.78%

More votes helped, but with diminishing returns. There was no reliable peak. The practical lesson is to choose the vote count based on cost rather than treating one noisy maximum as an optimum.

Early stopping reduced the number of calls by roughly 38% with no accuracy loss in the tested sample.

6. Why the trained model failed

We also tried a different approach: score each design using extracted features, then train a smaller model to predict outcomes cheaply. Eight model configurations failed to produce a reliable result. Adding four times more training data did not materially improve them.

The deeper problem appeared in the labels themselves. Of 673 distinct designs, 181 appeared in more than one pair. Among those repeated designs, 73, or 40.3%, had both winning and losing labels.

That matters because it shows that “how good is this design?” is often the wrong question. The same design can win against one alternative and lose against another. Outcome depends on the design and its context, including what it is compared with, the audience, the metric, and the time of the test.

This explains why scoring designs independently failed while pairwise comparison worked better. The pairwise system asks a question the data can actually answer: which of these two designs is more likely to win in this comparison?

7. What happened on unseen production data

The benchmark had been used throughout development, so we also tested the method once on a held-out corpus of 278 production pairs from 38 companies and 82 metrics.

Held-out results

• Sonnet 5, N=9: 36.3% Consistent Accuracy
• Opus 5, N=3: 40.3% Consistent Accuracy
• Chance: 25%

Both remained above chance, but performance fell substantially from the benchmark result. The two models were not significantly different from each other on this held-out set.

Several factors likely contributed. The benchmark contains curated case studies where visual differences are often easy to explain. The held-out corpus contains raw production outcomes, including many small lifts that may not be visible in a screenshot. In addition, roughly half of the before-state images were reconstructed rather than independently verified captures.

The safest claim is therefore narrow: the method retained an edge over chance on unseen data, but the margin was much smaller than on the benchmark.

8. Where a weak prediction is useful

We tested two possible uses through simulation. These results are modelling, not production measurements.

Using the prediction inside a running experiment

Seeding a bandit with the model prediction did not help. At roughly 57% accuracy, the model is still wrong often enough that a strong incorrect prior can take a long time for live traffic to overcome.

• Cold start: 100 simulated days
• Weak prior: about 96 days
• Capped prior: about 137 days
• Strong prior: about 171 days

The asymmetry is the problem. A correct prior saves a little time, but a wrong prior can cost much more.

Using the prediction before testing begins

The better use is candidate selection and filtering. Before a test starts, the model competes with a backlog decision rather than with live experimental evidence.

In simulation, selecting five candidates out of twenty using the model beat random selection at both benchmark and held-out accuracy levels. Filtering the weakest 20% also reduced the chance of discarding the true best candidate compared with a random filter.

This is where a modest signal can still be valuable: not by replacing A/B testing, but by helping decide what deserves to be tested at all.

9. What generalizes beyond A/B testing

Three lessons extend beyond this specific problem.

1. Counterbalance pairwise judgments. If a model compares two things, test both display orders. Otherwise position preference can look like intelligence.

2. Do not trust peaks from separate noisy runs. If possible, compare settings using the same underlying calls or samples.

3. Check whether the target is actually predictable from the input. If the same input repeatedly receives contradictory labels, better features or more data may not solve the problem.

Practical recommendations

1. Counterbalance from the beginning. Retrofitting it later can invalidate earlier results.

2. State the chance baseline next to the score. Consistent Accuracy has a 25% floor, not 50%.

3. Use held-out data once. Repeated tuning against a test set turns it into another development set.

4. Group repeated designs during validation. Row-wise splitting previously produced an inflated 61.3% result that fell to 51.3% after leakage was removed.

5. Expect large data losses during cleaning. In one corpus, 2,070 candidate experiments became 367 usable pairs.

6. Make failed ideas cheap to test. Most of the useful lessons in this project came from directions that did not work.

Bottom line

The strongest cleaned-benchmark result is 57.28% Consistent Accuracy, compared with a 25% chance floor and a 39.00% published benchmark figure [1]. On unseen production data, performance fell to 36.3–40.3%, still above the same 25% floor.

The durable result is not the headline percentage. It is the measurement lesson: models can have strong position preferences, and ordinary single-order scoring can hide them. Counterbalancing makes those biases visible and prevents them from earning credit.

At the accuracy levels we measured, the system is best treated as a screening tool for deciding what to test, not as a replacement for live experimentation and not as a strong prior inside a running test.

1. [1] C. Jeon, J. Kim, S. Yoon, H. Shim, J. Choi, J. Kim, Y. Kim, and H. Yu, “Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding,” in Proc. 64th Annu. Meeting Assoc. Comput. Linguistics (ACL 2026), 2026. [Online]. Available: https://arxiv.org/abs/2505.05026. [Accessed: Aug. 10, 2026].
2. [2] C. Jeon et al., “WiserUI-Bench,” Github Datasets, 2026. [Online]. Available: https://github.com/jeochris/wiserui-bench. [Accessed: Aug. 10, 2026].
3. [3] A. Ghosal et al., “VisRef: Visual Refocusing while Thinking Improves Test-Time Scaling in Multi-Modal Large Reasoning Models,” arXiv:2603.00207, Feb. 2026. [Online]. Available: https://arxiv.org/abs/2603.00207. [Accessed: Aug. 10, 2026].
4. [4] S. Kancheti, A. Kanade, V. N. Balasubramanian, and T. Ganu, “Chain-of-Thought Degrades Visual Spatial Reasoning Capabilities of Multimodal LLMs,” in Proc. 64th Annu. Meeting Assoc. Comput. Linguistics (ACL 2026), Short Papers, 2026. [Online]. Available: https://arxiv.org/abs/2604.16060. Preprint: arXiv:2604.16060.
5. [5] Jin et al., “Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do,” in Proc. 64th Annu. Meeting Assoc. Comput. Linguistics (ACL 2026), 2026. [Online]. Available: https://arxiv.org/html/2606.22565v1. [Accessed: Aug. 10, 2026].
6. [6] “Vision Language Models are Biased,” arXiv:2505.23941, 2025. [Online]. Available: https://arxiv.org/abs/2505.23941. [Accessed: Aug. 10, 2026].
7. [7] “Towards a Mechanistic Understanding of Large Reasoning Models,” arXiv:2601.19928, 2026. [Online]. Available: https://arxiv.org/abs/2601.19928. [Accessed: Aug. 10, 2026].

• Published WiserUI-Bench comparison figures are attributed to [1]. The dataset is listed separately in [2].
• All figures attributed to this work are internal and not peer-reviewed.
• The 57.28% cleaned-benchmark result has not been significance-tested on that corpus.
• Some closed-direction figures in the longer analysis came from internal notes rather than committed run artifacts.
• The simulation results are illustrative modelling, not observed production outcomes.