Predicting the winner before the traffic

We built a system that reads two design variants and predicts which one an A/B test will pick, before any traffic is spent on it. On a public benchmark of real, already-run tests it reaches 57.28% Consistent Accuracy, roughly 1.5× the best published figure. The gain came from correcting how the task is scored, not from a stronger model. Below: that result, the measurement problem behind it, and the ten approaches that closed.

The headline result

Consistent Accuracy on the cleaned 103-pair benchmark slice: 57.28%, meaning correct in both slot orderings — 59 of 103.

For reference:

• Chance: 25%
• Best published: 39.00%
• This work: 57.28%
• Perfect: 100%

Chance on this metric is 25%, not 50%. A prediction counts only if it holds when the two designs swap places, and independent guesses agree twice a quarter of the time. Rescaled so chance is zero and perfect is one hundred, the result is 43.04. No significance test has been run on this corpus, so read it as a point estimate; the p = 0.000145 result for nine votes against one was measured on the superseded 119-pair slate.

1. The problem: knowing before you spend

An A/B test is a purchase. What you pay is not engineering time but traffic: for the duration of the test, some share of visitors sees the variant that turns out to be worse, and the calendar only holds so many tests. Most teams running conversion work have more candidate designs than they have traffic to evaluate them with. The binding constraint is allocation: which ideas get tested at all, and in what order.

That makes a specific question valuable. Given two designs and nothing else, can a model say which one a real test would pick? A model that beats chance doesn't remove the need to test. It changes what you test first, and what you never bother testing at all.

The task is harder than it looks, and for reasons that have little to do with how good the model is. The label isn't a design judgment. It's a behavioural outcome, measured on strangers, at scale. Designs that violate every convention win regularly. Worse, the information that determined the outcome is often not in the image: who the traffic was, where it came from, what the rest of the funnel looked like, what season it was. And effect sizes are small. A statistically solid 1.5% lift is a real result and an invisible design difference. Nothing in a static screenshot distinguishes it from noise.

The published state of the art says roughly this. WiserUI-Bench (Jeon et al., ACL 2026) assembles 300 real-world UI pairs, each A/B-tested at scale by an actual company, each with an empirically validated winner. Evaluated across ten multimodal models, the paper concludes that they "struggle to consistently predict the more effective UI/UX design in real-world UI pairs." The best number in it is 39.00%, from a multi-agent debate configuration over GPT-4o. Two of the ten score below chance.

We reached 57.28% on that benchmark. Almost all of that distance came from one thing, and it isn't the model.

2. The measurement problem

Ask a multimodal model to compare two screenshots and it will answer. Swap the screenshots and ask again and it will often say the opposite. Not because the second look was better informed. Because it prefers a slot.

The cleanest way to see it is to stop asking which answer is right and just count which slot gets chosen. Across every model, dataset and vote count we measured, the model picks whichever design is displayed second more often than the one displayed first, at rates between 51.8% and 83.3%. An unbiased judge would sit at 50%. Nothing moved it: not a different provider, not more votes, not a different prompt schema.

That preference attaches to the slot, not to the designs in it, which is what makes it invisible to any evaluation that shows each pair once.

Now consider what that does to a benchmark score. If you evaluate each pair with a single call and the corpus is balanced, with the winner in slot one half the time, then a model with a total slot preference and no visual understanding whatsoever scores 50%. Single-call accuracy on a balanced corpus cannot distinguish a system that reads designs from a system that always answers "the second one." Both land near 50, and 50 is the number that looks like a coin flip and therefore looks honest.

The fix is old and cheap. Counterbalance. Run every pair twice, once in each slot order, and count the pair as correct only when both runs name the same design and that design is the verified winner. We call the resulting metric Consistent Accuracy.

How counterbalancing works — worked example

The same pair, both slot orders. Correct only if both agree.

Counted correct:

• Order 1: Design X, then Design Y (picked)
• Order 2: Design Y (picked), then Design X

Both orders name Design Y, and Y is the verified winner. The choice followed the design, not the slot.

Counted incorrect:

• Order 1: Design X, then Design Y (picked)
• Order 2: Design Y, then Design X (picked)

Both orders name whatever sits in the second slot. Single-call scoring records this pair as one hit and one miss, a tidy 50%, and reports no evidence of a problem.

Chance under this rule is 25%: two independent guesses land on the same design half the time, and that design is the winner half of those times. Consistent Accuracy must never be read against 50%.

How to read Consistent Accuracy, in full

One pair, worked through. A retailer tested two versions of a product page; version Y won the real A/B test. We ask the model twice.

• First call — slots are X, Y, and the model answers Y
• Second call — slots are Y, X, and the model answers Y
• Scored — correct, because both calls named the winner

Now suppose the second call had answered X. The pair scores incorrect, not half-correct, even though one of the two calls named the winner, and even though naive accuracy would book it as 50%. There is no partial credit and no averaging: a pair is correct or it is not.

So the floor is 25%, not 50%. Two independent guesses agree with each other half the time, and when they agree they are right half of those times. If you find yourself comparing a Consistent Accuracy figure against 50% and concluding it is barely above chance, you have compared it to the wrong number. That misreading happened repeatedly inside this project before the metric was named.

This is the whole trick, and it is worth being precise about what it buys. Counterbalancing does not make the bias go away. It makes the bias unable to earn credit. A slot-following model scores 0% under Consistent Accuracy instead of 50% under single-call accuracy, and the gap between our figure and the published ones is substantially a gap in what is being counted rather than in what the model can see. That framing matters more than the headline: we are not claiming a model that understands design better than the ones in the literature. We are claiming a score that a position-preferring model cannot fake.

It also does not make the bias small. Under full counterbalancing, accuracy is still dramatically higher when the true winner happens to occupy the second slot.

Residual asymmetry — per-call accuracy by which slot held the winner

• GPT-5.6-Sol: winner in slot A, 47.90%; winner in slot B, 65.55%; gap +17.65 percentage points
• Opus 5: winner in slot A, 51.26%; winner in slot B, 74.79%; gap +23.53 percentage points
• Sonnet 5: winner in slot A, 49.58%; winner in slot B, 73.11%; gap +23.53 percentage points
• Three-model ensemble: winner in slot A, 47.90%; winner in slot B, 74.79%; gap +26.89 percentage points

The ensemble amplified the asymmetry rather than cancelling it, because the three models share the preference and majority voting reinforces it. The effect survives filtering to unanimous panels (Opus 56.19% / 79.25%; Sonnet 48.28% / 79.12%; on a separate vendor-published corpus 77.61% / 95.89%), and per-call preference for the later slot runs between 51.8% and 83.3% across every model, dataset and vote count we measured. We don't know why it happens.

That last point is an open question, not a solved one, and we want to be plain about it. A bias that persists inside unanimous panels, across two vendors, across three datasets and across every vote count isn't a sampling artifact, and it isn't obviously a prompt artifact either. We report it because it bounds how much of the remaining headroom is available to inference-time tricks: a large, stable, direction-inconsistent position effect is the dominant error term in this task, and none of the ensembling or self-consistency work in Section 4 moved it.

One more note on the metric, because it matters later. Consistent Accuracy generalizes to more than two slot rotations, and the chance floor falls fast: requiring agreement across all N rotations of N designs puts chance at (1/N)^N, which is 3.7% at N = 3. Any number quoted on a multi-arm corpus needs its floor attached, and any comparison between a two-arm and a multi-arm figure without normalization is meaningless.

3. The result

The configuration behind the headline figure is unglamorous. It is a single model, prompted for a structured two-field comparison, called nine times per pair with the slots counterbalanced, a majority vote over the results, and early stopping once the vote can no longer change. On the cleaned 103-pair slice of the benchmark, Sonnet 5 at N = 9 gets 59 pairs consistently correct: 57.28%, or 43.04 on the normalized scale where chance is zero. No significance test has been run against that corpus, so it is a point estimate and nothing more.

The significance testing was done on an earlier 119-pair slate, which the cleaned corpus supersedes. On that slate Sonnet 5 at N = 9 reaches 56.30% (67 of 119), p = 0.000145 against the same model at N = 1, and Opus 5 at N = 3 reaches 59.66% (71 of 119), p = 0.0129. The two configurations were only ever run head to head there; there is no Opus figure on the cleaned corpus.

Against the published field (WiserUI-Bench). Chance is 25% for every row.

• This work: Sonnet 5, N=9 (103-pair cleaned slice): CA 57.28%, above chance 43.04
• This work: Opus 5, N=3 (superseded 119-pair slate): CA 59.66%, above chance 46.21
• GPT-4o + multi-agent debate (best published): CA 39.00%, above chance 18.67
• InternVL-2.5-38B (best zero-shot): CA 34.56%, above chance 12.75
• GPT-5.1: CA 33.33%, above chance 11.11
• Claude 4.5 Sonnet: CA 32.33%, above chance 9.77
• GPT-4o, zero-shot: CA 30.11%, above chance 6.81
• Claude 3.5 Sonnet: CA 24.22%, above chance −1.04
• o1: CA 15.56%, above chance −12.59
• LLaVA-NeXT-7B: CA 10.78%, above chance −18.96

Published rows are given to us as the figures reported in Jeon et al., ACL 2026 (arXiv:2505.05026); we have not checked them against the paper ourselves, and the 25% chance floor applied to them is operator-provided rather than verified. Our rows are internal, not peer-reviewed, and measured on a 103-pair cleaned slice and a superseded 119-pair slate rather than the full 300. Treat the whole comparison as indicative rather than a controlled head-to-head: on raw Consistent Accuracy the headline configuration is roughly 1.5× the best published figure, and roughly 2.3× on the normalized scale.

Two configurations at 56.30% and 59.66% on the 119-pair slate invites a ranking, and the statistics don't support one. McNemar's test on the paired outcomes gives p = 0.5847 at n = 119, and p = 0.1925 on a larger independent 278-pair sample. They're indistinguishable. They are not, however, the same price.

Cost — statistically indistinguishable, roughly 2× apart (119-pair slate)

• Sonnet 5 · N=9: Consistent Accuracy 56.30%; cost per pair $0.065; cost per correct pair $0.092
• Opus 5 · N=3: Consistent Accuracy 59.66%; cost per pair $0.103; cost per correct pair $0.172

Throughput, all runs at 150 concurrent workers

• 119 pairs: 195 seconds, 36.6 pairs/minute
• 103 pairs: 197 seconds, 31.3 pairs/minute
• 278 pairs: 346 seconds, 48.2 pairs/minute

Typical is 1.2–1.9 seconds per pair. One 107-pair run took 866 seconds, or 7.4 pairs/minute, a 4–5× outlier with zero errors and zero retries, still unexplained. Plan for three to six minutes per corpus and tolerate the occasional fifteen.

Accuracy and cost here are both from the 119-pair slate, the only corpus on which the two configurations were run head to head. Nine cheap votes buy the same accuracy as three expensive ones for roughly half the money.

The method transfers to other curated corpora and does better on them. On 61 genuinely two-arm cases drawn from a testing vendor's published case studies, the Opus configuration reaches 80.33% (73.77 normalized), the highest figure in the project and a reminder of how much corpus curation determines the number. On an internal two-arm corpus scored with an older single-call configuration it reaches 32.31% (9.74 normalized). Same method, same metric, a 64-point spread in normalized terms, driven by what the corpus contains rather than by what the model can do. Section 7 returns to this.

4. What didn't work

Twenty iterations produced one configuration worth shipping and a much longer list of closed directions. We think the list is the more useful half of the work. Most of these are the obvious things to try: ensembling, personas, richer prompts, confidence filtering, tools, more reasoning. Most of them work in other tasks. Knowing which ones fail here, and by what mechanism, is what saves someone else the same twenty iterations.

Each item below carries the numbers and, more importantly, the mechanism. Saying "persona panels didn't help" is bookkeeping. "Five personas on one model reached 84% pairwise agreement, so voting amplified a shared bias instead of cancelling it" is a finding you can take somewhere else.

Didn't work — closed, with the mechanism

Persona panels. Five prompted lenses on a single model, majority vote across them. Result: 29.41% vs. a 42.86% baseline, McNemar p = 0.009. Mechanism: the five lenses reached 84.29% pairwise vote agreement. They were near-clones, not independent viewpoints. Voting therefore amplified a shared bias rather than cancelling it: preference for the later slot rose 74.5% → 77.3%. On 4–1 splits the majority was right only 36.6% of the time, meaning the lone dissenter was more often correct than the panel.

Cross-provider ensembling. Three models from two providers, majority vote per ordering. Result: 47.06% vs. 51.26% for the best model alone, p = 0.3833, 8 errors fixed, 13 broken. Mechanism: all three models carried the displayed-B asymmetry in the same direction, so the ensemble amplified it to +26.89 percentage points, the largest gap we measured. Complementary signal does exist: oracle ceilings are 80/119 by model and 84/119 by ordering, well above any single configuration. Majority voting cannot reach it, because the models agree on their errors more readily than on their insights.

Confidence calibration. Filtering to high-confidence pairs by vote margin, unanimity, and early-stop agreement. Result: +0.76 / +4.64 / +0.02 / −0.94 percentage points across four corpora, all Holm-adjusted p = 1.000. Mechanism: vote agreement measures consensus, and part of that consensus is the shared position bias, and the asymmetry persisted even among fully unanimous panels. Unanimity therefore predicts that the models agree, not that they are right. On one corpus the "high-confidence" subset was less accurate than the unfiltered set. (Run directory not committed; figures from analysis notes.)

Higher reasoning effort. Sweeping the reasoning-effort setting on the strongest model. Result: Medium 51.26%, High 44.54%, XHigh 47.90%. Mechanism: non-monotonic, with medium at the peak. More deliberation made the prediction worse. This has its own section below (4a), because the mechanism is documented in the literature and it reframes what kind of task this is.

Temperature as a sampling lever. Varying temperature to control diversity across self-consistency votes. Result: HTTP 400, parameter deprecated on current models. Mechanism: not a finding about the task but about the tooling — the lever no longer exists, so vote diversity is whatever provider-side sampling gives you. Consequences below (4b).

Six-field evidence-and-risk schema. A structured output asking for evidence, counter-evidence, and risk per design. Result: roughly 13 percentage points worse than a two-field schema; P(pick the first slot) moved from 0.377 to 0.258–0.290. Mechanism: the extra structure moved the model's prior instead of sharpening its judgement. Asking for risk framing made it more reluctant to endorse the first option it saw, which shows up directly as a stronger pull toward the later slot. The schema made the measurement problem worse, not the reasoning better.

Independent scoring with self-consistency. Score each design alone on a scale, three times, then compare the means. Result: ties rose 13.59% → 16.50%; accuracy fell 52.43% → 47.57%. Mechanism: rating a design in isolation discards the comparison, which is where the signal lives. Absolute quality scores anchor to the middle of the scale and collapse toward each other, so the differences that decide a pair fall inside the rounding. Resampling does not fix it, because the anchoring is intrinsic to scoring one thing at a time rather than noise around a true score.

Directional change extraction. Ask a model to name what changed between the two designs, then reason over the change. Result: 56.7% agreement on the primary label across orderings, 30.0% on the full operation, 2 literal direction inversions, 4 of 7 canonical changes found, $0.476 for 30 pairs. Mechanism: the extractor inherits the same order sensitivity it was introduced to remove. If "what changed" flips when you swap the slots, and twice it inverted outright and reported the reverse operation, then no downstream reasoning over that description can be order-invariant.

Multi-arm decomposition. Split multi-arm experiments into pairwise comparisons to enlarge the corpus. Result: +68 honest effective observations against 114 naive, an 11% gain. Mechanism: 35 original designs span 47 experiments, so decomposition mostly manufactures pseudo-replication — the same design reappears across derived pairs and the effective sample size grows far slower than the row count. An 11% gain in honest observations does not buy a different answer.

Copy text and embeddings as features. Extract on-page copy, and image embeddings, as inputs to a trained model. Result: 934 rows with no before-state markup; 269 of 1,309 rows with genuine copy; 104 embedding rows over 47 images. Mechanism: blocked by data rather than by method. The before-and-after markup fields were empty everywhere we checked, so there was no baseline to diff against, and once framework class names were filtered out as false positives, fewer than a quarter of rows carried real copy. The embedding coverage was too thin to train on, and the comparison-ready view of it shipped zero rows.

Mixed — helped weaker models, not the stronger ones

Four-field observe-then-compare prompt. Describe each design first, then compare, in one structured call. Result: Haiku +17.46 percentage points at 2.06× cost; Sonnet +1 pair (noise); Fable abandoned at roughly 50× cost. Mechanism: scaffolding substitutes for capability the strong models already have. It bought a large gain on the weakest model and nothing on the ones actually worth deploying, which is the signature of a prompt that supplies structure rather than information.

Self-consistency as a step change. Repeat the same call several times and take the majority. Result: Sonnet 40.48% → 42.06% across N = 1–7; Haiku flat at 21.43%. Mechanism: repeated sampling averages out independent noise, and the dominant error here is not independent, because every call shares the same position preference. Votes do help, monotonically (see below), but the gains are small relative to the linear cost in N, so this is a knob rather than a lever.

Image tools: zoom and crop. Give the model tools to magnify and crop regions of each screenshot. Result: underperformed the no-tools baseline at every reasoning level tested. Mechanism: the decisive differences are global ones like layout, emphasis, offer, and hierarchy, and zooming trades that view for local detail that does not carry the outcome. Read together with the reasoning-effort result below, this says the problem is not insufficient visual access.

Worked — kept, with evidence

Counterbalancing and Consistent Accuracy. Evaluate every pair in both slot orders; count it correct only when both agree. Result: chance falls from 50% to 25%; a slot-following model scores 0% instead of 50%. Mechanism: it does not remove the position bias; it removes the bias's ability to earn credit. Everything else in this article rests on it, including the ability to tell that the persona and ensemble experiments made things worse rather than better.

Counterbalanced votes, monotonic in N. Majority vote over N counterbalanced calls, with N chosen on cost. Result, same-draw control: 46.22% / 52.10% / 52.94% / 53.78% at N = 1/3/5/7; Sonnet N=9 vs. N=1, p = 0.000145. Mechanism: more votes help monotonically with diminishing returns, and the improvement is real once measured against the same physical draw rather than against a fresh run. What does not exist is an optimum. See Section 5.

Early stopping. Stop issuing votes once the outcome can no longer change. Result: roughly 38% fewer calls, no accuracy cost, with pooled majority and strict Consistent Accuracy agreeing on 67 of 67 pairs. Mechanism: once a majority is arithmetically decided, further votes cannot alter the verdict, so the saving is free by construction rather than a bias-variance trade. The only reason to check empirically is to confirm the pooling rule and the strict metric agree, which they did on every pair.

One more direction closed before it started. Categorical change-type features, which encode what kind of change each pair represents, were blocked because the available metadata is non-directional: it records that a headline changed, not which way, and a win-rate feature built on that would be tautological given win-and-lose labels. Two of the items above (cross-provider ensembling and confidence calibration) rest on analysis notes rather than a committed run directory, and we would not defend those figures to the decimal.

The persona result is worth one more sentence, because it is a negative result about a popular idea and it does not appear to be ours alone. Simile, which raised $200M at a $2B valuation in July 2026, grounds its agents in real interview and transaction data specifically because generic LLM personas lack fidelity, which is the same conclusion reached from the opposite direction. Their headline 85% accuracy figure is self-reported, measured on the General Social Survey, and not comparable to this task; the design choice is the corroborating part, not the number. Prompting one model to role-play five people produces five correlated copies of that model, and averaging correlated copies amplifies whatever bias they share.

4a. Why more reasoning made it worse

The reasoning-effort sweep gave us the most counterintuitive result in the project. On the strongest model, the middle setting won.

Reasoning effort — Consistent Accuracy by setting, one model, one sweep

• Medium: 51.26% — the peak
• High: 44.54% — −6.72 percentage points from the peak
• XHigh: 47.90% — −3.36 percentage points from the peak

Non-monotonic, with the middle setting at the peak. This was a single sweep on one model, not a controlled study. It is consistent with the published literature below rather than independent confirmation of it.

The shape isn't novel. It replicates a well-documented phenomenon, and the literature hands us the mechanism too.

A 2026 survey of reasoning models states the curve plainly: accuracy "initially rises with reasoning length, but then peaks and declines as chains become excessively long", and notes that "incorrect answers often correspond to longer reasoning chains than correct ones." The same inverse-U shows up in vision specifically: Vision Language Models are Biased reports that "counting accuracy initially rises with thinking tokens" before falling away.

Why it happens is the useful part for us. VisRef describes it as an attention problem: in vision-dependent tasks, "extended textual reasoning at inference time can degrade performance as models progressively lose attention to visual tokens" and increasingly rely on textual priors alone. Two ACL 2026 papers report the consequence at task level. Chain-of-Thought Degrades Visual Spatial Reasoning Capabilities of Multimodal LLMs finds that "CoT prompting consistently degrades performance in visual spatial reasoning" across sixteen models and thirteen benchmarks. (The arXiv preprint of that work says seventeen models; the peer-reviewed version says sixteen.) Look Light, Think Heavy separates the two regimes directly: "for perception tasks, CoT can lead to undesirable side effects, such as reduced performance in visual grounding," while remaining effective for mathematical and scientific reasoning.

Which suggests we've been miscategorising the task. Judging which of two designs converts better looks like a reasoning task and behaves like a perception task. The information that decides it is in the image: the relative weight of two calls to action, how far down the fold the price sits, whether the form looks short. Extended deliberation pulls the model away from those pixels and toward textual priors about what good design is supposed to be. Section 6 is a long argument that those priors do not predict outcomes; this section is the inference-time version of the same problem.

That also explains the image-tools result. Giving the model zoom and crop did not help either, so the failure is not insufficient visual access. Both results point at the same thing: attention drifting off the visual evidence during deliberation, which more looking does not fix and more thinking makes worse.

4b. Temperature is no longer a lever

A practical note, documented because it will catch someone else. We tried to use temperature as a sampling-diversity control for the self-consistency runs: raise it to make the N votes less correlated, and the majority should carry more information. On current Anthropic models the call returns a 400 error, because the parameter is deprecated on the newer releases.

The consequence isn't cosmetic. Any repeated-sampling strategy now relies on whatever sampling variance the provider applies by default, which is neither documented nor controllable. You can't tune diversity. You can only observe whatever you get. For a method whose entire premise is that N independent-ish draws beat one draw, that is an uncontrolled term sitting underneath the result, and it is part of why the vote-count findings in Section 5 needed a same-draw control before we believed any of them.

5. The peaks that weren't there

For most of the project we believed each model had an optimal number of votes. The evidence looked clean: sweep N upward, accuracy climbs, peaks, then declines. We had eight such peaks across four models, and a tidy story about them: beyond some point, extra votes dilute the signal with marginal samples.

The story was an artifact of how we ran the sweeps. Each point in each sweep was a separately executed batch, so each was a single draw from a noisy distribution, and the maximum of eight noisy draws looks like a peak whether or not one exists. The control is straightforward. Instead of running N = 1, 3, 5, 7 as four batches, run one batch of nine votes and re-slice the same physical calls into nested prefixes. Same calls, same money, no between-run variance.

Same-draw control — re-slicing identical calls: smooth, monotonic, no peak

• N = 1: 46.22%
• N = 3: 52.10%
• N = 5: 52.94%
• N = 7: 53.78%

Nested-prefix analysis over one batch of calls, scaled from the 25% chance floor. Every post-peak decline in the original separately-executed sweeps was non-significant: McNemar p between 0.34 and 1.0 across six comparisons.

More votes help monotonically, with diminishing returns and no peak. The correct claim about any single-run maximum is that it is a single-draw maximum, and the correct way to choose N is on cost, not on a peak. This retires eight findings, one narrative, and the sentence "N = 9 is optimal for Sonnet," which is why Section 3 presents N = 9 as a cost choice rather than an optimum.

Stated generally, because it is not specific to vote counts: when you run a sweep as a series of separate executions, you are varying two things at once: the parameter and the sampling draw. Any peak you find may be a single-draw maximum, and the more points in the sweep, the more likely one of them looks like a peak. The control is cheap and should be standard practice: run the largest configuration once, re-slice that same set of calls into nested prefixes, and read the curve off a single draw. It costs one run instead of four, and it is the difference between a finding and an artifact. We ran eight sweeps before running the control, and the control retired all eight.

6. The trained model, and the question the data can't answer

Running a large model nine times per pair is an odd way to make a prediction. The obvious alternative is to train something small: extract features from each design once, learn the mapping from features to outcome, then score new designs cheaply. We built that branch and it failed completely. Why it failed turned out to be the most useful thing we learned.

The architecture rated each design on six dimensions using a language model, added a deterministic aesthetic score, and predicted an antisymmetric log-ratio target, the log of a design's own conversion rate over its partner's, so that swapping the pair flips the sign. Cross-validation was grouped by client and by image hash, so no design and no client could appear on both sides of a fold. Four target framings (absolute and pairwise, regression and classification) across two model families gave eight configurations.

We also quadrupled the training data mid-branch: 166 rows to 664, with the effective sample size under weighting rising from about 150 to 537 against a power threshold of roughly 1,000. Nothing changed at all.

Trained models — eight configurations, all null. Chance on this ranking task is 50%.

• Absolute, regression, ridge: 48.7% ranking, −1.3 above chance, p = 0.85
• Absolute, regression, lightgbm: 53.8% ranking, +3.8 above chance, p = 0.46
• Pairwise, regression, ridge: 51.3% ranking, +1.3 above chance, p = 0.855
• Pairwise, regression, lightgbm: 49.6% ranking, −0.4 above chance, p = 1.0
• Absolute, classification, logreg: 46.2% ranking, −3.8 above chance, p = 0.46
• Absolute, classification, lightgbm: 45.4% ranking, −4.6 above chance, p = 0.36
• Pairwise, classification, logreg: 40.3% ranking, −9.7 above chance, p = 0.043 (below chance)
• Pairwise, classification, lightgbm: 56.3% ranking, +6.3 above chance, p = 0.199

Every regression configuration produced negative R² on every cross-validation fold, which is worse than predicting the mean, on every split. The pairwise-regression ridge figure is bit-for-bit identical to the pre-expansion baseline (61 of 119) despite four times the training data.

Two figures we retracted. Earlier in this branch we recorded 61.3% at p = 0.0168 and, later, 58.8% at p = 0.066. Neither survives. The first came from row-wise cross-validation that allowed pseudo-replicates of the same design to fall on both sides of a fold; correcting the split brings it to 51.3% at p = 0.8546. The second collapsed to the same 51.3% once confidence weighting was applied to an otherwise identical corpus. Both are withdrawn, and we mention them because the mechanism, leakage through repeated designs, is the same structural fact the rest of this section is about.

Before concluding that the features were weak, we checked whether the models were at least well-calibrated, whether the probabilities they emitted were at least honest about their own uncertainty even where they were not accurate. All eight scored at or worse than the trivial baseline of always predicting 0.5, with Brier scores between 0.2539 and 0.2727 against that baseline's 0.25. Expected calibration error ran 0.11 to 0.17 against a target below 0.05. Reliability curves were non-monotonic even for the models that produced a genuine spread of probabilities, and accuracy in the most confident quintile never exceeded accuracy overall. There was no hidden signal to filter for.

Then we stopped looking at the features and looked at the labels, and there it was.

Context dependence — designs that appear in more than one test, and how often they contradict themselves

• Distinct designs in the corpus: 673
• Appearing in more than one pair: 181 (26.9%)
• Of those, carrying contradictory win-and-lose labels: 73 (40.3%)
• Arm-level rows belonging to a contradictory design: 212 of 942 (22.5%)

The rate is scale-invariant, which is what makes it structural rather than anecdotal. An early check found 3 of 7 (43%) at a sample size small enough to be an artifact; at 181 designs the rate is essentially unchanged, and the training split alone gives 39.8%. Two rates, doing different work: 40.3% is the finding — among designs that appear more than once, that share carries incompatible labels, and it is the rate that tells you the label is not a function of the image. Roughly 11% (73 of 673) is corpus-level exposure — the share of all distinct designs affected, which is what bounds how much of a training set the problem contaminates. The first is the argument; the second is its blast radius.

The conclusion isn't that these features carry no signal. It is that the branch was estimating a quantity that does not exist in the data. A per-design model asks: how good is this design? The data answers a different question: how did this design perform against that design, for that audience, on that metric, at that time. For 40% of the designs that appear more than once, those two questions have incompatible answers. The same artwork won one test and lost another, so no function from image to outcome can fit both rows. The outcome is a joint property of a design and its context, and scoring a design in isolation asks something the labels are structurally unable to answer.

That framing accounts for the pattern of failure better than any story about feature quality. Four target framings and two model families failed identically because the mis-specification is upstream of all of them; the choice of estimator cannot fix the choice of estimand. Negative R² on every fold is what a mis-specified target looks like, not what weak features look like. And quadrupling the corpus changed nothing, including one figure that came back bit-for-bit identical, because more observations of an ill-posed quantity are still observations of an ill-posed quantity. Power was never the binding constraint.

Read that way, the pairwise comparison approach in Section 3 is not merely a stronger method. It is the one that asks a question the data can answer. It never scores a design; it only ever ranks a pair, in a specific context, and reports whether the ranking survives a slot swap. The contrast is as direct as it gets, and it is the strongest evidence available for the argument: the same images that yield nothing when scored one at a time yield 57.28% when a model sees both at once and reasons about what distinguishes them. The features are identical in both cases. The architecture that keeps the pair intact works; the one that takes the pair apart does not. The measurement fix and the modelling failure are the same lesson, arriving twice.

7. Off the benchmark

Every prompt, schema and mechanism choice across twenty iterations was tuned against one benchmark. The honest test of a method like that is a corpus it has never met. We assembled one from a body of real production A/B tests: 278 pairs across 38 companies and 82 distinct metrics, filtered to a Bayesian confidence of 0.70 or better and deduplicated against everything used during development. We ran both shipping configurations on it once.

Accuracy dropped a long way.

Held-out corpus — 278 production pairs, first and only run

• Average Accuracy (chance 50%): Sonnet 5 N=9, 48.0% (267/556); Opus 5 N=3, 48.6% (270/556) — both below chance
• Consistent Accuracy (chance 25%): Sonnet 5 N=9, 36.3%, above chance 15.1; Opus 5 N=3, 40.3%, above chance 20.4
• Cost for the corpus: Sonnet 5 N=9, $21.93; Opus 5 N=3, $34.84

Both configurations remain above the 25% floor on Consistent Accuracy and both sit slightly below 50% on single-call Average Accuracy, which is a useful illustration of why the two metrics cannot be compared without normalization. The difference between them is not significant (McNemar p = 0.1925).

Setting 56.30% and 36–40% side by side invites the reading that the system got worse. That reading is wrong. So is the softer version, "this corpus is just harder." The distinction is between a distribution the method was iterated against twenty times and a distribution it had never seen. Four specific differences drive the gap.

Curation. The benchmark and the vendor corpora are published case studies. Somebody chose them because the result was explicable, because you could look at the two designs and see why one won. That selection filters for visual legibility, which is exactly the property a screenshot-based predictor needs. This corpus is raw production outcomes gated only on statistical confidence, with no requirement that the difference be visible at all.

Confidence is not visibility. 71 of the 278 pairs (26%) have an absolute relative lift under 5%, and 20 of them (7%) under 2%. A confidently real 1% lift can be driven by something no static image contains: page weight, a form validation change, a rendering difference on one device class.

Image provenance. Roughly 52% of the original-state images and 47% of the losing-state images are inferred reconstructions of the before-state rather than independently verified captures. This is a real limitation of the corpus, and we can't rule out that some fraction of the errors are the model correctly reading an image that does not accurately represent what visitors saw.

First unvalidated encounter. This was one run, no tuning, no second attempt. Every choice in the pipeline had been selected against a different distribution.

So the generalization claim we're willing to make is a narrow one. The method retains a measurable edge over chance on a distribution it was not tuned against, at roughly half the normalized margin. Anything stronger would require iterating against this corpus, at which point it stops being held out.

Assembling it also illustrated where the real constraint on work like this sits. It is not labels and it is not models. It is images.

Corpus construction — 2,070 candidate experiments in, 367 usable pairs out

• Candidates: 2,070
• After source exclusions: 930 (−1,140)
• After visual completeness: 840 (−90)
• After arm structure: 668 (−172)
• After deduplication: 367 (−301)

The largest single exclusion category at the first stage is experiments whose arms were not meaningfully distinct from one another, 569 of the 1,140, and that is an action-definition problem rather than a screenshot problem: the variants on offer were not meaningfully different treatments, so there was no contrast for any predictor to read. Verified visual exclusions, meaning missing or unusable screenshots, account for 563, or 49.4% of that stage. The remainder is 7 duplicate payloads and 1 invalid result. Later stages remove partial visual slates (where the recorded outcome is a treatment aggregate and attributing it to one captured arm would fabricate a per-arm causal label), multi-arm experiments, and overlap with corpora already used. Other source packages show the same shape: 32,000 variants yielding 70 usable pairs; 1,431 pairs yielding 74 verified, a 95% loss; 417 raw records yielding 107 after deduplication.

8. Where a weak signal pays, and where it costs

Simulated — read this section as modelling, not measurement. Every number below comes from a simulation, not from a deployment. Nothing here was observed in production. The simulations assume a lift distribution, a traffic rate, a stopping rule, and model errors that are independent of true design quality. All four are assumptions, and the third and fourth are the ones most likely to be wrong. The model itself is illustrative and its code is not committed, so read each figure as a consequence of the assumptions stated alongside it rather than as reproducible output. Where we write "simulation suggests," that is the strongest claim available.

The plan, from the first week of this project, was to feed the prediction into an allocation system as a prior. A bandit that starts from a uniform belief spends its opening hours learning an ordering; if a $0.10 prediction can supply part of that ordering up front, the bandit should converge sooner and spend less traffic on the losing arm. It's the obvious deployment path, and our own simulation says it doesn't work.

Prior seeding (simulated) — days to a decision when the prediction seeds the prior

• Cold start, uniform prior (baseline): 100 days
• Weak prior (10 pseudo-observations): ~96 days, −4% (within noise)
• Capped prior (25 pseudo-observations): ~137 days, +37%
• Strong prior (50 pseudo-observations): ~171 days, +71%

Simulated: 5% baseline conversion, 3% true lift, 2,000 visitors per day, a 95% posterior stopping rule, 150 runs per condition. Stronger priors monotonically delay the decision. One scenario only, and the distribution of outcomes is heavily skewed, so the mean and the median tell different stories. Treat the percentages as directional rather than as point estimates.

The mechanism is just arithmetic on the accuracy we actually have. At 57% Consistent Accuracy the model seeds the wrong arm about 43% of the time. In the 57% of cases where it is right, the bandit would have found that arm anyway, within a few days, using traffic it was going to spend regardless, so the prior buys a small head start on something that was not in doubt. In the 43% where it is wrong, the bandit has to overcome a confidently stated false belief before it can begin converging, and the stronger the prior, the more evidence it takes to dislodge. Being right saves days. Being wrong costs weeks. The asymmetry is what turns a better-than-chance prediction into a net loss.

Stated generally: a prediction placed inside a running experiment competes against live traffic data, and live traffic data wins quickly. Prior-seeding isn't a bad idea in principle, but it needs accuracy substantially higher than we achieved, high enough that the cost of the wrong-arm cases stops dominating. At 57%, and certainly at the 36–40% we saw on unseen production data, it is the wrong place to spend the signal.

Which points at where a weak signal does pay. Before any traffic exists. Selection and filtering do not compete against a working bandit. They compete against whichever candidates someone would otherwise have pulled off the backlog, and that is close to random with respect to outcome.

Selection (simulated) — pick 5 of 20 candidates: lift of the best one selected

• Benchmark level (41.7 normalized): +2.89 percentage points over random, 81% of oracle
• Held-out level, Opus (20.4 normalized): +1.71 percentage points over random, 48% of oracle
• Held-out level, Sonnet (15.1 normalized): +1.28 percentage points over random, 36% of oracle

Simulated: candidate variants with true lifts drawn from a normal distribution centred on zero with 5% standard deviation, model noise calibrated to each normalized accuracy level, 30,000–40,000 trials per condition. "Share of oracle" is the fraction of the gain a perfect selector would achieve. The benchmark level is set at 41.7 normalized, the figure from the superseded 119-pair slate; the cleaned corpus gives 43.04, close enough that the picture is unchanged. Even at the weakest level the signal recovers roughly a third of what perfect foresight would.

Filtering (simulated) — drop the bottom 20% of 20 candidates: is the true best still in the set?

• Random filter (baseline): true best retained 80.0%, best discarded 1 in 5
• Benchmark level (41.7 normalized): true best retained 98.9%, best discarded 1 in 91
• Held-out level, Opus (20.4 normalized): true best retained 93.3%, best discarded 1 in 15
• Held-out level, Sonnet (15.1 normalized): true best retained 90.4%, best discarded 1 in 10

Simulated, same generator as the selection figures above. Filtering is the easier task: it only requires recognising the worst candidates, not producing a correct ordering, and the cost of an error is bounded — you lose one candidate rather than mis-rank a whole backlog. Even at the weakest accuracy level, discarding the true best happens about a tenth of the time against a fifth for a random cut.

The practical version of that is a calendar, not a percentage point. A team with a hundred days of testing budget and a backlog it intends to work through gets those days back in proportion to how much of the backlog it never runs.

The saving, in days (simulated)

• No filter: 100 days
• 20% filtered: 80 days
• 30% filtered: 70 days

Note what sets what: the filter fraction determines the saving, and it is the same at both accuracy levels, since you save 20% of the calendar by running 20% fewer tests, whatever the model's accuracy. Accuracy determines only how often the filter throws away your best candidate. Per-test duration does not change; every test still runs to its own stopping rule. The saving is in running fewer tests, not faster ones.

Adjacent work — nearby published figures, none directly comparable

• SimGym (Shopify, 2026) — direction of an add-to-cart shift. Reported: 77% (chance 50%). Why it isn't a comparison: direction only, no magnitude, and a 50% floor rather than 25%.
• UIClip (UIST 2024) — pick the higher-quality UI. Reported: 75.12% (chance 50%). Why it isn't a comparison: design quality as judged by people, not the outcome of a live test.
• SimAB (ETH / Adobe, 2026) — direction, 47 historical tests. Reported: 67% (83% high-confidence). Why it isn't a comparison: easier task, small sample, 50% floor.
• LOLA (Marketing Science 2024) — best of many headlines. Reported: 46.86%. Why it isn't a comparison: top-1 among many candidates, a harder task on a different floor.
• Netflix surrogate index (2023) — long-term outcome from short-term data. Reported: 95% / 79%. Why it isn't a comparison: early stopping on a test already running, not cold prediction.

Figures as reported by their authors. Listed to show what the adjacent literature measures, not as a ranking. Every row differs from this work in task, chance floor, or both, and no row is a like-for-like comparison with a Consistent Accuracy number.

This is also the shape of the argument the CRO industry already runs on. Spiralyze monitors A/B tests across 78,000-plus sites, capturing around 8,000 new tests a month and inferring winners from which version a site keeps once a test ends. One pattern they publish as an example, a form placed on the left, won 45 times, showed no effect 29 times and lost 43 times. Read as a per-prediction hit rate that is roughly 38%, and it is a published example for one pattern rather than a validated accuracy figure for their engine; they publish no per-prediction accuracy number. They nonetheless offer a performance guarantee, which is coherent for exactly the reason the figures above are: selection does not require per-prediction reliability, only that the ordering beat chance across a sequence.

Three caveats that the simulation cannot resolve. The first is the lift distribution: we assumed one shape, and real candidate quality may be distributed differently. A backlog where every idea is mediocre, or one where a few are transformative, changes what selection is worth. The second one is more serious. The simulation assumes model errors are independent of true design quality. If the model is systematically wrong about a whole category of design, whether a layout convention it has learned to distrust or a visual style underrepresented in whatever it learned from, then filtering would reliably kill that entire category, every time, and the simulation has no way to see it. Given the position bias in Section 2 and the context dependence in Section 6, systematic error is the more likely regime, not the less. The third is that the prior-seeding result is a single scenario with a heavily skewed outcome distribution; we report it because the direction is unambiguous, not because the percentages are precise.

Multi-arm tests should be a better fit than two-arm ones, and we want to be clear that this is reasoning rather than evidence. Directionally the case is straightforward: traffic splits N ways, so every arm converges more slowly and each weak candidate that stays in the test costs more than it would in a two-arm split, which makes removing weak candidates worth more, not less. And filtering asks less of the model in a multi-arm setting than ranking would: it only has to identify the worst candidates, not produce a correct full ordering. But every accuracy figure in this article comes from two-arm comparisons. We have not measured the method on multi-arm experiments at all, and the multi-arm corpora we do have carry post-hoc selection problems of their own. Treat the multi-arm case as untested.

If you wanted to move the numbers rather than deploy them, Section 6 says where not to look. More data will not do it: quadrupling the training corpus changed nothing, and one figure came back bit-for-bit identical. Better features will not do it either, because the ceiling is in the target, not the inputs. What would move it is a change of estimand: a model that takes the pair as input and predicts a relative outcome, ideally with some representation of the context the test ran in, rather than scoring designs in isolation and subtracting. That is what the working system already does at inference time, which is why it remains a prompt loop rather than a trained model, and a trained pairwise model is the direction we would take next.

The finding is not that the system is useless at 57%. It is that 57% is worth something where the alternative is a coin flip, and worth less than nothing where the alternative is a bandit that already works.

9. What generalizes past this problem

Three of the findings here have nothing much to do with conversion optimization.

The first is about judges. If you are using a model to compare two things, it probably prefers a position, that preference is large, it does not cancel when you average several models, and single-call accuracy on a balanced set cannot tell you whether you have a judge or a coin with a bias. So counterbalance, and require agreement. Then publish the chance floor in the same breath as the score, because a metric with a 25% floor read against 50% looks like failure and a metric with a 50% floor read against 25% looks like a breakthrough. This applies to any pairwise preference evaluation: model-versus-model comparisons, preference data collection, automated grading.

The second is about sweeps. An apparent optimum built from separately executed runs may be the maximum of a noisy sample and nothing more. The same-draw control is cheap: run the largest configuration once and re-slice it. We had eight sweet spots and a mechanism story for them, and none of them survived that test.

The third is about estimands, and it is the one we would most want someone else to take. Before improving features, before adding data, before trying a different model class, check that the quantity you are predicting exists. If the same input carries contradictory labels a substantial fraction of the time it appears more than once, the target is not a function of the input, and no amount of representational work will make it one. Four framings and two model families failing identically is a signature. It points upstream of the estimator, at the question.

And the caveats belong in the same sentence as the result, so: the strongest figure here is 59.66% on the superseded 119-pair slate, internal and not peer-reviewed, on a benchmark against which the method was iterated twenty times; the headline configuration is 57.28% on the cleaned 103-pair slice, with no significance test run against it; and on a corpus of real production tests the method had never seen, both fall to 36–40% against a 25% floor, with about half of the before-state images in that corpus being reconstructions rather than verified captures. What we are confident about is the measurement argument, the closed directions, and the reason the trained branch failed. The number is the least durable part of it.

What we'd tell someone starting this

01. Counterbalance from day one. Retrofitting it invalidates every number you have already recorded, which is a slower way to arrive at the same place. Two calls per pair is not expensive; a quarter of a project spent on figures you have to withdraw is.

02. Compute your chance baseline before you quote a number. Not after, and not from intuition. We compared a 25%-floor metric against 50% more than once, and each time it made a real result look like a failure, or, on a different corpus, made a chance-level result look like a finding.

03. Run a same-draw control before you believe any peak. If your sweep points are separate executions, a peak is the expected shape of noise. One batch, re-sliced, settles it for the price of a single run.

04. Hold the test set genuinely out, and group your splits by the thing that repeats. Ours had to be grouped by client and by image, because the same design recurs across experiments. Row-wise cross-validation leaked pseudo-replicates across folds and produced a 61.3% that was really 51.3%.

05. Expect to lose 60–95% of any dataset you are handed. Ours went 2,070 to 367; other packages went 32,000 to 70, and 1,431 to 74. Budget for acquisition and filtering as the main body of work, not the preamble to it, and inspect what the filters remove. That is where we found the arms that were not meaningfully different.

06. Design so that negative results are cheap. Most of the closed directions in Section 4 cost under a dollar and under an hour to settle, because the harness made a new configuration a config change rather than a rewrite. The sixteen answers in that section are the most portable thing we produced.

Sources

The benchmark. Jeon, Kim, Yoon, Shim, Choi, Kim, Kim and Yu, "Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding," ACL 2026 Main. arXiv:2505.05026, latest revision 4 June 2026. 300 real-world UI pairs, each A/B-tested at scale, with verified winners and expert interpretations. All published figures in the field-comparison table are from this paper.Earlier arXiv versions of this work circulated under the title "G-FOCUS"; the peer-reviewed version drops that name, and it is cited here by its current title. Dataset: jeochris/WiserUI-Bench.

Bandit allocation. Coframe, "The Math Behind Coframe's Optimizers." coframe.com. Thompson sampling over a Beta posterior per variant, resampled hourly, with underperforming arms replaced rather than merely down-weighted.

Pattern-level test data. Spiralyze, A/B Test Scraper. spiralyze.com/scraper. 78,000-plus sites monitored, around 8,000 new tests captured monthly, winners inferred from which version is retained after a test ends. The 45 / 29 / 43 figures are their published example for a single pattern, not a validated accuracy figure for their engine; we are not aware of a published per-prediction accuracy number.

Synthetic users. TechCrunch, "Synthetic-user startup Simile raises $200M at $2B valuation," 30 July 2026. techcrunch.com. The company's 85% accuracy figure is self-reported, measured against the General Social Survey, and domain-specific; cross-domain validation is an open question by the company's own framing.

Reasoning that degrades perception (Section 4a). Ghosal et al., "VisRef: Visual Refocusing while Thinking Improves Test-Time Scaling in Multi-Modal Large Reasoning Models," arXiv:2603.00207, February 2026, for the attention-dilution mechanism. Kancheti, Kanade, Balasubramanian and Ganu, "Chain-of-Thought Degrades Visual Spatial Reasoning Capabilities of Multimodal LLMs," ACL 2026 Short Papers (preprint arXiv:2604.16060). Jin et al., "Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do," ACL 2026 Long Papers. "Vision Language Models are Biased," arXiv:2505.23941. Survey of the inverse-U curve: "Towards a Mechanistic Understanding of Large Reasoning Models," arXiv:2601.19928.One disagreement between versions worth flagging: the peer-reviewed ACL version of the spatial-reasoning paper reports sixteen models across thirteen benchmarks; its arXiv preprint says seventeen. We cite the peer-reviewed count.

All figures attributed to this work are internal, measured on the corpora and slice sizes named beside them, and not peer-reviewed. Two of the closed directions in Section 4 lack a committed run artifact and are marked accordingly, as does the allocation model in Section 8. The published rows in the field-comparison table are operator-provided and unverified. Internal corpora are described generically throughout; no client is named.