Interaction-as-a-Judge

Wayne Chi1,2* I-Hung Hsu2 Li Chen2 Wei-Lin Chiang2 Ion Stoica2,3 Anastasios Nikolas Angelopoulos2 Ameet Talwalkar1 Chris Donahue1 Cho-Jui Hsieh2,4

1 Carnegie Mellon University 2 Arena 3 University of California, Berkeley 4 University of California, Los Angeles

* Work done as part of an internship at Arena.

Interaction-as-a-Judge uses computer-use traces to predict human preferences, verify requirements, train reward models, and improve application generation.
Interaction traces connect application behavior to evaluation, feedback, and training.

Abstract

Coding agents are tasked with producing sophisticated interactive applications such as websites, desktop applications, and games. While humans assess these applications interactively, current automated LLM evaluations primarily focus on code analysis and unit tests. This mismatch can cause LLM judges to miss failures that are immediately apparent to human users. We study Interaction-as-a-Judge, an evaluation paradigm where computer-use agents (CUAs) directly interact with an application. We demonstrate that Interaction-as-a-Judge can improve both evaluation and generation of interactive applications. To this end, we collect and release two complementary datasets: 6,858 pairs of applications with human preferences and corresponding agent traces, and 222 applications with human-verified requirement labels. Relative to standard non-interactive approaches, we find that Interaction-as-a-Judge improves evaluation in terms of predicting human preferences (65.6% → 69.1%) and requirement verification (77.0% → 81.6%). We also observe scaling benefits: across both preference alignment and requirement verification, higher-quality interactions improve performance. Finally, through both inference- and training-time experiments, we demonstrate that Interaction-as-a-Judge can also improve the quality of generated artifacts. At inference time, Interaction-as-a-Judge repairs 75.5% of broken requirements within five attempts, compared to 55.4% for code-only self-feedback. At training time, using Interaction-as-a-Judge as a reward signal improves win-rate over the base model by 6.7pp while reducing both generated code length by 35.7% and build failure rates by 7.2pp compared to a code-only reward signal.

Human preferences

With iterative computer use, Gemini 3 Flash’s agreement with human preferences improves from 65.6% to 69.1%. A trained reward model reaches 70.8%, outperforming the off-the-shelf LLM judges. Interaction traces alone also support reward-model training without access to code.

Preference accuracy for code, code plus screenshot, and three interaction strategies. Iterative interaction reaches 69.1% with Gemini 3 Flash, 70.8% with a full-code reward model, and 69.0% with an interaction-only reward model.
Human preference alignment across LLM judges and reward models. GPT 5.6 Luna shows no statistically significant improvement from interaction.

Correctness verification

With GPT 5.6 Sol, interaction improves balanced requirement-verification accuracy from 77.0% to 81.6%. The largest gain comes from identifying broken requirements: 63.0% to 71.1%. Verification improves as the computer-use model becomes more capable.

Requirement verification and discovery across GPT 5.6 Luna, Terra, and Sol. Interaction improves verification for every model; broken-requirement accuracy rises from 65.2% with Luna to 71.1% with Sol. Discovery recall remains low for both approaches.
Verification of broken and working requirements, their unweighted average, and discovery of broken requirements. Requirement discovery remains challenging for both approaches.

Inference-time scaling

Interaction-based self-feedback repairs 75.5% of initially broken requirements within five attempts, compared with 55.4% for code-only self-feedback. The improvement persists under both interaction-based and code-only evaluation, with a smaller gap between self-reported and externally evaluated success.

Repair success over five iterations using code-only or interaction-based self-feedback. Sol’s interaction evaluation finds 75.5% repaired with interaction versus 55.4% with code only; its code evaluation finds 77.4% versus 63.8%. Interaction also reduces the self-verification gap.
GPT 5.6 Luna repairs and self-verifies; GPT 5.6 Sol evaluates the repairs. The 75.5% and 55.4% results use Sol’s interaction-based evaluation.

Training

Reinforcement learning with an interaction-only reward model improves Qwen-3.5-9B’s win rate over the base model by 6.7 percentage points, equivalent to over +50 points on Arena’s leaderboard according to our internal calibrations. Compared with a code-only reward signal, it generates 35.7% less code and increases build success from 81.3% to 88.5%.

Training curves compare code-only, interaction-only, and build-only reward signals. Interaction-based training improves build success and reduces generated code compared with code-only rewards, while an auto-rubrics judge measures a 6.72 percentage-point win-rate gain over the base model.
Reward scores, build success, code length, and quality after training. Win rates use an internal auto-rubrics judge on successful builds and renders. Training covers six non-game categories.

Datasets

ArenaPreferences. 6,858 application pairs across seven categories, with real user prompts, agentic traces, and human preference labels.

ArenaCorrectness. 222 application versions with 412 human-verified requirements, drawn from 67 game development traces.

Coming soon. Both datasets have not yet been publicly released. Download links will be added here when they are available.

Citation

Download BibTeX 
@misc{chi2026interactionasajudge,
  title  = {Interaction-as-a-Judge},
  author = {Wayne Chi and I-Hung Hsu and Li Chen
            and Wei-Lin Chiang and Ion Stoica
            and Anastasios Nikolas Angelopoulos
            and Ameet Talwalkar and Chris Donahue
            and Cho-Jui Hsieh},
  year   = {2026},
  url    = {https://interaction-judge.github.io}
}