Android Bench: Measuring Frontier Coding Agents on Long-Horizon Mobile Engineering

Lynn Ou
Siva Velusamy
2026

Abstract

Android Bench 2.0 measures coding agents on long Android engineering tasks. It has 30 tasks
written by Android developers: new app development from design mocks, library migrations in
open-source apps, new features in existing apps, and conversions of Flutter and React Native apps to Jetpack Compose. The human-written solutions change a median of 2,336 lines across 42 files. Each task runs against a live Android emulator. Deterministic tests check behavior and stored data, and an LLM judge compares screenshots and accessibility trees with reference screens. Every run gets a partial score from 0 to 1 as well as pass or fail. Each task has a closed variant, where the tests are hidden, and an open variant, where the agent can run them. The best model on the closed variant, claude-sonnet-5-5, passes 41% of tasks with a mean partial score of 0.85. With tests available, the best pass rate is 84% (claude-fable-5-1). Most failed runs miss a requirement rather than fail to build.

Research Areas

×