Ben Trengrove

Ben Trengrove

Research Areas

Authored Publications
Sort By
  • Title
  • Title, descending
  • Year
  • Year, descending
Preview abstract Android Bench 2.0 measures coding agents on long Android engineering tasks. It has 30 tasks written by Android developers: new app development from design mocks, library migrations in open-source apps, new features in existing apps, and conversions of Flutter and React Native apps to Jetpack Compose. The human-written solutions change a median of 2,336 lines across 42 files. Each task runs against a live Android emulator. Deterministic tests check behavior and stored data, and an LLM judge compares screenshots and accessibility trees with reference screens. Every run gets a partial score from 0 to 1 as well as pass or fail. Each task has a closed variant, where the tests are hidden, and an open variant, where the agent can run them. The best model on the closed variant, claude-sonnet-5-5, passes 41% of tasks with a mean partial score of 0.85. With tests available, the best pass rate is 84% (claude-fable-5-1). Most failed runs miss a requirement rather than fail to build. View details
×