Findings of EMNLP 2026

AnyAppBench

Same Tasks, Different Apps:
Why Mobile GUI Agents Fail to Generalize?

Tien Tran1*, Namho Koh1*, Daiki E. Matsunaga1, Ayush Jain2‡, Kee Eung Kim1‡†

1KAIST  ·  2Carnegie Mellon University

*Equal contribution   ‡Equal advising   †Corresponding author

Overview of AnyAppBench in three panels: single-app evaluation couples the task with one app; AnyAppBench keeps the user goal and target state fixed across several clock apps; each task instance runs under direct (C1) and sub-goal (C2) conditions, a programmatic verifier decides success, and failures are attributed to planning, grounding, mixed or other causes.
Overview of AnyAppBench. Single-app evaluation mixes up task difficulty, agent strategy and interface effects. AnyAppBench keeps the user goal and the success condition fixed and changes only the app within a functional category. Each task–app pair runs on live Android, directly (C1) and with app-independent sub-goals (C2). A programmatic verifier alone decides success; a VLM judge then labels why each failed run failed.
10
functional categories
100
task templates (48 from AndroidWorld + 52 new)
52
apps (11 original AndroidWorld + 41 new)
520
validated task–app pairs
13
agents evaluated

Abstract

Mobile GUI agents deployed in real settings must work across different applications that support the same functionality. Most existing benchmarks test each task in only one app, so a high score can mean the agent understands the task, or only that it knows that particular app. We introduce AnyAppBench, a category-controlled live Android benchmark that evaluates cross-application generalization while keeping the user goal fixed. It spans 10 functional categories, 100 task templates, and 520 task–application pairs over 52 applications. Agents run from raw instructions and with app-independent sub-goals, and a VLM judge labels every failed run under a fixed failure taxonomy whose reliability is measured by human annotation. We find that, across 13 agents, success on the original application does not transfer reliably to new applications with the same goal. Furthermore, providing high-level sub-goal decomposition produces only small, category-dependent changes that do not close the gap, and the mix of failure types changes with the target interface. Based on those insights, we believe the AnyAppBench benchmark provides an important stepping stone toward robust real-world deployment of mobile GUI agents.

Keep the goal, change the app

If we keep the user's goal fixed and only change the app, does the agent still succeed? A task template fixes the instruction, its parameters, the initial device state, the success condition and the step limit. The app supplies the interface and the internal state through which the agent has to reach that condition. “Set an alarm for 6:00 PM” is the same task in every clock app: success requires an enabled alarm at that time, however the app creates or stores it.

Construction of AnyAppBench in three stages: 10 category seeds from AndroidWorld; cross-product expansion of 100 app-independent task templates (48 AndroidWorld-derived, 52 new) with 52 interface-diverse apps (11 AndroidWorld baselines, 41 newly installed); validation gates requiring a state or UI verifier and human solvability, yielding 520 validated task–app pairs.
Construction. Each category's 10 templates are paired only with apps from the same category. A pair is kept only if it passes every validation gate.
  1. Task templates

    10 categories × 10 parameterized templates. 48 keep the intent of an AndroidWorld task; the other 52 cover operations AndroidWorld does not, such as making a task recurring, forwarding a message thread or adding a world clock. No template refers to a particular app.

  2. App expansion

    The original AndroidWorld app stays as the reference, and we add apps that support the same templates. Candidates are profiled on navigation depth, widget conventions, home-screen density, naming, gestures and number of primary screens; near-duplicate interfaces are dropped. All 52 apps were fixed before any agent was evaluated.

  3. Pair validation

    A pair is kept only if (i) the app supports the same user goal, (ii) an app-specific verifier checks the same success condition, (iii) initialization, execution and verification run end to end, and (iv) a person can finish the task with the agent's six primitives.

Action space tap(x,y)long_press(x,y)swipe(x1,y1,x2,y2)type(text)key(keycode)wait(ms)

Verifiers read durable app state, the final UI state, or both, and return a success label the agent never sees.

10 categories · 52 apps

CategoryOriginal AndroidWorld appNew appsTemplates
AW + new

Template provenance: “AW” templates keep an AndroidWorld task intent; “new” templates were written for AnyAppBench. The full template list for each category is in the per-app results below.

How is it different from existing benchmarks?

BenchmarkLive interactionFixed goal across alternative appsWithin-category app controlTrajectory-level failure diagnosis
AndroidWorld✓✗✗✗
AndroidLab✓✗✗✗
SPA-Bench✓✗†✗✗
GUI-Odyssey✗✗†✗✗
TransBench✗Partial‡✗✗
AnyAppBench (ours)✓✓✓✓

†Multi-app workflows, not controlled app substitution. ‡Static cross-app grounding transfer, not live end-to-end execution.

Success first, then why it failed

Verifiers say whether an episode succeeded, not why it failed. So every task–app pair is run under two conditions, and every failed run receives a diagnostic label. Diagnostic labels never change the verifier outcome or the success rates.

C1

Direct execution

The agent gets the instruction and the current observation, a screenshot plus a filtered accessibility tree. It must work out the steps, find the controls, act, and decide when to stop.

C2

Sub-goal augmented

An external generator reads only the instruction and writes a short app-independent list of sub-goals, prepended for the whole episode. It never sees the app, a screenshot or a trajectory, so the agent still has to find every control itself.

Diagnosis

Trajectory attribution

A two-stage Gemini 3.1 Pro judge labels verifier-failed C1 trajectories; Qwen3-VL-30B-A3B re-labels the same cases as a sensitivity check; 559 human-labelled episodes measure how reliable both judges are.

Six failure classes

Planning
Wrong or incomplete task-level strategy: wrong sub-goal, missing step, stopping early.
Grounding
Reasonable strategy, but the action lands on the wrong element or UI state.
Mixed
Planning and grounding errors both matter and neither alone explains the failure.
Execution / tooling
The action interface or agent framework fails: malformed action, parser error, timeout.
Environment / evaluator
App, emulator, permission or network failure, or evidence of a verifier mismatch.
Unknown
The trajectory evidence does not support a reliable attribution.

Human reference labels

Web annotation tool: a case list on the left; the goal 'In the Clock You app, add a new alarm at 13:50', screenshots and the step trace in the middle; on the right a form with the primary failure mode, planning and grounding scores, confidence and a short rationale. The verifier verdict is shown read-only.
Annotation interface. For 559 sampled verifier-failed C1 episodes across all ten categories, an annotator reviews the goal, the action–thought trace, key screenshots and the read-only verifier outcome, and assigns one failure label with every judge output hidden.

What we found across 13 agents

RQ1 · Cross-app transfer

The same task gets much harder when only the app changes

38.7% → 27.3% −11.4 pts

With every app weighted equally, the 11 instruction-only agents succeed on 38.7% of tasks on the original AndroidWorld apps and 27.3% on new apps (cell-level bootstrap CI [−14.1, −9.1]); with the two hybrids, 43.6% → 31.4%. Every one of the 13 agents drops on both template groups. The strongest agents lose the most: on AW-inherited templates Gemini-3.1-Pro loses 29.0 points and UI Voyager-4B 25.3.

RQ2 · Template provenance

It is not an artifact of the new tasks

−13.3 vs. −13.8 pts

On AW-inherited templates the 13 agents average 46.0% on original apps and 32.7% on new apps; on new AnyAppBench templates, 45.2% and 31.4%. Templates taken from AndroidWorld, on which several agents were tuned, transfer no better. Pairing each template only with its original app overstates generalization by about 13 points.

RQ3 · Sub-goal augmentation

App-independent sub-goals do not close the gap

+3.2 and −3.5 pts

On SMS, File Manager, Maps, Contacts and Clock, sub-goals raise success from 37.8% to 40.9%, but more on original apps (+6.2) than on new ones (+2.3), so the gap widens from 14.9 to 18.8 points. On Tasks, Notes, Finance, Music and Calendar, success falls from 18.3% to 14.8%. Sub-goals say what to do, not where each control is in an unfamiliar interface.

RQ4 · Failure composition

The failure mix shifts with the target app

11.3% → 18.3%

From original-app to new-app failures, the execution/tooling share rises from 11.3% to 18.3% under the Gemini judge, and the grounding share from 42.2% to 46.7% under Qwen. Failure profiles are agent-specific, and the two judges disagree on the fine-grained split, so conclusions rest on the coarse planning/grounding-related vs. other distinction.

Same task, different app

Eight-step trajectory of GPT-5.1 on Google Contacts: the agent creates a contact for Hannah Wilson with phone 366-530-2548, saves it and stars it; the final frame shows verifier success. Trajectory of GPT-5.1 on Right Contact for the same task: the agent enters the name and number but never completes the save and favorite flow; the final frame shows verifier failure.
GPT-5.1 performs the same semantic task, create a contact with a given phone number and mark it as a favorite, in two Contacts apps. Top: on Google Contacts, the AndroidWorld-original app, it fills every field, saves and stars the contact (verifier success). Bottom: on Right Contact, a new app, it enters the name and number but never completes the save-and-favorite flow (verifier failure).

Leaderboard

Signed cross-app change over all ten categories, split by template provenance (Table 1 of the paper). AW app: success on the original AndroidWorld apps. New: success on the new same-category apps. Std: spread across new apps. Δ = New − AW app. Click a column header to sort.

(a) AW-inherited templates (b) New AnyAppBench templates

Success rates are equal-app-weighted percentages; Δ is in percentage points. C1 runs three seeded runs per model–app–template cell. †Hybrid: closed-source planner + UI-Venus-Ground-72B grounder. There is no single all-template score, so sorting ranks by the chosen column only. To add an agent, run the full 520-pair suite and open an issue with the result files.

Sub-goals widen the gap (RQ3)

AllAW appNewGap
C1 (instruction)37.849.434.514.9
C2 (+ sub-goals)40.955.636.818.8
Δ (C2 − C1)+3.2*+6.2+2.3+3.9
95% CI+0.3, +6.1−2.0, +14.8−0.6, +5.2−4.4, +13.1

2,285 matched C1/C2 pairs on SMS, File Manager, Maps, Contacts and Clock, five locally served agents, two runs per condition. Sub-goals are written by Gemini-3.1-Pro from the instruction alone. *Exact McNemar, p < 0.001 (265 pairs gained, 193 lost); CIs from an agent × app cluster bootstrap, B = 10,000. On the other five categories (2,880 pairs) the effect reverses: 18.3% → 14.8%, −3.5 points, CI [−5.0, −2.0].

How far can the judges be trusted?

Judge vs. human6-class acc.κBinary acc.κ
Gemini 3.1 Pro45.1%0.27775.3%0.353
Qwen3-VL-30B-A3B36.4%0.13368.6%0.22

559 verifier-failed C1 episodes with a human label (558 for Qwen), spanning all ten categories. “Binary” is planning/grounding-related (planning, grounding, mixed) vs. the other three classes. Agreement supports the coarse split over fine-grained six-class attribution.

Stacked bars of six-class failure attribution per agent, comparing Gemini 3.1 Pro (G) and Qwen3-VL-30B-A3B (Q) on the same failed episodes. Gemini assigns most failures to planning, while Qwen assigns more to grounding and execution/tooling.
Six-class failure attribution (RQ4). Per-agent label distributions from Gemini 3.1 Pro (G) and Qwen3-VL-30B-A3B (Q) on the same verifier-failed C1 episodes; n is the number of matched episodes per agent. Gemini attributes most failures to planning, Qwen to grounding and execution/tooling, and no class dominates every agent.

Per-app results

Same templates, same agents, different apps. Pick a category to see C1 success for every model on every app, pooled over three runs. The shaded columns are the original AndroidWorld app(s).

Quickstart

The harness extends AndroidWorld and runs on a live Android emulator. Full instructions, the C2 sub-goal pipeline and the failure-analysis scripts are in the code repository.

1
Install

Python 3.11+ environment with the vendored AndroidWorld harness.

2
Emulator + apps

Pixel 6 AVD on API 33 with gRPC; fetch and install the 52 pinned APKs.

3
Run

Launch C1 (or C2) across models and categories; verifiers record success.

# 1. install
git clone https://github.com/AnyAppBench/AnyAppBench-Code.git
cd AnyAppBench-Code/benchmark
conda create -n catbench311 python=3.11 -y && conda activate catbench311
pip install -r requirements.txt && pip install -e .

# 2. emulator (Pixel 6, API 33) and apps
$ANDROID_HOME/emulator/emulator -avd <AVD_NAME> -no-snapshot -grpc 8554
./download_app_generalization_apks.sh
./install_app_generalization_apks.sh
./grant_permissions_app_generalization.sh
python scripts/verify_catbench_task_support.py   # Runnable apps: 52, Scheduled task-app pairs: 520

# 3. run C1 (direct execution), three seeded runs per cell
python scripts/run_catbench_5cat_matrix.py \
  --models UI-Venus-7B,GUI-Owl-7B,MAI-UI-8B \
  --n_task_combinations 3 --task_random_seed 30 --run_id c1_k3

The project was previously called CATBench; module names, CATBENCH_* environment variables and result ledgers keep that name.

BibTeX

@article{tran2026anyappbench,
  title   = {Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize?},
  author  = {Tran, Tien and Koh, Namho and Matsunaga, Daiki E. and Jain, Ayush and Kim, Kee Eung},
  journal = {arXiv preprint arXiv:2609.34139},
  year    = {2026},
  note    = {Findings of the Association for Computational Linguistics: EMNLP 2026}
}