Phone-use agents on real Android: how they work and where they break

AndroidWorld, the benchmark Google researchers published in 2024, has 116 tasks across 20 Android apps, and it runs them on an emulator. For a benchmark that’s the right call. It’s reproducible, it resets cleanly, and nobody’s real account is on the line.

Production is a different place. The apps most teams want a phone agent to operate are the ones people actually use: a delivery app, a marketplace seller app, a messaging app, their own company’s app. A lot of those check what they’re running on. And the device, its network and the account signed in on it all have to stay healthy for weeks, not for one benchmark run.

This deep dive covers how an agent drives a phone, what changes between an emulator and a real handset, and the failure modes that decide whether a phone agent is still useful after its first week.

Why phones at all

If a browser can do the job, use a browser. It’s cheaper, easier to scale and much easier to observe. The browser agent production checklist is the better starting point for most teams.

Phone-use agents earn their keep when the task only exists on mobile. An app with no web version. A flow that’s mobile-only, which is common for some verification and payment steps. End-to-end testing of your own Android app. Acting inside an app for a user who asked the agent to.

All of those are legitimate. What follows assumes the agent is operating accounts it’s entitled to use, inside the app’s terms. Some apps prohibit automated use outright, even of your own account, so read the terms before you build.

Background and prior art

Automating Android is old. Google’s UI Automator framework, built for testing, drives user apps and system apps from outside the app’s own process, including permission dialogs and split-screen layouts. Appium’s Android driver sits on top of it for cross-platform test suites. And adb has let developers tap, type and capture the screen from a workstation for well over a decade.

What’s new is the thing making decisions. A test script knows exactly which button to press. A phone-use agent gets a goal in plain language, looks at the screen, and picks the next action with a model. The baseline agents that ship with AndroidWorld (M3A and a simpler variant) follow that loop, and so does nearly every phone agent built since.

The core mechanism

Every phone-use agent is a loop: observe, decide, act, check. The interesting part is what each step is built from.

Observing the screen

There are two sources, and good agents use both.

Screenshots. adb shell screencap /sdcard/screen.png writes the current screen to a PNG; it’s in the adb docs. A vision model reads it the way a person would. Screenshots work on anything, including games, WebViews and custom-drawn interfaces. They’re expensive in tokens, though, and the model has to work out for itself what’s tappable.

The view hierarchy. adb shell uiautomator dump writes the current UI tree to an XML file (by default /sdcard/window_dump.xml) with each element’s class, text, resource id, content description, bounds and whether it’s clickable. An accessibility service can read the same kind of tree live: AccessibilityService exposes getRootInActiveWindow(), which returns the root AccessibilityNodeInfo of the active window.

The tree is cheap, precise, and tells you what’s actually a button. It’s also incomplete. WebViews, apps drawn by game engines or some cross-platform toolkits, and badly labelled apps can give you a tree full of unnamed nodes.

The pattern that works: send the model a trimmed tree (only visible, interactable nodes with their text and ids) plus a downscaled screenshot for context, and have it pick elements by index. Coordinates come from the tree, never from the model’s guess about pixels.

Acting on the device

Three levels, from blunt to precise.

adb shell input tap x y, plus input swipe, input text and input keyevent. Nothing to install on the device and it works on any app. Coordinates break the moment the layout shifts, and input text has quirks with spaces and non-ASCII text.
An accessibility service installed on the device. It can act on specific nodes (click, set text, scroll) and send gestures with dispatchGesture(). Actions target elements instead of pixels, so they survive layout changes. You have to install it and the user has to enable it in settings, which is a deliberate Android safeguard.
UI Automator through instrumentation. Precise, cross-app, and designed for exactly this, but it’s a test framework, so you run a test APK alongside your agent.

Most production setups end up with adb for device management and screenshots, plus an accessibility service or UI Automator for element-level actions.

Every pass through the loop pays for a screen capture, a model call and an action, and the model call is usually the slow part. So a phone agent will always be slower than a script doing the same thing. Split the work accordingly. Plain code launches the app, opens a known screen directly where the app allows it, and dismisses the dialogs you already know about. The model handles only the steps that need judgement. Fewer model turns per task means lower cost and fewer chances to go wrong.

Checking the result

After every action, observe again and confirm the expected change happened. This is where phone agents differ most from test scripts. A script fails loudly when an element is missing. An agent will decide the screen looks about right and carry on.

Write explicit post-conditions per step: this text is now visible, this app is in the foreground, this list has one more item. They’re boring to write, and they catch most of the silent failures before they compound.

Emulator or real device

Emulators are the right tool for development and for testing your own app. They cost nothing per device, they snapshot and restore in seconds, and one well-specced server with hardware virtualisation can run several. AndroidWorld’s choice makes sense for exactly those reasons.

The differences show up in three places.

Device integrity

Apps can ask Google’s Play Integrity API what they’re running on. A genuine, certified device gets the MEETS_DEVICE_INTEGRITY label. The verdicts documentation says that on Android 13 and higher this needs hardware-backed proof that the bootloader is locked and the OS is a certified manufacturer image. A stock emulator isn’t a certified device, and the one emulator label Google does issue, MEETS_VIRTUAL_INTEGRITY, is for Google Play Games for PC.

What an app does with the verdict is the developer’s decision. Some ignore it. Some refuse to run. Some let you in and flag the session. You can’t configure your way past it, and trying to spoof integrity is a reliable way to get an account closed.

Network

An emulator on a cloud server goes out through that server’s datacenter IP, with all the reputation baggage described in why AI browser agents get blocked. Apps and their backends see it the same way websites do. A phone on a SIM gets a carrier IP. You can route an emulator through a mobile proxy, and for some tasks that’s enough, but the device is still an emulator.

Hardware

Real phones have real sensors, a camera, a battery that charges and drains, and the manufacturer’s build properties. Emulators simulate all of that. For testing your own app the simulation is fine. For a third-party app that looks at its environment, it shows.

Real devices charge you for all this in maintenance. They get hot, their batteries age, USB connections drop, OS updates arrive on their own schedule, and none of them restore to a clean state in seconds. Resetting a real phone means clearing app data or factory resetting, then waiting.

My split: emulators to build and test the agent and your own app, real devices for production work in third-party apps that care what they’re running on. People will argue for emulators everywhere to save money, and for some apps they’ll be right.

Worked examples

These are illustrative walkthroughs of how the loop plays out, not measurements.

Checking an order for a user

A user asks the agent to check where their delivery is, in an app with no web tracking page. The agent launches the app by package name, reads the tree, finds the orders tab, taps it, reads the status text from the tree, and reports back. A handful of steps.

What goes wrong: the app shows a promotional overlay on some launches. A tree-only agent sees extra nodes and may tap straight through to whatever is underneath, while the screenshot makes the overlay obvious. The fix is a small dismiss-known-overlays step before the main loop, and a post-condition that the orders screen really is showing.

The bigger issue is the login. It has to survive between checks. On a device that’s wiped after each task, the agent logs in every time, which often means a verification code every time, which means a human every time. A persistent device per user removes most of that.

This is emulator territory. Snapshot a clean image, run the agent through sign-up with a generated test account, assert on the final screen, restore the snapshot. Tree-based actions hold up well here because you control the app and can give every element a resource id.

On a real device the reset is the bottleneck. adb shell pm clear <package> wipes one app’s data, which is usually enough for a sign-up test and a lot faster than a factory reset.

Posting from an account the business owns

A small business wants its agent to post updates to its own account in an app with no API for posting. Check the terms first. If automation is allowed, device integrity and network start to matter, because the app sees a login from an emulator on a datacenter IP very differently from the phone that account normally lives on. A persistent real device with its own mobile IP is the closest match to how the account is normally used.

Edge cases and failure modes

Screens the agent can’t see. Apps can mark a window with FLAG_SECURE, which keeps its content out of screenshots, so screencap gives you a black frame. Banking and payment screens often do this. The tree may still be readable. If it isn’t, that step belongs to a human.
Coordinate drift. Different screen sizes, font scaling, a notification banner sliding in at the wrong moment. Prefer element targets, and when you must use coordinates, compute them from the current tree’s bounds every time.
The keyboard. The on-screen keyboard covers half the screen and changes the tree underneath it. Setting text through the accessibility service sidesteps it more reliably than typing does.
Unexpected dialogs. Permission prompts, rate-this-app nags, OS update notices, low-battery warnings. Handle the known ones in code before the model ever sees the screen.
Device health. A hot phone throttles, every step slows down, and your timeouts start firing on tasks that are actually fine. Watch battery temperature and free storage per device; adb shell dumpsys battery reports level and temperature.
The adb connection itself. The adb server on your machine manages connections to the daemon on each device, and it can wedge. Restarting it fixes more than you’d expect. Wireless debugging works on Android 11 and higher, which helps when USB is flaky and adds pairing and network dependencies of its own.
App updates. Apps update themselves overnight, a button moves or gets renamed, and a flow that worked yesterday breaks this morning. Turn off automatic updates on agent devices, update on purpose, and re-run a short smoke test for each app afterwards.
Sign-in challenges. Apps often re-verify a login when something about the device or network changes, with a code sent by SMS or email or a prompt on another device. That’s a human step. Build the pause for it into the task runner before you need it, the same way the checklist describes for browser agents.

What running real phones teaches you

We run cloud Android phones in Singapore, so this section is the one based on our own operations, and it’s about the unglamorous layer under the agent.

Heat is the one that surprises people. Phones packed together without enough airflow throttle, and a throttled phone makes every agent step slower in a way that looks like a model or network problem until you check the temperature.

A phone can answer adb perfectly and still have no working mobile data. The device is “online” to your tooling, and every task on it fails at the first network call. Check the data path itself before you hand a phone to an agent, not only that the device responds.

Persistence is the point. The value of a real device for an agent is the same phone, the same carrier identity and the same signed-in apps, day after day. That’s what we built cloudf.one for: real Android phones hosted in Singapore on dedicated hardware, each with a persistent Singapore mobile IP. It’s located in Singapore, so it suits tasks that are location-agnostic or aimed at Southeast Asia. An agent that needs a US carrier IP should look elsewhere. Whatever host you use, ours included, confirm it supports the control method your agent needs (adb, an accessibility service or UI Automator) before you commit.

To debug any of this you need a screenshot and a tree dump for every step, stored with the task. Observing AI agents: traces, replays and cost per task covers how to set that up. More pieces are in the article index.

References and further reading

AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents, and the code on GitHub
Android Debug Bridge (adb), Android Developers
UI Automator, Android Developers
AccessibilityService API reference
Play Integrity verdicts, Android Developers

Written by Xavier Fok

disclosure: The Agent Ops Report is published by the team that runs Singapore Mobile Proxy and cloudf.one. Any link to those two products is ours and is marked sponsored. No vendor paid for a mention in this article. Last reviewed by Xavier Fok on 2026-09-11.