You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
System dialogs (runtime permissions, SystemUI alerts) are not clickable — agent stalls until step budget exhausts #180
When a system-owned dialog surfaces mid-task — a runtime permission dialog (com.android.permissioncontroller), a SystemUI alert (USB debugging authorization, ANR / "app isn't responding", crash dialog), or similar — ARTEMIS cannot click its buttons. The dialog's nodes appear to be filtered out of the perceived hierarchy (or taps are dispatched to the underlying app window), so the agent retries the same action until the step budget is exhausted and the task fails.
Reproduction
Fresh install an app that requests a runtime permission (e.g. Camera).
Run a goal that triggers the dialog, e.g. Open the Camera app and allow camera access.
When the permission dialog appears, the planner correctly decides to tap While using the app / Allow — but the dialog is never dismissed. Subsequent screenshots still show it, and the task ends in FAIL after the step budget.
Observed on Pixel 7 Pro emulator (API 33) with the flash profile while testing the Android Studio plugin (#153); the same class of failure applies to any SystemUI / permissioncontroller dialog.
Expected vs actual
Expected: dialog buttons are exposed to the explorer as clickable targets and taps dispatch to the dialog window, so the flow continues.
Actual: the dialog stays up; the agent loops on it.
Analysis / pointers into the code
artemis/utils/ui_filter.py (~L486, ~L497): nodes whose package is com.android.systemui are treated as high-confidence fixed bars (status bar / nav bar) and recursion stops there. That is right for bars, but SystemUI- or permissioncontroller-owned dialogs risk being classified away by the same path.
artemis/utils/app_launch_utils.py (~L60): task tracking already recognizes permission dialogs / account pickers / system pages as foreground helper activities — so ARTEMIS knows the dialog is in front; the perception → action path just doesn't follow through.
artemis/utils/ocr_xml_fusion.py (~L96): SystemUI is special-cased at the top of the fusion pass.
Proposed direction
Either:
Only apply the fixed-bar filtering when the window actually is a full-width bar (bounds/height check), letting dialog-type SystemUI/permissioncontroller windows through to the clickable set; or
Add a fallback: when the foreground package is com.android.permissioncontroller / com.android.systemui and a dialog window is present, bypass the bar filter and expose the raw hierarchy (plus OCR fusion) for that frame.
Happy to send a PR with unit tests (fake-XML fixtures including a com.android.permissioncontroller grant-dialog hierarchy and a SystemUI alert) if the direction sounds right.
Activity