AIAgent evaluationStory 06
Agents stumble when work crosses devices
What changedJarvisGUI builds tasks that cross Android, Windows and Ubuntu instead of testing an agent inside one tidy application. Leading open agents struggled to carry state between devices, preserve context and finish long chains of dependent actions. The benchmark exposes a mundane but important truth: real work often changes screens, operating systems and interaction styles halfway through.
The useful part
Why it matters
Cross-device failure is a better warning for automation builders than a high score on isolated clicks inside one application.
Worth doing
What to do next
Add explicit handoff state, checkpoints and recovery tests before asking an agent to span a phone and desktop workflow.
Keep in mind
Good to know
The benchmark focuses on open agents and dynamically composed tasks; results may change quickly as systems and environments improve.
Evidence