Zhang, J.; Yu, Y.; Liao, M.; Li, W.; Wu, J.; Wei, Z. UI-Hawk: Unleashing the Screen Stream Understanding for GUI Agents. Preprints2024, 2024082137. https://doi.org/10.20944/preprints202408.2137.v1
APA Style
Zhang, J., Yu, Y., Liao, M., Li, W., Wu, J., & Wei, Z. (2024). UI-Hawk: Unleashing the Screen Stream Understanding for GUI Agents. Preprints. https://doi.org/10.20944/preprints202408.2137.v1
Chicago/Turabian Style
Zhang, J., Jihao Wu and Zhongyu Wei. 2024 "UI-Hawk: Unleashing the Screen Stream Understanding for GUI Agents" Preprints. https://doi.org/10.20944/preprints202408.2137.v1
Abstract
Graphical User Interface (GUI) agents are expected to precisely operate on the screens of digital devices. Existing GUI agents merely rely on current visual observations and plain-text action history, ignoring the significance of history screens. To mitigate this issue, we propose UI-Hawk, a visual GUI agent specially designed to processing screen streams encountered during GUI navigation. UI-Hawk incorporates a history-aware visual encoder and an efficient resampler to handle the screen sequences. To acquire a better understanding of screen streams, we define four fundamental tasks—UI grounding, UI referring, screen question answering, and screen summarization. We develop an automated data curation method to generate the corresponding training data for UI-Hawk. Along with the efforts above, we have also created a benchmark FunUI to quantitatively evaluate the fundamental screen understanding ability of MLLMs. Extensive experiments on FunUI and GUI navigation benchmarks consistently validate that screen stream understanding is not only beneficial but also essential for GUI navigation.
Keywords
GUI Agents; Screen Stream Understanding
Subject
Computer Science and Mathematics, Artificial Intelligence and Machine Learning
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
