en.Wedoany.com Reported - Researchers from Zhejiang University and Kuaishou have proposed MemGUI-Agent, enabling mobile GUI Agents to actively manage contextual memory during long-range tasks, preventing the loss of critical information. The team, from the APRIL Lab at Zhejiang University and the Main Station Technology Department of Kuaishou, has also open-sourced MemGUI-3K, the handheld device GUI Agent dataset with the longest average step count. The MemGUI-8B-SFT model, trained on this dataset, achieved state-of-the-art results among open-data models on the long-range task benchmarks MemGUI-Bench and MobileWorld.
Previously, mobile GUI Agents were prone to information degradation when executing long-range tasks spanning multiple apps and steps. Key information obtained early on, such as prices, contacts, specifications, and dates, would be diluted, misrepresented, or completely forgotten in subsequent steps. Traditional ReAct-style agents append the thought, action, and result of each step to the context, leading to two problems: first, the history grows linearly, causing input length and computational costs to rise continuously; second, critical facts become buried in noise, where the agent might remember it looked up a parameter but forget the specific value.
The core design of MemGUI-Agent is Context-as-Action (ConAct), which elevates context management to the same level as UI clicks, inputs, and swipes. At each step, the agent outputs not only the next action but also how to organize its own working memory. ConAct decomposes the context into three structured fields: Folded Action History, which stores the compressed historical trajectory, with the model folding completed tasks into summaries; Folded UI State, which stores complete UI key facts such as phone numbers, product prices, and specification parameters; and Recent Step Record, which records the observation, intention, action, and result of the most recent step.
The researchers further constructed the MemGUI-3K dataset to train smaller-scale models. This dataset was expanded from 128 seed tasks in MemGUI-Bench, enhanced through entity replacement, memory operation augmentation, and task simplification. It ultimately contains 2,956 successful trajectories, covering 26 Android apps and 7 functional scenarios, yielding 64,430 SFT samples. The average trajectory length is 28.8 steps, 65.1% of trajectories use at least one memory operation, and 88.7% of trajectories include at least one span-level fold. These statistics indicate that the dataset is designed to teach models how to manage working memory in long-range tasks.
On the MemGUI-Bench benchmark, MemGUI-Agent-235B (zero-shot) achieved 37.5% Pass@1, 62.5% Pass@3, and 46.8% IRR, improving Pass@1 by 13.3 percentage points compared to the ReAct-style baseline using the same 235B backbone. MemGUI-8B-SFT achieved 23.4% Pass@1, 35.9% Pass@3, and 30.2% IRR on the same benchmark, improving by 14.0, 15.6, and 15.1 percentage points respectively over the Qwen3-VL-8B-Instruct baseline. On the out-of-distribution benchmark MobileWorld GUI-Only, MemGUI-Agent-235B achieved a 29.1% success rate, 14.6 percentage points higher than the baseline; MemGUI-8B-SFT achieved a 17.9% success rate, surpassing OpenMobile-8B's 17.7%.
Ablation studies on components show that adding UI memory operations, history folding, or self-description steps individually all yield improvements, but all three are indispensable. The complete ConAct achieved a Pass@1 of 40.0% on MemGUI-Bench-40, a significant improvement over the ReAct baseline's 5.0%. Failure analysis indicates that ConAct primarily reduces context-induced hallucinations, with total failures decreasing by approximately 41%, and the most notable improvements seen in process hallucinations and output hallucinations. This research advances the GUI Agent's memory problem from external engineering modules into the strategic model itself, enabling the agent to learn to manage the execution process while performing actions.
The first author of the paper is Liu Guangyi, a doctoral student at the APRIL Lab, Zhejiang University, and the corresponding author is Professor Liu Yong from Zhejiang University. MemGUI-Agent is fully open-sourced, with code, data, models, and training and evaluation pipelines all publicly available.
The bottleneck for long-range mobile tasks is that the model needs to retain UI-derived facts across screens, steps, and apps, including prices, phone numbers, product specifications, social media post content, and text to be copied. Passively appending history is neither compact nor reliable. ConAct enables the agent to output five structured segments at each step: thought, fold, tool call, UI observation, and action intention. Tool calls include both regular UI actions and memory operations such as memory_add, memory_update, and memory_delete. This means context management is handled by the same multimodal policy model in a single forward pass.
The researchers found that simply applying the ConAct protocol to existing models does not automatically improve performance. Zero-shot experiments on different scales of Qwen3-VL models showed that only the strongest, Qwen3-VL-235B-Thinking, significantly benefited from ConAct; smaller models or 235B-Instruct actually saw performance decline when using ConAct in a zero-shot setting. This confirms that active context management is not a simple prompting trick; the model must learn when to compress history, when to write to UI memory, and how to generate reusable step descriptions.
During the data collection phase, the authors used Qwen3-VL-235B-Thinking as a teacher model, executing tasks in an Android environment following the complete ConAct protocol. Subsequently, trajectory-level and step-level plausibility filtering ensured that the supervised training data contained no redundant, erroneous, or detour steps.
The dataset yielded 57,951 plausible training steps and 6,479 plausible test steps. According to data statistics, the average trajectory length of MemGUI-3K is 28.8 steps, with a median of 25 steps; 23.8% of folds are span-level folds, meaning multiple steps are compressed into a single subtask summary.
The experiments report two types of agent settings: the first is MemGUI-Agent-235B, which keeps the Qwen3-VL-235B-Thinking weights unchanged and uses the ConAct protocol in a zero-shot setting; the second is MemGUI-8B-SFT, which starts from Qwen3-VL-8B-Instruct and undergoes LoRA SFT on MemGUI-3K. On MemGUI-Bench, MemGUI-Agent-235B achieved 37.5% Pass@1, 62.5% Pass@3, and 46.8% IRR. Compared to the ReAct-style baseline using the same 235B backbone, Pass@1 improved by 13.3 percentage points, Pass@3 by 15.6 percentage points, and IRR by 16.8 percentage points. It also demonstrated stronger overall performance compared to the agentic workflow driven by Gemini-2.5-Pro.
On MobileWorld GUI-Only, MemGUI-Agent-235B achieved a 29.1% success rate, 14.6 percentage points higher than the Qwen3-VL-235B-Thinking baseline. MemGUI-8B-SFT achieved a 17.9% success rate, 8.5 percentage points higher than Qwen3-VL-8B-Instruct, and also surpassed OpenMobile-8B's 17.7%.
Component ablation experiments were conducted on MemGUI-Bench-40. The ReAct baseline had a Pass@1 of only 5.0%. Adding only UI memory operations raised Pass@1 to 17.5%; adding only history folding raised it to 22.5%; adding only self-description steps raised it to 25.0%. The complete ConAct achieved a Pass@1 of 40.0%, Pass@3 of 62.5%, and IRR of 51.0%. These three components address different issues: history folding controls context growth, UI memory preserves precise facts, and self-description steps provide reliable material for subsequent memory and folding operations.
Case studies show that in long-range mobile GUI tasks, critical information often appears only briefly on an intermediate page, but the agent needs to use it accurately dozens of steps later, even after switching apps. MemGUI-Agent's approach is to actively write key facts into UI memory upon seeing them, and after completing a phase subtask, fold the history into a compact history, allowing the model to directly read the structured state and carry previously seen information into subsequent apps.
Classification of failure cases in the zero-shot ablation setting on MemGUI-Bench-40 shows that the complete ConAct reduced the total number of failures from 99 to 58, a decrease of approximately 41%. Among these, process hallucinations dropped from 52 to 30, and output hallucinations dropped from 30 to 13, showing the most significant improvement. This indicates that ConAct primarily addresses context-induced hallucinations, while improvements in knowledge gaps and intention understanding are smaller, suggesting that the model's app knowledge, task intention understanding, and environmental robustness remain subsequent bottlenecks.
For long-range mobile automation tasks, pushing the GUI Agent's memory problem from external modules into the strategic model itself, enabling the agent not only to execute actions but also to learn to manage the execution process, is a crucial step towards moving from being able to click to being able to get things done.
