en.Wedoany.com Reported - In the reinforcement learning post-training of large language models, the rollout process of generating numerous reasoning answers often incurs high computational costs, yet a large number of these samples may not provide effective training signals. To address this issue, a research team from the Department of Automation at Tsinghua University has proposed an optimization framework named POPO (Group Prioritized Off-Policy Optimization), aimed at significantly reducing the consumption of ineffective samples in RLVR (Reinforcement Learning with Verifiable Rewards) training through group-level off-policy optimization.
The core advantage of RLVR lies in the fact that its reward signals can be automatically verified through rules. However, when training with methods such as GRPO (Group Relative Policy Optimization), the model generates a set of responses for the same prompt. If all responses in this set are correct or all are incorrect, the reward variance is zero, causing the relative advantage term within the group to disappear, and the samples contribute almost nothing to parameter updates. This situation, defined as "ineffective samples," is very common in RLVR, especially for overly simple or overly difficult problems, where the model consumes substantial computational resources without obtaining effective gradients.
Existing coping methods mainly include: DAPO, which filters out ineffective groups by expanding the candidate batch and performing additional rollouts, but at the cost of increased generation expenses; predictive sampling methods, which estimate the prompt success rate before rollout but suffer from insufficient stability in large-scale data or low-epoch training; and trajectory replay methods, which, although saving historically successful responses, cannot address challenges such as intra-group validity judgment and off-policy bias.
The core idea of POPO is to maintain a small replay buffer that stores effective response groups encountered during recent training. In each training step, the model first performs a rollout on the current batch, then divides the response groups into effective groups (variance greater than zero) and ineffective groups (variance equal to zero) based on reward variance. POPO retains the effective groups. For the positions of ineffective groups, instead of re-rolling out, it retrieves the most recent effective groups from the buffer to fill them. This strategy ensures that the training batch is primarily composed of effective samples without incurring additional rollout costs.
Unlike common trajectory-level replay, POPO chooses to perform replay at the group level. A group contains a prompt and its corresponding set of responses, and the entire set of data originates from the same historical policy, preserving intra-group consistency and facilitating subsequent off-policy correction. For the selection of replay groups, POPO primarily relies on two criteria: "quality" (non-zero reward variance) and "distance from the current policy" (prioritizing recently stored effective groups).
To address the bias issue of replayed data, POPO introduces decoupled off-policy optimization, which separates the roles of the "behavior policy" and the "proximal constraint policy." For replayed samples, the algorithm uses importance sampling to correct off-policy bias while maintaining the same trust region constraints as on-policy samples, balancing bias correction and training stability.
The paper evaluates POPO's performance on three types of tasks: mathematical reasoning, numerical planning, and visual geometry. In the mathematical reasoning task, the DeepSeek-R1-Distill-Qwen-1.5B and 7B models were trained on the DeepScaleR dataset; the numerical planning task used the Qwen2.5-3B model trained on the Countdown-34 subset; the visual geometry task used the Qwen2.5-VL-3B-Instruct model trained and tested on Geometry3k. Comparison methods include baselines such as GRPO, DAPO, MoPPS, and ARPO.
From the training curves, POPO converges faster than GRPO, ARPO, and MoPPS on multiple tasks, with overall performance close to DAPO, but with significantly reduced rollout costs. On DSR-1.5B, POPO achieved an in-distribution average score of 55.1, on par with DAPO, and an out-of-distribution average score of 32.9, slightly higher than DAPO's 32.7, with training time reduced from 30 hours for DAPO to 16 hours. On DSR-7B, POPO achieved an in-distribution average score of 63.3, higher than DAPO's 63.2, and an out-of-distribution average score of 51.2, higher than DAPO's 50.8, with training time reduced from 55 hours to 34 hours. On the Countdown task, POPO achieved an average accuracy of 60.4, close to DAPO's 61.5, but with only 205k rollouts (compared to DAPO's 877k), and training time reduced from 5.6 hours to 3.2 hours. On the Geometry task, POPO achieved a score of 50.0, close to DAPO's 50.6, with 492k rollouts (compared to DAPO's 1438k), and training time reduced from 11.2 hours to 6.8 hours.
Ablation studies validated the necessity of each design component in POPO. The method of using KL divergence to find replay groups (POPO-KL) had similar performance but lower computational efficiency; the method of only filtering ineffective groups without filling the batch (GRPO-filter) performed worse than POPO; replay without distinguishing effective groups (POPO-ineff) showed a significant performance drop; and replaying samples from distant history (POPO-stale) led to performance collapse. Regarding the optimization objective, simply treating replayed data as current on-policy data caused performance collapse, while the decoupled off-policy objective performed best.
Further analysis shows that POPO consistently outperforms GRPO and approaches DAPO under different response group sizes (k=4, 8, 16, 32), and can be applied to other basic RLVR algorithms such as RLOO and PPO, and can be combined with active sampling methods like MoPPS.
The paper points out that the core value of POPO lies in shifting RLVR training from "generate more, filter more" to "waste less," by caching and reusing high-quality effective samples that have already been generated, reducing the waste of ineffective rollouts while maintaining training stability. Taking the data in this paper as an example, on the DeepScaleR-7B task, POPO achieved performance close to DAPO using only about 30% of the rollout budget. In the future, the team plans to explore a hybrid strategy combining short-term FIFO cache with long-term reservoir to further reduce the number of online rollouts and increase controllable offline data utilization.
The paper was completed by Professor Ji Xiangyang's team from the Department of Automation at Tsinghua University, with authors including Yixiu Mao, Yun Qu, Cheems Wang, Heming Zou, and Xiangyang Ji. The first author, Yixiu Mao, is a Ph.D. student in the Department of Automation at Tsinghua University and is currently interning at Alibaba's Tongyi Qianwen Large Model Team. The paper is published on arXiv under the number 2606.01281v1.
