RL for LLMs: reading sprint map

A fast study map for the 96-paper RL-for-LLMs reading list: what each cluster is about, why the technique appeared, and what to focus on first.

Source

Original reference homepage: https://algoroxyolo.github.io/blog/2026/rl-reading-list/

This Teo Wiki page is a study companion built while following Yunze “Lorenzo” Xiao’s RL for LLMs: The Reading List. The original homepage is the main reference for this week’s RL paper sprint; Teo Wiki preserves its 96-paper taxonomy, reading-depth recommendations, and adds Korean study notes about motivation and key techniques.

How to use this page this week. Read by motivation first, not by chronology. For each section, ask: what bottleneck did the technique try to remove? Then read the A/A− papers in that cluster before skimming B/B+ papers for variations and failure modes.

Reading-depth ranking from the source

The original reading list assigns a reading-depth grade to each paper. Teo Wiki keeps that ranking here so the weekly sprint can distinguish papers to deeply study from papers to skim for context.

A Deep read A− Method + experiments B+ Problem + conclusion B Skim C Core idea only NV NVIDIA paper

Grade	How deeply to read	Practical rule for Teo
A	Deep read	문제의식, 방법, objective/algorithm, 실험, ablation, failure mode까지 읽는다.
A−	Method + experiments	intro와 related work는 빠르게, method와 experiments/ablation 중심으로 읽는다.
B+	Problem + conclusion	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B	Skim	abstract, intro, figure/table, conclusion 위주로 훑고 taxonomy에 위치만 잡는다.
C	Core idea only	핵심 아이디어 한 문장만 저장한다. 원문 목록에는 현재 C 등급 논문이 없다.

Priority rule. Read A papers first as full paper studies. Read A− papers for method and experiments. Use B+ papers to understand variants, failure modes, and conclusions. Skim B papers only enough to place them in the taxonomy.

A Deep read 17 papers

Paper	Cluster	Focus note from source
A 2025 · AceReason-Nemotron 1.1 NV	Algorithms / Cascade & Stage-wise RL (Nemotron)	문제의식, 방법, objective/algorithm, 실험, ablation, failure mode까지 읽는다.
A 2026 · Nemotron-Cascade 2 NV	Algorithms / Cascade & Stage-wise RL (Nemotron)	IMO/IOI/ICPC gold medals; 30B MoE; on-policy distillation
A 2024 · DeepSeekMath (GRPO origin)	Algorithms / Core RL: PPO → GRPO → DAPO → CISPO → MaxRL → DPPO	Start your interview story here
A 2025 · DAPO	Algorithms / Core RL: PPO → GRPO → DAPO → CISPO → MaxRL → DPPO	Recipe + system-level scaling
A 2025 · DeepSeek-R1	Algorithms / Core RL: PPO → GRPO → DAPO → CISPO → MaxRL → DPPO	문제의식, 방법, objective/algorithm, 실험, ablation, failure mode까지 읽는다.
A 2025 · MiniMax-M1 / CISPO	Algorithms / Core RL: PPO → GRPO → DAPO → CISPO → MaxRL → DPPO	Changed clipping target — key interview talking point; also notable for long-context RL (see §4.3)
A 2025 · Qwen3 Technical Report	Algorithms / Scaling Laws & Meta	Thinking/non-thinking unified framework + thinking budget + distillation recipe; source of Nemotron 3 reasoning budget control
A 2025 · ScaleRL / The Art of Scaling RL Compute	Algorithms / Scaling Laws & Meta	문제의식, 방법, objective/algorithm, 실험, ablation, failure mode까지 읽는다.
A 2025 · OLMo 3 (SFT-DPO-RL-RL Zero)	RLHF & Preference Optimization / From Simplification to Online RL	Fully transparent & reproducible; all data/code/checkpoints
A 2024 · Generative Verifiers: RM as Next-Token Prediction	Reward Modeling / Generative Reward Models	346 citations; GRM origin; yes/no token + majority vote
A 2025 · DeepSeek-GRM / SPCT	Reward Modeling / Generative Reward Models	SPCT online RL for GRM training; 193 citations
A 2025 · HelpSteer3-Preference NV	Reward Modeling / Generative Reward Models	40k samples; NVIDIA latest; RM-Bench SOTA
A 2025 · Magistral	Systems / Sync vs Async Training	문제의식, 방법, objective/algorithm, 실험, ablation, failure mode까지 읽는다.
A 2025 · RAGEN	Tasks & Agents / Agent RL Challenges & Self-Evolution	148 citations; Echo Trap — why agent RL fails
A 2025 · Search-R1	Tasks & Agents / Deep Research & Search Agents	790 citations
A 2025 · Nemotron-Tool-N1 NV	Tasks & Agents / Tool-Use RL	NVIDIA; binary reward; must-discuss in interviews
A 2025 · ToolRL	Tasks & Agents / Tool-Use RL	191 citations; reward design textbook

A− Method + experiments 38 papers

Paper	Cluster	Focus note from source
A− 2025 · Nemotron-Cascade 1 NV	Algorithms / Cascade & Stage-wise RL (Nemotron)	Foundational motivation for Cascade 2
A− 2026 · Nemotron 3 Super Technical Report NV	Algorithms / Cascade & Stage-wise RL (Nemotron)	LatentMoE + NVFP4 + multi-env simultaneous RL + reasoning budget control
A− 2022 · InstructGPT	Algorithms / Core RL: PPO → GRPO → DAPO → CISPO → MaxRL → DPPO RLHF & Preference Optimization / Foundations	RM + PPO + KL penalty; shared with §3.1
A− 2024 · RLOO / Revisiting REINFORCE	Algorithms / Core RL: PPO → GRPO → DAPO → CISPO → MaxRL → DPPO	intro와 related work는 빠르게, method와 experiments/ablation 중심으로 읽는다.
A− 2025 · GSPO	Algorithms / Core RL: PPO → GRPO → DAPO → CISPO → MaxRL → DPPO	intro와 related work는 빠르게, method와 experiments/ablation 중심으로 읽는다.
A− 2026 · MaxRL	Algorithms / Core RL: PPO → GRPO → DAPO → CISPO → MaxRL → DPPO	pass@k objective; RL–MLE continuum
A− 2025 · On-Policy Distillation (Thinking Machines)	Algorithms / Distillation (on-policy / self / context)	Clearest explanation of compute tradeoffs
A− 2026 · MiMo-V2-Flash	Algorithms / Distillation (on-policy / self / context)	Multi-teacher on-policy distillation; core Cascade 2 technique
A− 2023 · DPO	RLHF & Preference Optimization / Foundations	Know why it eventually fell short
A− 2024 · Online Iterative RLHF	RLHF & Preference Optimization / From Simplification to Online RL	Turning point: online >> offline DPO
A− 2025 · PPP: Proactive and Personalized Agents	RLHF & Preference Optimization / Multi-turn / Social / Creative RLHF	Three-objective RL; UserVille benchmark
A− 2025 · RL from User Conversations	RLHF & Preference Optimization / Multi-turn / Social / Creative RLHF	Persona-conditioned rewards
A− 2026 · HER: RL for Role-playing	RLHF & Preference Optimization / Multi-turn / Social / Creative RLHF	Dual-layer thinking; relevant to InCharacter work
A− 2025 · RM-R1: Reward Modeling as Reasoning	Reward Modeling / Generative Reward Models	Reason first, then score; 101 citations
A− 2026 · P-GenRM (ICLR 2026 Oral)	Reward Modeling / Generative Reward Models	Personalized GRM — directly relevant to persona research
A− 2024 · PAV / Rewarding Progress	Reward Modeling / Process Reward Models (PRM)	ICLR; 225 citations; process reward = advantage
A− 2025 · GenPRM	Reward Modeling / Process Reward Models (PRM)	PRM test-time generative reasoning; 21 citations
A− 2025 · Lessons of Developing PRMs	Reward Modeling / Process Reward Models (PRM)	Qwen/Tongyi; PRM training practice: what works, annotation, stability
A− 2025 · ReasonFlux-PRM	Reward Modeling / Process Reward Models (PRM)	NeurIPS 2025 Spotlight; trajectory-aware step + trajectory dual supervision
A− 2025 · ThinkPRM	Reward Modeling / Process Reward Models (PRM)	Long CoT verifier; 58 citations
A− 2024 · How to Evaluate RM for RLHF / PPE	Reward Modeling / RM Evaluation & Benchmarks	61 citations; offline-online gap
A− 2025 · Rubrics as Rewards (RaR)	Reward Modeling / Rubrics-as-Rewards	111 citations; rubrics structure subjective preferences
A− 2026 · Golden Goose NV	Reward Modeling / Rubrics-as-Rewards	Non-verifiable → verifiable bridge; high value for research narrative
A− 2025 · AReaL	Systems / Sync vs Async Training	intro와 related work는 빠르게, method와 experiments/ablation 중심으로 읽는다.
A− 2025 · slime (SGLang RL blog)	Systems / Sync vs Async Training	intro와 related work는 빠르게, method와 experiments/ablation 중심으로 읽는다.
A− 2025 · Defeating Nondeterminism (Thinking Machines)	Systems / Train-Inference Mismatch & RL Stability	intro와 related work는 빠르게, method와 experiments/ablation 중심으로 읽는다.
A− 2025 · Give Me FP32 or Give Me Death	Systems / Train-Inference Mismatch & RL Stability	intro와 related work는 빠르게, method와 experiments/ablation 중심으로 읽는다.
A− 2025 · SGLang Deterministic Inference	Systems / Train-Inference Mismatch & RL Stability	intro와 related work는 빠르게, method와 experiments/ablation 중심으로 읽는다.
A− 2025 · Stabilizing RL with LLMs	Systems / Train-Inference Mismatch & RL Stability	IS correction + Routing Replay theory
A− 2025 · TP Sizes Determinism	Systems / Train-Inference Mismatch & RL Stability	intro와 related work는 빠르게, method와 experiments/ablation 중심으로 읽는다.
A− 2025 · Unified FP8 (SGLang blog)	Systems / Train-Inference Mismatch & RL Stability	intro와 related work는 빠르게, method와 experiments/ablation 중심으로 읽는다.
A− 2025 · Absolute Zero	Tasks & Agents / Agent RL Challenges & Self-Evolution	161 citations; zero-data self-play
A− 2026 · iStar	Tasks & Agents / Agent RL Challenges & Self-Evolution	ICLR 2026; implicit PRM for credit assignment
A− 2026 · Qwen3-Coder-Next	Tasks & Agents / Coding Agents	intro와 related work는 빠르게, method와 experiments/ablation 중심으로 읽는다.
A− 2025 · Tongyi DeepResearch	Tasks & Agents / Deep Research & Search Agents	intro와 related work는 빠르게, method와 experiments/ablation 중심으로 읽는다.
A− 2025 · AGENTRL	Tasks & Agents / Generalist Agent RL Frameworks	Async + cross-policy sampling
A− 2026 · RLAnything	Tasks & Agents / Generalist Agent RL Frameworks	Joint optimization of env + policy + RM
A− 2025 · ReTool	Tasks & Agents / Tool-Use RL	231 citations; cold-start synthesis + outcome RL

B+ Problem + conclusion 30 papers

Paper	Cluster	Focus note from source
B+ 2025 · Dr. GRPO	Algorithms / Core RL: PPO → GRPO → DAPO → CISPO → MaxRL → DPPO	Fixes length bias + std normalization
B+ 2025 · PPO Collapse in Long-CoT	Algorithms / Core RL: PPO → GRPO → DAPO → CISPO → MaxRL → DPPO	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B+ 2026 · DPPO	Algorithms / Core RL: PPO → GRPO → DAPO → CISPO → MaxRL → DPPO	TV/KL divergence trust region
B+ 2026 · Self-Distilled Reasoner	Algorithms / Distillation (on-policy / self / context)	Teacher = self with privileged information
B+ 2025 · BroRL NV	Algorithms / Scaling Laws & Meta	Rollouts scaling dimension; complementary to ProRL
B+ 2025 · ProRL NV	Algorithms / Scaling Laws & Meta	Steps scaling dimension; prerequisite to ScaleRL
B+ 2022 · Bai et al. / HH-RLHF	RLHF & Preference Optimization / Foundations	Safety alignment; HH-RLHF dataset
B+ 2025 · Asynchronous RLHF (ICLR 2025)	RLHF & Preference Optimization / From Simplification to Online RL	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B+ 2025 · R-PRM	Reward Modeling / Process Reward Models (PRM)	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B+ 2026 · PRISM	Reward Modeling / Process Reward Models (PRM)	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B+ 2024 · HelpSteer3 dataset NV	Reward Modeling / RM Evaluation & Benchmarks	Feedback + edit; inference-time scaling
B+ 2025 · RewardBench 2	Reward Modeling / RM Evaluation & Benchmarks	63 citations; reward hacking
B+ 2025 · OpenRubrics	Reward Modeling / Rubrics-as-Rewards	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B+ 2026 · Rubric-ARM	Reward Modeling / Rubrics-as-Rewards	Alternating optimization of rubric generator + judge
B+ 2025 · Context-Folding	Systems / Long-Context RL	Long-horizon agent context compression/folding management
B+ 2025 · A-3PO	Systems / Sync vs Async Training	Decoupled PPO; independent dimension
B+ 2026 · GAC	Systems / Sync vs Async Training	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B+ 2026 · StaleFlow	Systems / Sync vs Async Training	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B+ 2025 · FP16 Mismatch	Systems / Train-Inference Mismatch & RL Stability	Counter-intuitive: train/infer path consistency matters more
B+ 2025 · R3: MoE Routing Replay	Systems / Train-Inference Mismatch & RL Stability	Theoretically explained by Stabilizing RL paper
B+ 2025 · AgentPRM	Tasks & Agents / Agent RL Challenges & Self-Evolution	MC rollout actor-critic; 29 citations
B+ 2026 · SWE-Master	Tasks & Agents / Coding Agents	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B+ 2025 · ComputerRL	Tasks & Agents / Computer-Use & GUI Agents	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B+ 2026 · GUI-Libra	Tasks & Agents / Computer-Use & GUI Agents	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B+ 2026 · OmegaUse	Tasks & Agents / Computer-Use & GUI Agents	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B+ 2026 · How to Train DR Agent	Tasks & Agents / Deep Research & Search Agents	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B+ 2026 · Yunque DeepResearch	Tasks & Agents / Deep Research & Search Agents	왜 나온 논문인지와 결론/한계를 잡고, 필요한 경우 method만 선택적으로 본다.
B+ 2025 · Kimi K2.5 Agent Swarm	Tasks & Agents / Generalist Agent RL Frameworks	Multi-agent orchestration for agentic tasks
B+ 2025 · WebAgent-R1	Tasks & Agents / Generalist Agent RL Frameworks	73 citations; end-to-end web-agent RL
B+ 2026 · ProRL Agent NV	Tasks & Agents / Generalist Agent RL Frameworks	NVIDIA; rollout-as-a-service for multi-turn agent RL

B Skim 11 papers

Paper	Cluster	Focus note from source
B 2017 · PPO	Algorithms / Core RL: PPO → GRPO → DAPO → CISPO → MaxRL → DPPO	Know ratio clipping + trust region
B 2025 · SAPO	Algorithms / Core RL: PPO → GRPO → DAPO → CISPO → MaxRL → DPPO	abstract, intro, figure/table, conclusion 위주로 훑고 taxonomy에 위치만 잡는다.
B 2026 · On-Policy Context Distillation	Algorithms / Distillation (on-policy / self / context)	abstract, intro, figure/table, conclusion 위주로 훑고 taxonomy에 위치만 잡는다.
B 2025 · Scaling Behaviors of LLM RL Post-Training	Algorithms / Scaling Laws & Meta	Power-law: model scale × data × compute; complements ScaleRL
B 2024 · SimPO	RLHF & Preference Optimization / From Simplification to Online RL	DPO simplification endpoint
B 2025 · RLMR: Mixed Rewards for Creative Writing	RLHF & Preference Optimization / Multi-turn / Social / Creative RLHF	Subjective aesthetic RM + objective constraint verifier
B 2026 · OMAR (One Model All Roles)	RLHF & Preference Optimization / Multi-turn / Social / Creative RLHF	abstract, intro, figure/table, conclusion 위주로 훑고 taxonomy에 위치만 잡는다.
B 2026 · Social-R1	RLHF & Preference Optimization / Multi-turn / Social / Creative RLHF	abstract, intro, figure/table, conclusion 위주로 훑고 taxonomy에 위치만 잡는다.
B 2026 · Long-form RewardBench	Reward Modeling / RM Evaluation & Benchmarks	abstract, intro, figure/table, conclusion 위주로 훑고 taxonomy에 위치만 잡는다.
B 2025 · QwenLong-L1.5	Systems / Long-Context RL	abstract, intro, figure/table, conclusion 위주로 훑고 taxonomy에 위치만 잡는다.
B 2025 · DeepSeek-V3.2	Systems / Train-Inference Mismatch & RL Stability	abstract, intro, figure/table, conclusion 위주로 훑고 taxonomy에 위치만 잡는다.

Why RL for LLMs became important

Supervised fine-tuning teaches a model to imitate demonstrations, but many modern LLM behaviors are not just imitation problems. Reasoning, tool use, long-horizon agents, coding, search, and preference-sensitive conversation require models to optimize outcomes that may only be visible after several steps. RL re-enters because it gives a language model an objective for trying, receiving feedback, and improving policies beyond static next-token imitation.

The reading list is best understood as a sequence of bottlenecks: stable policy optimization, better reward signals, preference learning, scalable systems, and finally task-specific agents.

Technique map

01 · Algorithms

PPO → GRPO → DAPO/CISPO/MaxRL

Motivation: LLM RL needs stable updates without destroying the base model. Long-CoT and verifiable tasks made old PPO recipes look expensive or brittle.

Technique focus: clipping, KL/trust regions, critic-free or group baselines, pass@k objectives, length-bias fixes, scaling recipes.

02 · Reward modeling

Outcome rewards → generative/process/rubric rewards

Motivation: Many useful tasks are subjective, long-form, partially correct, or non-verifiable. A single scalar outcome reward is too weak.

Technique focus: generative reward models, process reward models, rubrics-as-rewards, reward benchmarks, offline-online reward gaps.

03 · Preference optimization

RLHF → DPO/SimPO → online RLHF

Motivation: Human preference learning began with reward models + PPO, then moved toward simpler offline objectives, then came back to online RL when static pairs hit limits.

Technique focus: preference datasets, DPO-style simplification, online iterative RLHF, personalized and multi-turn preferences.

04 · Systems

RL at scale needs deterministic, stable infrastructure

Motivation: RL amplifies small numerical and serving mismatches. Training and inference paths diverge; async rollout systems become stale; MoE routing can break reproducibility.

Technique focus: sync/async RL, routing replay, FP32/FP16/FP8 consistency, deterministic inference, long-context rollout management.

05 · Tasks and agents

RL for tool use, coding, search, GUI, and general agents

Motivation: Agentic tasks are long-horizon and interactive. The model must choose actions, observe results, recover from failures, and handle sparse delayed reward.

Technique focus: tool-use reward design, web/search agents, coding agents, GUI agents, agent PRMs, echo-trap failures, self-play/self-evolution.

Paper index by category

Algorithms

Core question: how do we update an LLM policy stably and efficiently when feedback comes from rewards rather than demonstrations?

Sub-topic	Key papers	Technique / motivation
Core RL	PPO; InstructGPT; RLOO / Revisiting REINFORCE; DeepSeekMath / GRPO; DeepSeek-R1; DAPO; MiniMax-M1 / CISPO; MaxRL	PPO gives trust-region-style stable updates; GRPO and later variants reduce critic/reward-shaping pain for reasoning RL; MaxRL reframes optimization around pass@k-style objectives.
Scaling laws	Qwen3 Technical Report; ScaleRL; Scaling Behaviors; ProRL; BroRL	Once RL works, the next bottleneck is compute allocation: model scale, data, rollout count, training steps, and reasoning budget.
Cascade / stage-wise RL	AceReason-Nemotron 1.1; Nemotron-Cascade 1; Nemotron 3 Super; Nemotron-Cascade 2	Stage-wise recipes appear when one monolithic RL run is not enough; distillation, budget control, and multi-environment RL become part of the training curriculum.
Distillation	On-Policy Distillation; MiMo-V2-Flash; Self-Distilled Reasoner; On-Policy Context Distillation	RL can be expensive at inference or training time; distillation transfers on-policy behavior into cheaper or more controllable models.

Reward modeling

Core question: what reward signal is rich enough for reasoning, long-form answers, subjective preferences, and agent trajectories?

Sub-topic	Key papers	Technique / motivation
Generative reward models	Generative Verifiers; HelpSteer3-Preference; DeepSeek-GRM / SPCT; RM-R1; P-GenRM	Instead of a black-box scalar scorer, the reward model reasons or generates verification signals. This helps when reward itself requires reasoning.
Process reward models	PAV / Rewarding Progress; Lessons of Developing PRMs; ThinkPRM; GenPRM; ReasonFlux-PRM	Outcome reward is too sparse for long reasoning. PRMs supervise intermediate steps, making credit assignment more local.
Rubrics as rewards	Rubrics as Rewards; OpenRubrics; Rubric-ARM; Golden Goose	Subjective tasks need structured evaluation. Rubrics turn vague preferences into checkable dimensions.
Reward evaluation	PPE / How to Evaluate RM; HelpSteer3 dataset; RewardBench 2; Long-form RewardBench	Offline reward accuracy does not guarantee online RL improvement; evaluation must catch reward hacking and long-form failures.

RLHF and preference optimization

Core question: how do we convert human preferences into model behavior without overfitting static preference pairs?

Sub-topic	Key papers	Technique / motivation
Foundations	InstructGPT; HH-RLHF; DPO	The historical arc: train reward model + PPO, build helpful/harmless data, then simplify preference optimization with direct objectives.
Back to online RL	SimPO; Online Iterative RLHF; Asynchronous RLHF; OLMo 3	Offline DPO-style objectives are convenient but static. Online RL reappears when the model must explore its own current distribution.
Multi-turn / personalized	PPP; RL from User Conversations; RLMR; HER; OMAR; Social-R1	Real assistants need personalized, social, creative, and role-playing behavior; reward must model user context, conversation history, and subjective constraints.

Systems

Core question: what infrastructure is needed so RL results are not artifacts of stale rollouts, nondeterminism, or train-inference mismatch?

Sub-topic	Key papers	Technique / motivation
Sync vs async	Magistral; AReaL; slime; A-3PO; StaleFlow; GAC	RL for LLMs needs fast rollout generation and policy updates. Async systems improve throughput but introduce stale data and off-policy drift.
Mismatch / determinism	Stabilizing RL with LLMs; Give Me FP32; Defeating Nondeterminism; SGLang Deterministic Inference; R3 Routing Replay; FP16 Mismatch; Unified FP8	Small numerical differences can change rollouts and rewards. Deterministic inference and consistent precision/routing become RL correctness issues.
Long context	Context-Folding; QwenLong-L1.5	Long-horizon tasks exceed naive context windows. Context compression and folding are needed to keep agent memory usable during RL.

Tasks and agents

Core question: how does RL train agents that interact with external environments, tools, codebases, browsers, and users?

Sub-topic	Key papers	Technique / motivation
Coding agents	Qwen3-Coder-Next; SWE-Master	Software tasks give relatively concrete rewards but require long-horizon planning, editing, testing, and error recovery.
Search / deep research	Search-R1; Tongyi DeepResearch; Yunque DeepResearch; How to Train DR Agent	Search agents need to decide when to query, read, synthesize, and stop. RL helps optimize multi-step information gathering.
GUI / computer use	OmegaUse; GUI-Libra; ComputerRL	GUI agents operate in messy environments with delayed rewards and partial observability, making trajectory-level learning important.
Tool use	ToolRL; Nemotron-Tool-N1; ReTool	Tool use needs reward design for choosing the right tool, formatting calls, using observations, and not over-calling tools.
Generalist agent RL	RLAnything; AGENTRL; Kimi K2.5 Agent Swarm; WebAgent-R1; ProRL Agent	Generalist agents require environment generation, cross-policy sampling, multi-agent orchestration, and rollout-as-a-service infrastructure.
Failures / self-evolution	RAGEN; iStar; AgentPRM; Absolute Zero	Agent RL can collapse into bad loops such as echo traps. New methods target credit assignment, actor-critic rollouts, and self-play/self-generated tasks.