Blog / Score centering

Beyond Score Centering:
What Else Can We Optimize In Async RL?

Score centering (SC) has recently drawn wide attention in LLM RL. In async RL the sampler and the trainer hold different weights, so the trajectories the trainer learns from often come from a policy several steps old, which easily makes training unstable. SC's fix is simple: subtract from each token's score its expectation under the sampler, which cancels the drift that the old policy brings in. This post uses experiments to explain the core idea of SC and how it works. Beyond SC itself and the tricks widely discussed alongside it, we follow the same idea to revisit other key components of async RL and make training more stable and efficient. At the end we present two findings that, with SC and the KL penalty on, further improve training quality and stability markedly: (1) replacing the mask based on token-level probability ratios with a mask based on state-level KL divergence; (2) switching how the trainer draws trajectories from the buffer from greedy to random sampling.

First, some notation. In async RL the sampler \(\mu\) and the trainer \(\pi\) are decoupled: the sampler generates trajectories with the weights it holds, and the trainer takes optimizer steps on that data while it keeps updating. If a trajectory reaches the trainer at step \(k\) and was sampled by \(\mu = \pi_{\theta_{k-\mathrm{a}}}\), we say its policy age is \(\mathrm{a}\), i.e. the trainer is now computing gradients on data from a policy \(\mathrm{a}\) steps old.

1Why Do Old Trajectories Dominate The Update In Async RL?

Rather than starting with theory and notation, let us first look at what happens when trajectories of different ages reach the trainer. An interesting finding is that without SC, a trajectory's policy-gradient norm grows roughly linearly with its age, while with SC this trend is well suppressed.

The former is clearly bad. In principle, a token's influence on the policy update should depend roughly only on its advantage and otherwise be treated equally; in some settings one may even want to weaken the influence of very old trajectories. Here instead, the older a trajectory, the more it dominates the gradient update. We discuss the bias introduced by stale trajectories later, but even from a training stability perspective alone, the large differences in gradient norms across samples can already destabilize training.

Left: median gradient norm grows from 1.56 to 5.05 with age for PG, but only to 2.40 for SC. Middle: drift share of the PG gradient rises from 0.09 to 0.86. Right: per-token KL grows from 0.0005 to 0.104.
Figure 1. Gradient profile by policy age. Task: Countdown; model: Qwen3-0.6B, trained at staleness 64. At steps 200, 400 and 600, checkpoints of age 0 to 64 each sample 8 completions for the same 64 prompts, and each trajectory's gradient is evaluated at the current weights. Left: median gradient norm. Middle: drift share of the PG gradient, \(\|g_{\mathrm{PG}} - g_{\mathrm{SC}}\| / \|g_{\mathrm{PG}}\|\), where the drift is the part of the PG gradient that SC removes (derived below). Right: per-token sampler–trainer KL.

The figure above naturally raises two questions: why does the gradient grow linearly with age without SC, and how does SC flatten that line? To answer them, we go back to the policy gradient itself. On-policy, the gradient of the objective is \[ \nabla J = \mathbb{E}_{a\sim\pi}\big[A\,\nabla\log\pi(a)\big]. \] Under async, samples come from the sampler \(\mu\), and the trainer actually computes \[ \mathbb{E}_{a\sim\mu}\big[A\,\nabla\log\pi(a)\big]. \] To make this easy to see, we work in the logit space of a single token. Write \(e_a\) for the one-hot vector of \(a\) and \(p = \pi_{\theta_k}(\cdot\mid s)\). Under a softmax, \(\nabla_z\log\pi(a) = e_a - p\), so the PG gradient for one sampled token is \[ g_{\mathrm{PG}} = A\,(e_a - p). \] But \(a\) was drawn by the sampler, so we take the expectation over the sampler with \(A\) inside. Write \(q = \mu(\cdot\mid s)\) and note that \(\mathbb{E}_{a\sim\mu}[e_a] = q\). Then \[ \mathbb{E}_{a\sim\mu}\big[A\,(e_a - p)\big] = \underbrace{\mathbb{E}_{a\sim\mu}\big[A\,(e_a - q)\big]}_{\text{signal}} \;+\; \underbrace{\mathbb{E}_{a\sim\mu}[A]\,(q - p)}_{\text{drift}} . \] The signal term does not depend on the gap between the two policies, while the norm of the drift term grows linearly with that gap. The gap itself grows roughly linearly with age: every trainer step moves the policy by about \(\delta\) in a similar direction, so at age \(\mathrm{a}\) \[ \lVert q - p\rVert \approx \mathrm{a}\,\lVert\delta\rVert , \] and the drift term grows linearly in \(\mathrm{a}\). Even if the trainer were doing a random walk, the gap would still grow as \(\sqrt{\mathrm{a}}\), which is more than enough to wreck training stability. This explains what we observe.

It is worth adding that, as several earlier blogs point out (for example this one), the drift term of the policy gradient, mapped back to the objective, adds a term to the on-policy objective being maximized: \[ -A\,D_{\mathrm{KL}}(\mu\|\pi) \] So far we argued for removing the drift from the standpoint of training stability. Another reading is that it hands the trainer a KL loss whose strength we do not control, which is not a desirable objective; we come back to this later.

This form also restores some good properties of the on-policy version, such as subtracting a constant baseline leaving the expected policy gradient unchanged; we do not dwell on them here.1

1 For a constant baseline \(c\), \(\mathbb{E}_{a\sim\mu}\big[c\,(e_a - q)\big] = c\,(q - q) = 0\), so subtracting it leaves the expected SC gradient unchanged; without centering, \(\mathbb{E}_{a\sim\mu}\big[c\,(e_a - p)\big] = c\,(q - p) \neq 0\). A fuller derivation is in these slides.

Three rows of box plots of per-trajectory gradient norm against policy age, at steps 200, 400 and 600 and pooled: PG rises with age, SC stays flat, and the drift, PG minus SC, starts near zero at age 0 and rises steadily.
Figure 2. Distribution of per-trajectory gradient norms. Same data as Figure 1. Rows: PG, SC and the drift (PG − SC). Columns: steps 200, 400, 600 and pooled. The x-axis is the policy age of the trajectory. The y-axis (log scale) is the L2 norm, over all model parameters, of one trajectory's loss gradient at the current weights, with the loss divided by the batch's completion tokens and no gradient clipping. Each box covers the 25th to 75th percentile of the trajectories at that age, with a line at the median; whiskers reach 1.5 times the interquartile range. Trajectories with zero advantage are left out.

2What SC Solves, And What Is Left To Solve

With that motivation, SC simply removes the drift term: in policy-gradient space it subtracts \[ \mathbb{E}_{a\sim\mu}[A]\,\mathbb{E}_{a\sim\mu}[\nabla\log\pi(a)] , \] and it is natural that the gradient norm under SC shows no clear correlation with age.

After SC, what on-policy actually wants and the gradient we actually compute are, respectively, \[ g_{\mathrm{PG}} = \mathbb{E}_{a\sim\pi}\big[A\,(e_a - p)\big], \quad g_{\mathrm{SC}} = \mathbb{E}_{a\sim\mu}\big[A\,(e_a - q)\big]. \] The Jacobian behind them is the same, so the difference between the two depends only on the gap between \(p\) and \(q\) (token level) and between \(\pi\) and \(\mu\) (policy level).

This suggests that any method that starts from the gap between \(\mu\) and \(\pi\) may be a good way to improve further on top of SC.

3What Should We Do On Top Of SC?

So what is not a good method? SC combined with importance sampling (IS), we suspect.

Doing SC first and then IS is clearly off. Ignoring truncation, the gradient for one token is \(\tfrac{p(a)}{q(a)}\,A\,(e_a - q)\). Taking the expectation over the sampler, the IS ratio first moves the distribution back to \(\pi\): \[ \mathbb{E}_{a\sim\mu}\Big[\tfrac{p(a)}{q(a)}\,A\,(e_a - q)\Big] = \underbrace{\mathbb{E}_{a\sim\pi}\big[A\,(e_a - p)\big]}_{\text{on-policy}} \;+\; \underbrace{\mathbb{E}_{a\sim\pi}[A]\,(p - q)}_{\text{new drift}} . \] This does nothing to shrink the gap between \(p\) and \(q\); it introduces a new drift instead.

What if we do IS first and then SC, as the original paper describes: use IS to move the distribution on-policy, then apply SC? Without truncation and with exact IS, this is pointless. After IS the expectation is already under \(\pi\), where the score is centered at \(p\), \[ \mathbb{E}_{a\sim\mu}\Big[\tfrac{p(a)}{q(a)}\,A\,(e_a - p)\Big] = \mathbb{E}_{a\sim\pi}\big[A\,(e_a - p)\big], \] so SC does nothing: what it subtracts already has zero expectation under \(\pi\).

With truncation, write the post-IS weight as \(w(a) = f\big(p(a)/q(a)\big)\). SC is then effectively applied on a new distribution \[ \nu(a) = \frac{w(a)\,q(a)}{Z},\qquad Z = \sum_{a} w(a)\,q(a), \] where \(f(\cdot)\) is the processing after IS, such as the truncation in the paper. The score is then centered at \(\nu\), and the expected gradient becomes \[ \mathbb{E}_{a\sim\mu}\big[w(a)\,A\,(e_a - \nu)\big] = Z\,\mathrm{Cov}_\nu(A,\, e_a). \] Clearly, there is no guarantee whether this new distribution or the plain \(\mu\) is closer to the target \(\pi\). It is a rather ill-defined operation.So the reading in the SC paper that IS and SC are orthogonal, and can simply be stacked, is not right: neither SC followed by IS nor IS followed by SC gives a well-defined correction. See these slides for details.

So what is a good method? Yihua's blog has one that follows this design principle: \[ \mathcal{L}_{\mathrm{SC}} \;-\; \beta_{\mathrm{a}}\,D_{\mathrm{KL}}(\mu\,\|\,\pi),\qquad \beta_{\mathrm{a}} = \frac{\beta_0}{\sqrt{\mathrm{a}}} , \] where \(\mathcal{L}_{\mathrm{SC}}\) is the objective after SC. It first removes the whole drift term with SC, then explicitly adds back a KL penalty that pulls the trainer toward the sampler that generated each token. This shrinks the gap between the two distributions and, with it, the bias of the policy gradient after SC. Earlier work has also explored such a penalty: Divergence-Augmented Policy Optimization adds a divergence between the behavior policy and the current policy to keep off-policy updates small and safe.

Note that, unlike the implicit \(-A\,D_{\mathrm{KL}}(\mu\|\pi)\) in PG, the coefficient here is always positive and independent of the advantage. It is a controllable knob that helps us reduce the discrepancy caused by the difference between the two distributions.

One more point: why give the new KL penalty a mysterious coefficient proportional to \(\frac{1}{\sqrt{\mathrm{a}}}\)? Recall that the gradient of the KL penalty is troublesome because it gives trajectories of different ages an unbalanced influence on the batch. If every update step were fully independent, the norm of the KL term would grow as \(\sqrt{\mathrm{a}}\). Later in training, however, consecutive updates point in more and more consistent directions: the gap between an old policy and the current one looks more like steps stacking up along one line, and the norm grows closer to \(\mathrm{a}\) (Figure 3). So \(\beta_{\mathrm{a}}\) can decay over the course of training, \[ \beta_{\mathrm{a}} = \frac{\beta_0}{\sqrt{\mathrm{a}}} \;\longrightarrow\; \frac{\beta_0}{\mathrm{a}} , \] further reducing the influence of very old policies on the KL penalty.

Figure 3. Summing \(\mathrm{a}\) update steps. Schematic with \(\mathrm{a} = 3\) unit steps \(v_1, v_2, v_3\). Left: the steps are mutually orthogonal (independent steps in high dimensions are nearly orthogonal), and the sum has norm \(\sqrt{\mathrm{a}}\). Right: every step points the same way, and the sum has norm \(\mathrm{a}\).

4What Kind Of Mask Do We Actually Need?

We saw above that combining SC with IS does not make much sense. So why do combinations such as SC + TIS and SC + MIS still help stabilize training in many experiments?TIS (truncated importance sampling), first proposed in Yao et al. (2025), reweights each token by the trainer-to-sampler probability ratio and caps it at a constant, \(\min\big(\pi(a)/\mu(a),\,C\big)\), to correct the training-inference mismatch. MIS (masked importance sampling), first proposed in Liu, Li et al. (2025), instead sets the importance weight to zero, masking the update, when the ratio falls outside a trusted range.

The reason is simple: the masking or truncation these combinations carry on top of IS also happens to restrict updates at token positions where \(p\) and \(q\) differ a lot. They use \(\frac{\pi(a\mid s)}{\mu(a\mid s)}\) as a surrogate to decide whether to restrict an update.

Since we already know that bad updates happen when \(p\) and \(q\) differ too much, why take the long way round through the IS ratio to make this mask decision? A metric that follows the idea of SC more directly is the KL between the two distributions at each state \(s\): \[ d(s) = D_{\mathrm{KL}}\big(\mu(\cdot\mid s)\,\|\,\pi(\cdot\mid s)\big) = \sum_{a} q(a)\,\log\frac{q(a)}{p(a)}, \] and we keep only the positions with \(d(s) \le \tau\): \[ g = \mathbb{1}\big[d(s) \le \tau\big]\; A\,(e_a - q). \] Although this metric does not consider the outer, policy-level layer, it looks at the full next-token distribution at each state, which is more information than the ratio of a single token.

It also preserves the good baseline property. For a ratio-based mask, write the per-token weight it puts on the gradient as \(\lambda(a)\): for example the truncated IS weight \(\min\big(\pi(a)/\mu(a),\,C\big)\) in TIS, or in MIS the IS weight \(\pi(a)/\mu(a)\) that is set to zero once the ratio leaves \([l, h]\). Either way \(\lambda(a)\) changes with the sampled token, so for a constant baseline \(c\), \[ \mathbb{E}_{a\sim\mu}\big[\lambda(a)\,c\,(e_a - q)\big] = c\,\Big(\mathbb{E}_{a\sim\mu}\big[\lambda(a)\,e_a\big] - \mathbb{E}_{a\sim\mu}\big[\lambda(a)\big]\,q\Big) \neq 0 , \] whereas \(d(s)\) depends only on the state, \[ \mathbb{E}_{a\sim\mu}\Big[\mathbb{1}\big[d(s)\le\tau\big]\,c\,(e_a - q)\Big] = \mathbb{1}\big[d(s)\le\tau\big]\,c\,(q - q) = 0 , \] so the baseline property is fully preserved.

A similar idea is echoed in Trust Region Masking, which uses the same per-position KL \(d(s)\), but masks at the sequence level: a whole sequence is dropped based on the max or mean of \(d(s)\) over its positions, whereas we mask individual positions.

With \(d(s)\) in hand, we now put the KL mask side by side with TIS and MIS. First, what does the gradient profile look like with the KL mask? An interesting observation is that the KL mask does not know the age of each trajectory, yet the older a trajectory, the larger the gap between \(\mu\) and \(\pi\), the more easily \(d(s)\) exceeds the threshold, and so the more positions get masked. So without any help from IS, the KL mask flattens the profile: at \(\tau = 0.3\) it is even flatter than TIS + SC, and at \(\tau = 0.1\) the trend reverses and the influence of very old trajectories is weakened, which is exactly the effect we said one might want in some settings.

Left: median norm relative to age 0 for six methods. Right: age 64 to age 0 ratio, 3.24 for PG, 1.54 for SC, 1.34 for mask 1.0, 1.15 for TIS plus SC, 1.06 for mask 0.3, 0.74 for mask 0.1.
Figure 4. Gradient profile with the KL mask. Same data as Figure 1. Left: median gradient norm of each method relative to age 0. Right: the age-64 / age-0 ratio, with the fraction of tokens masked at age 64 in parentheses.

The profile is flat, but what about training? We first look at two settings where staleness is not too large. The first is the setting of the original SC paper, staleness 64: the sampler pulls new weights only once every 64 trainer steps, so the age of a batch climbs from 0 to 64 and then drops back to 0 at each sync. The second is a more common async setup: the sampler syncs weights every step, the buffer drops trajectories older than 32 steps, and the trainer takes the oldest ones first (greedy), so the batch age stays at about 31. In both settings SC alone either collapses or oscillates, while adding any one of TIS, the KL mask or the KL penalty on top of SC stabilizes training, and the final results are close.

Final training accuracy for seven methods in two settings. SC plus KL penalty reaches 0.710 and 0.720; SC plus KL mask 0.3 0.716 and 0.726; TIS plus SC 0.734 and 0.701; SC collapses at staleness 64 and reaches 0.486 at age 31; PG and TIS collapse in both.
Figure 5. Final training accuracy at small staleness. Countdown, Qwen3-0.6B, SGD at learning rate \(10^{-2}\), 64 prompts × 8 completions per step, 1,200 steps, one seed per method. Left: the async setting of the SC paper, staleness 64. Right: the common open-source async RL setting, greedy (FIFO) sampling, batch age about 31. Bars: mean training accuracy over the last 50 steps.

The differences only show up at larger staleness. We enlarge the buffer to 128 steps so that every batch has age 127, and now TIS + SC and MIS + SC start to look off: two TIS + SC seeds drop sharply early on, and one MIS + SC seed falls straight to 0 midway through training, while all three seeds of the KL mask climb steadily. In final level the three are about the same; the real advantage of the KL mask is stability.

Figure 6. SC + KL mask against TIS + SC and MIS + SC. Countdown, Qwen3-0.6B, SGD at learning rate \(10^{-2}\), 64 prompts × 8 completions per step, 128-step buffer with every batch at age 127, 2,400 steps, 3 seeds. Top: training accuracy, 50-step trailing mean. Bottom: validation accuracy every 50 steps.

5Combining With The KL Penalty

Since the KL mask alone already stabilizes training, can it be stacked with Yihua's KL penalty from earlier? Recall the design principle: both depend only on the state, not on the sampled token, so stacking them does not bring the drift back, which is why we combine them into a recipe. In full, the recipe maximizes, per token, \[ \mathbb{1}\big[d(s)\le\tau\big]\,\Big(\mathcal{L}_{\mathrm{SC}} \;-\; \beta_{\mathrm{a}}\,D_{\mathrm{KL}}\big(\mu(\cdot\mid s)\,\|\,\pi(\cdot\mid s)\big)\Big),\qquad \beta_{\mathrm{a}} = \frac{\beta_0}{\sqrt{\mathrm{a}}} , \] that is, at positions where \(d(s)\) exceeds the threshold, both the policy term and the KL term are dropped. We compare this recipe with the baselines in three settings with a 128-step buffer: greedy, and random sampling at two learning rates.

Training and validation curves over 1,200 steps in three settings with a 128-step buffer. The recipe and SC plus KL penalty rise smoothly in all three; TIS plus SC and MIS plus SC dip sharply under greedy sampling and at the higher learning rate; DPPO collapses under greedy sampling and at the higher learning rate.
Figure 7. The recipe against the baselines with a 128-step buffer. Setup as in Figure 5, with a 128-step buffer. Columns: greedy at learning rate \(10^{-2}\), random at \(10^{-2}\), and random at \(4\times10^{-2}\) (3 seeds, band = range across seeds). Top: training accuracy, 50-step trailing mean. Bottom: validation accuracy every 50 steps. Recipe: SC + KL penalty + KL mask, \(\tau = 0.3\) in the first two columns and \(0.1\) in the third.
Summary bars for the three 128-step settings. Top: final training accuracy; recipe 0.688, 0.703 and 0.707, 0.735; SC plus KL penalty 0.690, 0.716, 0.751; TIS plus SC 0.593, 0.691, unstable; MIS plus SC 0.666, 0.680, collapsed; DPPO collapsed, 0.700, collapsed. Bottom: steps to reach 0.6.
Figure 8. Summary of the runs in Figure 7. Top: mean training accuracy over the last 50 steps. Bottom: first step at which the 50-step trailing mean reaches 0.6. Third column: mean over seeds.

6Beyond The Algorithm: Design Space And Free Lunches In The System

Beyond the algorithm, async RL as a system leaves a lot of design space, and some free lunches. Let us first walk through it at the system level.

In short, async RL is a set of decoupled samplers and a trainer that talk through a buffer. The sampler \(\mu\) generates trajectories with its current weights and puts them into the buffer; the trainer \(\pi\) picks a batch from the buffer by some rule and takes an optimizer step. Concretely, if a trajectory reaches the trainer at step \(k\) and was sampled by \(\mu = \pi_{\theta_{k-\mathrm{a}}}\), then \(\mathrm{a}\) is its policy age. The trainer is then computing gradients on data from an older policy.

The trainer-sampler loop: six samplers send finished groups to a rollout buffer, the trainer takes 64 groups per step, greedily or at random, groups older than K are dropped, and new weights are pushed to the samplers after every step.
Figure 9. The trainer–sampler loop of async RL. Six vLLM samplers generate continuously and put each finished group, tagged with the weight version \(v\) that produced it, into the buffer. Each step the trainer takes 64 groups, takes one step and pushes the new weights to the samplers; groups older than \(K\) are dropped. A group generated with \(\theta_v\) and trained at step \(k\) has age \(\mathrm{a} = k - v\).

Note that even without touching the loss or the policy gradient, once the rollout buffer holds enough trajectories, simply changing how the trainer samples minibatches from it can change training outcomes. This point gets very little attention in the literature.

For example, the setting used in the original SC paper (staleness 64) does not use a buffer at all. The trainer and the sampler instead synchronize every 64 steps, so the age of each minibatch update is not uniform but follows a sawtooth.

Back to buffer-based sampling. A common small mechanism is a threshold \(K\) on the buffer that dynamically evicts trajectories that are too old (\(\mathrm{a} > K\)), to avoid an overly large gradient mismatch. On top of that, the possible strategies come down to the following three. There is also room for extra knobs, which we skip here.

First in, first out (FIFO): take older trajectories first. It comes in a strict and a non-strict form, depending on whether it skips rollouts in the buffer that have not finished yet; we call the non-strict form greedy. As far as we know, many async RL systems use FIFO or a variant of it. Its advantage is that it wastes as little inference as possible. Its obvious drawback is that every update is built around older trajectories and actively ignores the newer ones in the buffer.

Last in, first out (LIFO): take newer trajectories first, the opposite of FIFO. Its advantage is very low staleness, but its drawback is fatal: it is biased against long trajectories, which then rarely make it into a trainer update, so almost no async RL recipe uses it.

Random sampling (RS): the plainest, uniform random sampling. Its advantage is that it picks up some newer trajectories without biasing against older ones as strongly as LIFO does. As far as we know, some people use variants of it, and for different workloads it is natural to add a knob here to balance staleness against long-trajectory bias. As we show later, without SC, RS also runs into trouble that hurts its stability.

Four panels of batch policy age against training step: a 0 to 64 sawtooth for staleness 64; greedy with a 128-step buffer, climbing to 127 and staying there; LIFO, climbing to 46 and then holding at 3 from step 53; random with mean settling near 52 and max near 127.
Figure 10. Policy age of the training batch under each strategy. Staleness 64 follows from the sync schedule. The three \(K = 128\) panels are measured on the SC + KL penalty runs (setup as in Figure 5); for random sampling, the mean (10-step smoothing) and the maximum are shown.

Of the three strategies above, LIFO has the lowest staleness, but its bias against long trajectories is fatal and almost no one uses it. So is it worth switching between the two most common ones, greedy and RS? We compare greedy and RS on the same 128-step buffer, with no prefill on either side. RS has an obvious cost: some groups exceed age \(K\) and are dropped before they are ever drawn, so we pay compute for generations that are never trained on. Fortunately, over 1,200 steps RS spends only 4% to 7% more FLOPs, because training itself dominates the compute. For a fair comparison, the figure below uses actual training time as the x-axis.

Training and validation accuracy against wall-clock hours for four methods, greedy against random sampling from a 128-step buffer. Random is at or above greedy everywhere.
Figure 11. Greedy (FIFO) against random sampling, against training time. Setup as in Figure 5, 128-step buffer, no prefill. Top: training accuracy (faint: per step; bold: 50-step trailing mean). Bottom: validation accuracy.

The result: at the same training time, RS is better for every method. TIS + SC and MIS + SC drop sharply under greedy but not under RS, and DPPO collapses outright under greedy. Of course, this comparison mixes two effects, because RS also lowers the mean age from 127 to 52. It is also worth noting, as said earlier, that without SC RS cannot save training either: plain PG collapses under both strategies.

The reason RS works so well is also simple: smaller staleness means a smaller gap between \(\pi\) and \(\mu\), which fits exactly the methodology we proposed for improving recipes on top of SC.

One more point: if RS is this good, can we use it without SC? The answer is not so certain. Recall that without SC, old trajectories have a larger norm and more influence. Under greedy, the grad norms within an optimizer minibatch are all similar, so the gradient itself is fairly stable; under RS the effective batch size shrinks, which makes stability worse.

Of course, a good buffer strategy for async RL is far from as simple as shown here, and RS is not a universal solution. But we hope this discussion of SC and async RL draws more attention to these open problems, such as the bias from long trajectories, the remaining effect of staleness under SC, and the extra rollout cost. Going further, how to strike a better trade-off among staleness, cost and bias remains a research direction worth exploring in depth.

7Conclusion

  1. We showed visually what SC actually fixes: the drift term that grows with age and lets old trajectories dominate the update.
  2. We discussed what kind of improvement fits the methodology of SC: start from the gap between \(\mu\) and \(\pi\), and depend only on the state.
  3. We proposed two methods that mitigate what SC leaves unsolved in async RL, the gap between \(\mu\) and \(\pi\): the state-level KL mask, and a new knob, the buffer sampling strategy.

8Limitations

  1. We do not decouple the two sources of the gap between \(\mu\) and \(\pi\): the kernel-level training-inference mismatch, and the policy gap introduced by asynchrony, either in the tail or in the average of the per-state KL gap. We also did not have time to port batch-invariant kernels to ablate this, nor did we analyze which kernels matter most for minimizing (not strictly eliminating) the training-inference mismatch. These are all important open questions.
  2. We do not compare against sequence-level masking, such as the sequence-level masked importance sampling in When Speed Kills Stability, which masks by the sequence-level ratio, and Trust Region Masking, which masks by the max or mean of \(d(s)\) over the sequence. It is possible that sequence-level masking is already enough.Sequence-level masking targets a bias of token-level TIS/MIS: it ignores the mismatch accumulated along the prefix, so the bias grows with length. Deciding at the sequence level accounts for the whole prefix. Note that SC is also a way to reduce variance; for how the bias and variance change across IS, TIS and SC, see these slides.
  3. Our buffer management targets the trainer-bound regime. Random sampling may drop some rollouts that age out before they are drawn, but we find it works well even without any staleness bound. In the rollout-bound regime, trajectories are still taken first come, first served.
  4. Our test models are small. Scaling these experiments up needs more GPUs; if you can offer GPU support, please reach out, we would really appreciate the help.
  5. If you need the code, email yichuan_wang@berkeley.edu. We have been lazy, so the code is not very clean yet.

Acknowledgements

We thank Modal for providing GPUs, Joseph Gonzalez for his advice, and Yingru Li for helpful feedback.

References

  1. Martin Marek and Max Ryabinin. Score Centering Stabilizes Off-policy Reinforcement Learning. arXiv:2609.20807, 2026.
  2. Shu Liu, Qi Fang and Shubham Agarwal. Drift and Distance: When Is Score Centering Enough in Async RL? Blog post, 2026.
  3. Yihua Zhang. Score Function Centering Made Simple: Understanding Trainer-Generator Mismatch in Async RL from the Score Function. Blog post, 2026.
  4. Yihua Zhang. Score Function Centering: Why a Stale Rollout Does Not Have to Be Importance Sampled. Blog post, 2026.
  5. Yichuan Wang. From REINFORCE to Score Centering. Talk slides, 2026.
  6. Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang and Jianfeng Gao. Your Efficient RL Framework Secretly Brings You Off-Policy RL Training. Blog post, 2025.
  7. Jiacai Liu, Yingru Li et al. When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch. Blog post, 2025.
  8. Penghui Qi et al. Rethinking the Trust Region in LLM Reinforcement Learning. ICML 2026. arXiv:2602.04879.
  9. Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li and Baoxiang Wang. Trust Region Masking for Long-Horizon LLM Reinforcement Learning. ICML 2026, PMLR 306. arXiv:2512.23075.
  10. Qing Wang, Yingru Li, Jiechao Xiong and Tong Zhang. Divergence-Augmented Policy Optimization. NeurIPS 2019.
  11. Kanishk Gandhi et al. Stream of Search (SoS): Learning to Search in Language. arXiv:2404.03683, 2024.