Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training

Zhiyuan Wang¹

Shengcai Liu¹

Jiahao Wu^1,2

Ning Lu^1,3

Hui Ouyang^1,5

Shaofeng Zhang^1,4

Haoze Lv^1,4

Ke Tang¹

Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation, Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China
Hong Kong Polytechnic University
Hong Kong University of Science and Technology
Zhongguancun Academy
Shenzhen Loop Area Institute

Corresponding authors:: Ke Tang

本文总阅读量次

Abstract

Tool-using large language model (LLM) agents produce long, multi-turn trajectories, making gradient-based post-training memory-intensive. Evolution strategies (ES) enable memory-efficient full-parameter post-training without backpropagation and can eventually match the performance of gradient-based reinforcement learning (RL). However, resource-constrained settings typically offer only a few GPUs, so the high GPU-hour requirements of ES translate into prohibitively long training times. To address this, we introduce Cooperative Parameter-subspace Evolution Strategy (CoPES), a cooperative coevolutionary method that decomposes the full parameter space into lower-dimensional subspaces and searches over them cooperatively to improve optimization efficiency. We post-train a Qwen3.5-4B tool-using agent for the math task and evaluate it on five benchmarks of varying difficulty. Under the GPU-hour budget of full-parameter GRPO's best validation checkpoint, CoPES recovers 92% of GRPO's validation-accuracy gain, versus 67% for standard ES, while its theoretical GPU memory requirement is less than one-eighth that of full-parameter GRPO. It consistently outperforms standard ES and LoRA-based GRPO on all evaluated pass@ \(k\) metrics across the five benchmarks. Additional experiments further show the advantage of CoPES on the question-answering task. These results demonstrate an improved trade-off between memory requirements and training time for agentic LLM post-training under resource constraints. The code is open-sourced in https://github.com/MetaronWang/CoPES

1Introduction

Tool-using large language model (LLM) agents solve complex tasks by interleaving language generation with external actions and environmental feedback [1]-[3]. Post-training with verifiable rewards offers a scalable way to improve such behavior without requiring annotated trajectories [4]. Reinforcement learning (RL) has consequently become a mainstream paradigm for agentic post-training, with current approaches dominated by gradient-based methods [5],[6]. However, unlike models solving conventional single-turn tasks, tool-using agents often produce long, multi-turn trajectories. Backpropagating through these trajectories creates a substantial GPU memory bottleneck for agentic LLM post-training.

Gradient-based post-training must store model weights, gradients, optimizer states, and the intermediate activations required for backpropagation. Prior work has proposed several approaches to reduce GPU memory usage during post-training. Group Relative Policy Optimization (GRPO) eliminates the need for a learned critic [7], while low-rank adaptation (LoRA) reduces gradient and optimizer-state memory by updating only low-rank adapters [8]. However, neither avoids backpropagation through the policy model and the resulting need to store intermediate activations. Because activation memory grows with context length, this unresolved cost becomes especially severe for agentic post-training with long, multi-turn trajectories [9]. In practice, high-memory GPUs are costly, and obtaining them at scale is often difficult. These constraints motivate our focus on resource-constrained settings with limited per-GPU memory and only a small number of available GPUs, where the activation cost of long trajectories can make gradient-based agentic LLM post-training difficult to deploy.

Evolution strategies (ES) alleviate this memory bottleneck by avoiding backpropagation. Standard ES evaluates randomly perturbed models using scalar rewards and aggregates the results into a parameter update, requiring only forward generation and reward evaluation [10]. It can therefore optimize all model parameters without storing parameter gradients or backpropagation activations, substantially reducing GPU memory usage [11]. Recent work shows that ES can eventually achieve performance comparable to gradient-based RL in LLM post-training [12], but typically requires substantially more GPU-hours to do so [11]. Moreover, existing studies focus primarily on single-turn reasoning tasks. In agentic settings, the high cost of generating each long, multi-turn trajectory further magnifies this computational disadvantage. Although ES evaluations are highly parallelizable, resource-constrained settings provide too few GPUs to absorb the additional GPU-hours through parallelism, resulting in impractically long wall-clock training times. The central challenge is therefore to improve the optimization efficiency of ES without sacrificing its memory advantage.

To address this challenge, we propose a cooperative coevolutionary method [13] for ES-based LLM post-training, termed Cooperative Parameter-subspace Evolution Strategy (CoPES). High dimensionality often reduces the effectiveness of evolutionary optimization, motivating cooperative coevolution to divide the search space into smaller subproblems [14]. CoPES applies this principle to full-parameter ES by cooperatively searching lower-dimensional parameter subspaces. By confining each perturbation to one subspace, each scalar reward reflects a lower-dimensional change under a shared full-model context rather than simultaneous changes across the entire parameter space, allowing the fixed evaluation budget to be used more effectively. CoPES scales perturbations by subspace dimensionality to match the expected squared norm of a full-space perturbation, jointly standardizes rewards across subspaces, and synchronously composes the resulting updates. Together, these designs improve parameter exploration under fixed evaluation and GPU-hour budgets while retaining the forward-only, memory-efficient, and full-parameter nature of standard ES.

Our contributions are threefold:

We introduce CoPES, which combines parameter-space decomposition and joint reward standardization for ES-based LLM post-training. To our knowledge, CoPES is the first cooperative coevolutionary method for full-parameter agentic LLM post-training.
We show that CoPES improves fixed-budget optimization efficiency while preserving the memory-efficient, full-parameter advantages of ES, consistently outperforming standard ES and LoRA-based GRPO within the GPU-hour budget of full-parameter GRPO's best validation checkpoint.
Experiments on five math and three question-answering benchmarks, together with hardware-feasibility tests and controlled ablations, validate CoPES across tasks and multiple metrics while isolating the effects of its key components.

2Related Work

2.1Agentic LLM Post-Training

Recent agentic reinforcement learning methods optimize multi-turn tool-use trajectories using outcome-based or verifiable rewards [5],[15]. For example, Search-R1 trains LLMs to issue multiple search queries during reasoning and incorporate retrieved evidence [16], while ReTool combines a supervised cold start with outcome-driven RL to learn when and how to invoke a code interpreter [17]. These methods commonly build on gradient-based policy optimization. GRPO removes the learned critic by estimating advantages from groups of sampled responses [7], and LoRA reduces the number of trainable parameters through low-rank adapters [8]. Neither approach eliminates policy-model backpropagation. CoPES instead avoids policy-model backpropagation while optimizing all model parameters, addressing the broader GPU memory overhead of gradient-based post-training that critic-free and parameter-efficient methods only partially reduce.

2.2Evolution Strategies for LLM Post-Training

Evolution strategies estimate parameter updates from the rewards of randomly perturbed models and therefore do not backpropagate through sampled trajectories [10],[18]. Recent work has demonstrated that evolutionary optimization can scale directly to billion-parameter LLMs [12],[19]. ESSAM combines ES with sharpness-aware maximization to improve generalization on mathematical reasoning while requiring only inference-level GPU memory [11]. EGGROLL instead represents individual perturbations with low-rank factors to support batched evaluation at large population sizes [20]. Collectively, these studies establish evolutionary optimization as a viable backpropagation-free alternative for LLM post-training. Nevertheless, standard ES is computationally intensive, often requiring substantial GPU-hours [11]. In resource-constrained settings, limited parallelism translates this computational demand into long wall-clock training times. CoPES targets this efficiency bottleneck in agentic LLM post-training through cooperative parameter-subspace searches while retaining the forward-only, full-parameter nature of standard ES.

2.3Cooperative Coevolution

Cooperative coevolution was introduced to evolve complex solutions as interacting, coadapted subcomponents [13],[21]. For continuous optimization, it decomposes a high-dimensional decision vector into subspaces, optimizes the corresponding subproblems, and evaluates each partial solution within a context supplied by the remaining subcomponents. Subsequent studies developed a large-scale optimization framework with random grouping and adaptive weighting [14] and introduced differential grouping to identify interacting variables and construct subproblems that better reflect variable interdependence [22]. Recent work has extended this paradigm to 1.7-million-dimensional neural policy search and scalable many-objective optimization [23]-[25]. CoPES builds on this literature by adapting cooperative coevolution to full-parameter ES-based LLM post-training.

3Methodology

Building on cooperative coevolution [13], we propose Cooperative Parameter-subspace Evolution Strategy (CoPES) for full-parameter ES-based agentic LLM post-training. At each training step, CoPES randomly partitions the parameter space into equally sized subspaces, allocates a fixed total of \(N\) perturbations across them, and evaluates every subspace perturbation in the context of the current full model. It then jointly standardizes the resulting rewards and composes the subspace directions into a full-parameter update. Figure 1 illustrates this workflow. We first formalize agentic post-training and the standard ES update, then present the three central designs of CoPES and its memory-efficient implementation.

Overview of CoPES. At each training step, the model parameters are randomly partitioned into K disjoint subspaces, and N/K perturbations are evaluated within each subspace under a shared full-model context. Rewards from all subspaces are jointly standardized, and the resulting subspace estimates are composed into a full-parameter update. — **Figure 1.** Overview of CoPES. At each training step, the model parameters are randomly partitioned into \(K\) disjoint subspaces, and \(N/K\) perturbations are evaluated within each subspace under a shared full-model context. Rewards from all subspaces are jointly standardized, and the resulting subspace estimates are composed into a full-parameter update.

3.1Problem Formulation and Evolution Strategies

Agentic Post-Training Objective.

Given a task prompt \(x\sim\mathcal{D}\), an LLM policy \(\pi_{\boldsymbol{\theta}}\) interacts with a tool environment \(\mathcal{E}\) to produce a multi-turn trajectory \(\tau=(x,a_1,o_1,\ldots,a_T,o_T)\), where actions are sampled from the policy and observations are returned by the environment. A verifier assigns the completed trajectory a scalar reward \(R(\tau)\). The post-training objective is

\[J(\boldsymbol{\theta})=\mathbb{E}_{x\sim\mathcal{D},\tau\sim(\pi_{\boldsymbol{\theta}},\mathcal{E})(\cdot\mid x)}\left[R(\tau)\right],\](1)

where \(\boldsymbol{\theta}\in\mathbb{R}^{d}\) contains all trainable model parameters. The post-training algorithm only observes completed trajectories and their scalar rewards; it does not require gradients through either the environment or the reward function. For an optimization algorithm \(\mathcal{A}\) and a GPU-hour budget \(B\), let

\[\boldsymbol{\theta}_{B}=\mathcal{A}(\boldsymbol{\theta}_{0},\mathcal{D},B)\](2)

denote the parameters obtained within that budget. Our objective is to maximize \(J(\boldsymbol{\theta}_{B})\) under a fixed GPU-hour budget \(B\). The experimental protocol defines how \(B\) is measured across methods, while the derivations below use the per-step perturbation budget to compare standard ES and CoPES.

Standard Evolution Strategy.

Following the one-sided ES update adopted for full-parameter LLM post-training [10],[12], standard ES samples \(N\) independent Gaussian directions for the current parameters \(\boldsymbol{\theta}\) and constructs

\[\boldsymbol{\epsilon}_{i}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{d}), \qquad \boldsymbol{\theta}_{i}=\boldsymbol{\theta}+\sigma\boldsymbol{\epsilon}_{i},\](3)

where \(\sigma\) is the full-space perturbation scale. Each perturbed model generates one trajectory \(\tau_{i,x}\) for every \(x\in\mathcal{B}\), producing the batch-mean reward

\[r_i=\frac{1}{|\mathcal{B}|}\sum_{x\in\mathcal{B}} R(\tau_{i,x}).\](4)

ES standardizes these \(N\) rewards using

\[\begin{gathered} \mu_r=\frac{1}{N}\sum_{i=1}^{N}r_i, \qquad s_r=\sqrt{\frac{1}{N}\sum_{i=1}^{N}(r_i-\mu_r)^2},\\ \hat r_i=\frac{r_i-\mu_r}{s_r}, \end{gathered}\](5)

and applies

\[\boldsymbol{\theta}^{+}=\boldsymbol{\theta}+\frac{\alpha}{N}\sum_{i=1}^{N}\hat r_i\boldsymbol{\epsilon}_{i},\](6)

where \(\alpha\) is the effective update step size and is set independently of \(\sigma\). This update uses no mirrored samples, unperturbed-model baseline, momentum, or gradient-based optimizer. With a limited population, each sampled direction spans the entire \(d\) -dimensional parameter space and is evaluated through a single scalar reward. This motivates decomposing the search while keeping the total number of perturbation evaluations fixed.

3.2Cooperative Parameter-Subspace Evolution Strategy

Cooperative Subspace Search.

CoPES replaces the monolithic \(d\) -dimensional search with \(K\) cooperative searches over lower-dimensional subspaces. At every training step, it samples a uniform random partition \(\{\mathcal{S}_{k}\}_{k=1}^{K}\) of all parameter indices such that

\[\begin{gathered} \mathcal{S}_{k}\cap\mathcal{S}_{\ell}=\varnothing\ (k\neq\ell), \quad \bigcup_{k=1}^{K}\mathcal{S}_{k}=\{1,\ldots,d\},\\ d_k=|\mathcal{S}_k|=\frac{d}{K}. \end{gathered}\](7)

We assume that \(d\) and \(N\) are divisible by \(K\) to simplify notation. The partition is resampled each step so that parameters are not permanently restricted to a fixed grouping. Let \(P_k:\mathbb{R}^{d_k}\rightarrow\mathbb{R}^{d}\) embed a subspace vector into the full parameter space. CoPES allocates \(N_k=N/K\) perturbations to each subspace and constructs

\[\boldsymbol{\epsilon}_{k,i}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{d_k}), \qquad \boldsymbol{\theta}_{k,i}=\boldsymbol{\theta}+\sigma_k P_k\boldsymbol{\epsilon}_{k,i}.\](8)

Only parameters in \(\mathcal{S}_k\) are perturbed; all remaining parameters retain their current values and provide a shared full-model context. Each perturbed model is evaluated on the same batch \(\mathcal{B}\), yielding

\[r_{k,i}=\frac{1}{|\mathcal{B}|}\sum_{x\in\mathcal{B}}R(\tau_{k,i,x}).\](9)

Although each subspace receives only \(N/K\) perturbations, its search dimension is reduced from \(d\) to \(d/K\). More importantly, each scalar reward is associated with a perturbation confined to one subspace under a shared full-model context, rather than with simultaneous perturbations across the entire parameter space. Because the subspaces have disjoint supports, CoPES obtains \(K\) mutually orthogonal subspace estimates and composes them into a full-parameter update.

Dimension-Aware Perturbation Scale.

Using the full-space scale unchanged would reduce the magnitude of a subspace perturbation simply because it contains fewer dimensions. For \(\boldsymbol{\delta}=\sigma\boldsymbol{\epsilon}\) and \(\boldsymbol{\delta}_{k}=\sigma_kP_k\boldsymbol{\epsilon}_{k}\), their expected squared norms are

\[\begin{aligned} \mathbb{E}\!\left[\|\boldsymbol{\delta}\|_2^2\right] =d\sigma^2,\quad \mathbb{E}\!\left[\|\boldsymbol{\delta}_{k}\|_2^2\right] =d_k\sigma_k^2. \end{aligned}\](10)

Matching these quantities gives

\[\sigma_k=\sigma\sqrt{\frac{d}{d_k}}=\sqrt{K}\,\sigma.\](11)

This adjustment matches the expected squared perturbation norm of full-space ES while retaining the same update step size \(\alpha\).

Joint Reward Standardization and Cooperative Update.

After evaluating all \(N=\sum_k N_k\) perturbed models, CoPES pools their batch-mean rewards and computes

\[\begin{gathered} \mu_c=\frac{1}{N}\sum_{k=1}^{K}\sum_{i=1}^{N_k}r_{k,i}, s_c=\sqrt{\frac{1}{N}\sum_{k=1}^{K}\sum_{i=1}^{N_k}(r_{k,i}-\mu_c)^2},\\ \hat{r}_{k,i}=\frac{r_{k,i}-\mu_c}{s_c}. \end{gathered}\](12)

Independent standardization would estimate a separate mean and standard deviation from only \(N_k\) samples in each subspace. Joint standardization instead uses all \(N\) rewards to estimate a shared mean and standard deviation, providing more stable normalization statistics and allowing the subspace directions to be combined on a common reward scale. The shared training batch, matched expected squared perturbation norm, and common reward function provide a consistent evaluation basis across subspaces.

CoPES estimates each subspace direction and composes the full update as

\[\begin{aligned} \boldsymbol{g}_{k}=\frac{1}{N_k}\sum_{i=1}^{N_k}\hat r_{k,i}\boldsymbol{\epsilon}_{k,i}, \quad \boldsymbol{\theta}^{+}=\boldsymbol{\theta}+\alpha\sum_{k=1}^{K}P_k\boldsymbol{g}_{k}. \end{aligned}\](13)

All perturbed models are evaluated and their rewards are standardized before any parameter update is applied. Consequently, every \(\boldsymbol{g}_k\) is estimated from the same pre-update model, and their composition is a synchronous full-parameter update at the algorithmic level. Algorithm 1 summarizes one CoPES training step.

Input:Parameters \(\boldsymbol{\theta}\), shared batch \(\mathcal{B}\), reward \(R\), population \(N\), and subspace count \(K\)

Parameters: Full-space scale \(\sigma\) and step size \(\alpha\)

Output:Updated parameters \(\boldsymbol{\theta}^{+}\)

1Sample an equal-size random partition \(\{\mathcal{S}_{k}\}_{k=1}^{K}\);

2Set \(d_{k}=d/K\), \(N_{k}=N/K\), and \(\sigma_{k}=\sqrt{K}\sigma\);

3for \(k=1,\ldots,K\) do

4for \(i=1,\ldots,N_{k}\) do

5Sample \(\boldsymbol{\epsilon}_{k,i}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{d_{k}})\);

6Set \(\boldsymbol{\theta}_{k,i}=\boldsymbol{\theta}+\sigma_{k}P_{k}\boldsymbol{\epsilon}_{k,i}\);

7Generate \(\tau_{k,i,x}\) for every \(x\in\mathcal{B}\);

8Compute \(r_{k,i}\) using Equation ((9));

10end for

12end for

13Jointly standardize all \(r_{k,i}\) using Equation ((12));

14for \(k=1,\ldots,K\) do

15Compute \(\boldsymbol{g}_{k}\) using Equation ((13));

17end for

18\(\boldsymbol{\theta}^{+}\leftarrow\boldsymbol{\theta}+\alpha\sum_{k=1}^{K}P_{k}\boldsymbol{g}_{k}\);

return \(\boldsymbol{\theta}^{+}\)

Algorithm 1 CoPES

3.3Memory-Efficient Implementation and Cost

Seed Replay, Chunked Processing, and Weight Backup.

Following OpenAI-ES [10], CoPES represents each perturbation by a random seed and regenerates its Gaussian direction during the update, avoiding storage of full perturbation vectors; model parameters are processed in chunks. Unlike prior work that restores parameters by subtracting replayed perturbations, CoPES backs up the pre-perturbation weights in comparatively inexpensive CPU memory and restores them after each perturbed-model evaluation, avoiding residual errors from the non-reversibility of floating-point addition and subtraction and ensuring that every perturbation is evaluated from the same pre-update model. CoPES also stores a partition seed and replays the chunk-to-subspace assignments, avoiding a parameter-level mask. These mechanisms support stable, memory-efficient execution rather than constituting separate algorithmic contributions.

Evaluation and Memory Costs.

CoPES matches standard ES in using \(N\) perturbed-model evaluations and the same number of generated trajectories per step; each perturbed model generates a trajectory for every prompt in \(\mathcal{B}\). Thus, decomposition does not reduce the cost of an individual evaluation but reallocates the fixed perturbation budget across lower-dimensional subspaces. Like standard ES, CoPES requires only forward generation and reward evaluation, storing no parameter gradients, optimizer states, or backpropagation activations.

4Experiments

We organize the main experiments around three research questions. RQ1: In agentic LLM post-training, can ES-based methods match gradient-based RL in performance, and how do their memory efficiency and GPU-hour cost compare? RQ2: Under resource constraints, how does CoPES compare with standard ES and GRPO variants in hardware feasibility and fixed-budget performance? RQ3: How do the number of subspaces and joint reward standardization affect CoPES? We study these questions on the math task evaluated across five benchmarks and conclude with additional experiments on multi-hop question-answering.

4.1Experimental Setup

Models and Tasks.

We post-train Qwen3.5-4B [26] on the math task and multi-hop question-answering (QA) task. Our main experiments focus on math, while the QA experiments examine the applicability of CoPES to a different task. For math, we post-train on the MATH training set [27] and evaluate on AIME 2024^{https://huggingface.co/datasets/HuggingFaceH4/aime_2024}, AIME 2025^{https://huggingface.co/datasets/test-time-compute/aime_2025}, GSM8K [28], MATH-500 [29], and MATH-Test; MATH-500 is a curated subset of MATH-Test. For QA, we post-train on the HotpotQA training set and evaluate on HotpotQA, 2Wiki, and MuSiQue [30]-[32]. Dataset descriptions and split details are provided in Supplementary A.

Baselines and Hyperparameters.

We compare CoPES with Qwen3.5-4B and Qwen3.5-9B without post-training, full-parameter GRPO [7], LoRA-based GRPO [8], and standard ES. Following recommended settings from prior work, all post-training methods use a prompt batch size of \(64\). Full-parameter and LoRA-based GRPO use learning rates of \(1\times10^{-6}\) and \(5\times10^{-6}\), respectively [7],[33]. Standard ES and CoPES both use \(N=40\) perturbations per step, \(\sigma=1\times10^{-3}\), and \(\alpha=5\times10^{-4}\) [12],[19]. CoPES additionally uses \(K=4\), yielding \(\sigma_k=\sqrt{K}\sigma=2\times10^{-3}\). We evaluate model performance on a held-out validation set every 8 steps and select each method's best observed validation checkpoint. Full-parameter GRPO's selected step defines the fixed budget in RQ2. Other hyperparameters and implementation details are provided in Supplementary A.

Rewards and Metrics.

The base task score is answer accuracy for math and F1 for QA. For both tasks, the training reward additionally includes a tool-use bonus for a positive task score and format penalties. We report pass@ \(k\) [34] for math based on repeated sampling. For QA, we report accuracy, exact match (EM), and F1, each averaged over 32 samples per problem. Since our experiments were conducted on servers equipped with different GPU types, we report class-level GPU-hour estimates. We profile CoPES and full-parameter GRPO for 32 steps on the same machine as representatives of the ES-based and RL-based method classes, respectively, and extrapolate their average per-step costs to the reported training steps. Complete reward definitions, metric calculations, sampling protocols, and GPU-hour accounting are provided in Supplementary A.

Agent and Compute Environment.

We use a tool-using agent equipped with Local Wiki Search over Wiki18 [16] and a Python Sandbox. GPU-hour profiling is performed on a server with \(8\times\) RTX 5880 Ada GPUs. To accelerate training and evaluation, we additionally use servers equipped with NVIDIA A30, RTX A6000, and A100 GPUs.

4.2Memory and Computational Efficiency of ES

	Pass@1 (%) on Benchmarks					Train Steps	Est. GPU- hours	Valid Acc.	Run on a 24GB GPU
	AIME 2024	AIME 2025	GSM8K	MATH-500	MATH-Test	Train Steps	Est. GPU- hours	Valid Acc.	Run on a 24GB GPU
Reference Model
Qwen3.5-4B	17.13	35.04	9.22	57.44	59.02	-	-	49.20	-
Validation-Selected Checkpoints
\(+\) Full-param GRPO	44.28	58.68	94.19	89.71	91.66	16	78.61	91.47	\(\times\)
\(+\) LoRA-based GRPO	41.86	54.75	94.38	89.34	91.40	64	314.45	91.07	\(\times\)
\(+\) Standard ES	45.26	61.69	94.02	89.80	91.47	112	480.60	91.73	\(\surd\)
Post-Training for 16 Update Steps (Comparable Estimated GPU-hours)
\(+\) LoRA-based GRPO	28.14	40.76	47.49	77.19	79.18	16	78.61	73.60	\(\times\)
\(+\) Standard ES	31.33	44.45	46.88	78.88	80.43	16	68.65	77.60	\(\surd\)
\(+\) CoPES (ours)	38.39	52.75	92.57	88.29	90.16	16	68.65	88.00	\(\surd\)
Larger-model Reference (No Additional Post-Training)
Qwen3.5-9B	16.67	30.42	51.23	77.72	81.24	-	-	-	-

Table 1. Pass@1 (%) and training costs on five math benchmarks. The 16-step block matches update count and unique prompt batches; estimated GPU-hours are 78.61 for the GRPO variants and 68.65 for the ES variants. It is therefore not an exact GPU-hour-matched comparison. MATH-500 is a subset of MATH-Test.

Theoretical GPU memory requirements versus context length with FlashAttention dao2022flashattention and KV caching, excluding other memory optimizations. — **Figure 2.** Theoretical GPU memory requirements versus context length with FlashAttention [35] and KV caching, excluding other memory optimizations.

Effectiveness in Agentic Post-Training.

Table 1 shows that standard ES performs comparably to the RL-based methods at its selected checkpoint, demonstrating its viability for agentic post-training. Standard ES obtains the highest pass@1 on three of the five math benchmarks.

Theoretical Memory Efficiency.

Figure 2 compares the theoretical GPU memory requirements of different methods on Qwen3.5-4B across context lengths. At 128K, the theoretical requirement of even LoRA-based GRPO is more than \(30\times\) that of ES. These are accounting estimates rather than measured peak GPU memory; calculation details are provided in Supplementary B.

Pass@k of four post-training methods under the 16-step full-parameter GRPO budget, with Qwen3.5-4B and Qwen3.5-9B without post-training as references. — **Figure 3.** Pass@ \(k\) of four post-training methods under the 16-step full-parameter GRPO budget, with Qwen3.5-4B and Qwen3.5-9B without post-training as references.

Computational Efficiency.

Despite competitive performance and memory efficiency, standard ES reaches its selected checkpoint at substantially higher computational cost. As shown in Table 1, standard ES is estimated to require 480.60 GPU-hours, more than six times the estimated 78.61 GPU-hours for full-parameter GRPO. In resource-constrained settings with only one or two GPUs, this computational demand translates into impractically long wall-clock training times.

Answering RQ1, standard ES can match RL-based performance with substantially less GPU memory but at considerably greater GPU-hour cost.

Method	1\(\times\) 24GB	1\(\times\) 48GB	4\(\times\) 48GB	8\(\times\) 48GB
Full-param GRPO	\(\times\)	\(\times\)	\(\times\)	\(\surd\)
LoRA-based GRPO	\(\times\)	\(\times\)	\(\times\)	\(\surd\)
Standard ES / CoPES	\(\surd\)	\(\surd\)	\(\surd\)	\(\surd\)

Table 2. Training feasibility with a 128K context limit. \(\surd\) denotes successful training, whereas \(\times\) indicates an out-of-memory failure.

4.3Feasibility and Performance under Resource Constraints

Feasibility across Hardware Configurations.

We evaluate training feasibility on actual hardware with all memory optimizations supported by the respective frameworks enabled; implementation details are provided in Supplementary A. As shown in Table 2, the ES-based methods can be trained on a single 24 GB GPU, whereas, among the evaluated configurations under the 128K context setting, both GRPO variants succeed only on \(8\times48\) GB GPUs. The smallest tested configuration supporting ES therefore provides less than one-eighth of the aggregate GPU memory capacity of the smallest tested configuration on which GRPO succeeds. Thus, when only one or a few GPUs are available, the GRPO-based methods are infeasible while the ES-based methods remain trainable.

Matched-step performance.

In resource-constrained settings, a limited number of GPUs also limits the GPU-hours that can be spent within an acceptable wall-clock time. Full-parameter GRPO reaches its best observed validation checkpoint at step 16, as shown in Table 1. We therefore train all four post-training methods for 16 steps to ensure that they process the same number of training prompts while keeping their GPU-hour costs comparable. The ES-based methods have slightly lower estimated GPU-hour costs than the RL-based methods, but the difference is small. Under this budget, Table 1 and Figure 3 show that CoPES consistently outperforms standard ES and LoRA-based GRPO across the five benchmarks while remaining competitive with full-parameter GRPO. Relative to Qwen3.5-4B without post-training, CoPES recovers 92% of the validation-accuracy gain achieved by full-parameter GRPO, compared with 67% for standard ES.

Larger- \(k\) Results on AIME.

Figure 3 shows that, as \(k\) increases, CoPES surpasses full-parameter GRPO on the challenging AIME 2024 and AIME 2025 benchmarks despite using the same fixed 16-step training budget.

Answering RQ2, CoPES retains the single-GPU feasibility of ES while outperforming standard ES and LoRA-based GRPO under the fixed budget; at larger \(k\), it also surpasses full-parameter GRPO on AIME.

4.4Ablation Studies

The Number of Subspaces.

Holding all other settings fixed, we compare \(K\in\{1,2,4,8\}\) using pass@1 on the five benchmarks after 16 training steps. As \(K\) changes, we set \(N_k=N/K\) and adjust the perturbation scale as \(\sigma_k=\sqrt{K}\sigma\). Table 3 shows that performance improves as \(K\) increases from 1 to 4, but drops sharply at \(K=8\). At \(K=8\), the population per subspace falls to \(N_k=5\), while the parameter partition becomes highly fragmented; both factors may reduce update quality and degrade performance. For this setup, we recommend \(N_k\geq10\); this threshold may not generalize to other configurations.

Joint Reward Standardization.

The Indep. Z-score row in Table 3 reports the results when perturbation rewards are standardized independently within each subspace. Joint reward standardization achieves higher pass@1 on every benchmark, validating its effectiveness.

Perturbation-Scale Control.

To test whether the improvement of CoPES over standard ES is attributable solely to its larger per-coordinate perturbation scale (\(\sigma_k=2\sigma\)), we additionally evaluate standard ES with the same scale under the fixed budget. The ES- \(2\sigma\) row in Table 3 shows that increasing the perturbation scale of standard ES degrades its performance across the benchmarks rather than improving it.

Variant	AIME 2024	AIME 2025	GSM 8K	MATH 500	MATH Test
\(K=1\) (Std. ES)	31.33	44.45	46.88	78.88	80.43
\(K=2\)	34.48	46.98	82.07	85.96	87.70
\(K=4\) (Default)	38.39	52.75	92.57	88.29	90.16
\(K=8\)	16.61	26.56	37.32	58.07	59.38
Indep. Z-score	21.71	30.30	90.98	87.68	89.45
ES-\(2\sigma\)	24.56	41.28	15.12	65.28	52.25

Table 3. Ablation results after 16 training steps (pass@1, %). \(K\) denotes the number of subspaces; Indep. Z-score uses per-subspace reward standardization; ES- \(2\sigma\) is standard ES with twice the perturbation scale.

Answering RQ3, a moderate subspace count and joint reward standardization improve CoPES, while scale control rules out larger perturbations as the sole source of its gains.

4.5Additional Experiments on Multi-hop QA

We further examine whether the effectiveness of CoPES extends beyond the math task. Full-parameter GRPO's best observed validation checkpoint is at step 48, defining the common 48-step post-training budget. All other method-specific hyperparameters and agent settings remain the same as in the math task. As shown in Table 4, all three post-training methods substantially improve over Qwen3.5-4B across the multi-hop QA benchmarks.

Benchmark	Metric	Qwen	GRPO	Std. ES	CoPES
2Wiki	Acc.	23.41	60.95	55.59	59.06
	EM	6.06	52.16	45.28	48.29
	F1	8.68	58.49	51.53	55.50
HotpotQA	Acc.	29.51	53.87	53.22	53.30
	EM	8.27	47.33	46.52	46.96
	F1	12.70	59.38	58.22	58.83
MuSiQue	Acc.	10.69	29.89	29.07	29.68
	EM	2.67	23.07	21.55	23.34
	F1	4.75	33.28	31.70	33.73

Table 4. Multi-hop QA performance (%). Qwen: Qwen3.5-4B without post-training; GRPO: full-parameter GRPO; Std. ES: standard ES; CoPES: ours. Post-training methods run for 48 steps. Bold and underlining indicate the best and second-best results, respectively.

CoPES outperforms standard ES on all nine benchmark-metric combinations, extending its advantage to another agentic task and reward signal. Against full-parameter GRPO, it achieves higher EM and F1 with comparable accuracy on MuSiQue, remains close across metrics on HotpotQA, and trails on 2Wiki. Together with its hardware feasibility, these results support CoPES as a practical option for resource-constrained QA post-training. We additionally evaluate accuracy-based pass@ \(k\) on QA; CoPES notably surpasses full-parameter GRPO on 2Wiki and MuSiQue for \(k\geq2\), as detailed in Supplementary E.

5Conclusion

We introduced Cooperative Parameter-subspace Evolution Strategy (CoPES), a cooperative coevolutionary method that improves the fixed-budget optimization efficiency of ES-based agentic LLM post-training through parameter-subspace search and joint reward standardization. Experiments on math and multi-hop QA show that CoPES substantially improves over standard ES while approaching full-parameter GRPO, without sacrificing the memory-efficient, full-parameter nature of ES. These findings make ES-based agentic LLM post-training more practical under limited GPU resources. Current experiments focus on Qwen3.5-4B and two agentic tasks; future work will extend CoPES to a broader range of models and agent environments, with particular emphasis on adaptive parameter-space partitioning.

References

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, Y. Cao, "ReAct: Synergizing Reasoning and Acting in Language Models", in Proceedings of ICLR 2023, 2023.
C. Jiang, X. Shu, H. Qian, X. Lu, J. Zhou, A. Zhou, Y. Yu, "LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch", in Proceedings of ICLR, 2025.
H. Lv, N. Lu, Z. Zhou, S. Liu, "AHD Agent: Agentic Reinforcement Learning for Automatic Heuristic Design", CoRR, vol. abs/2605.08756, 2026.
N. Lu, S. Liu, J. Wu, W. Chen, Z. Zhang, Y. Ong, Q. Wang, K. Tang, "Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets", in Proceedings of ICML 2025, 2025.
G. Zhang, H. Geng, X. Yu, Z. Yin, Z. Zhang, Z. Tan, H. Zhou, Z. Li, X. Xue, Y. Li, Y. Zhou, Y. Chen, C. Zhang, Y. Fan, Z. Wang, S. Huang, F. P. Velez, Y. Liao, H. Wang, M. Yang, H. Ji, J. Wang, S. Yan, P. Torr, L. Bai, "The Landscape of Agentic Reinforcement Learning for LLMs: A Survey", Trans. Mach. Learn. Res., vol. 2026, 2026.
J. Wu, N. Lu, S. Liu, K. Wang, Y. Yang, L. Qing, K. Tang, "Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Model", CoRR, vol. abs/2603.25184, 2026.
Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, D. Guo, "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models", CoRR, vol. abs/2402.03300, 2024.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, "LoRA: Low-Rank Adaptation of Large Language Models", in Proceedings of ICLR 2022, 2022.
V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, B. Catanzaro, "Reducing Activation Recomputation in Large Transformer Models", in Proceedings of MLSys 2023, 2023.
T. Salimans, J. Ho, X. Chen, S. Sidor, I. Sutskever, "Evolution Strategies as a Scalable Alternative to Reinforcement Learning", CoRR, vol. abs/1703.03864, 2017.
Z. Sun, S. Dang, G. Dai, H. Ye, "ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning", CoRR, vol. abs/2602.01003, 2026.
X. Qiu, Y. Gan, C. F. Hayes, Q. Liang, Y. Xu, R. Dailey, E. Meyerson, B. Hodjat, R. Miikkulainen, "Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning", in Proceedings of ICML 2026, 2026.
M. A. Potter, K. A. D. Jong, "A Cooperative Coevolutionary Approach to Function Optimization", in Proceedings of PPSN 1994, pp. 249--257, 1994.
Z. Yang, K. Tang, X. Yao, "Large Scale Evolutionary Optimization Using Cooperative Coevolution", Inf. Sci., vol. 178, no. 15, pp. 2985--2999, 2008.
D. Jiang, Y. Lu, Z. Li, Z. Lyu, P. Nie, H. Wang, A. Su, H. Chen, K. Zou, C. Du, T. Pang, W. Chen, "VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use", Trans. Mach. Learn. Res., 2026.
B. Jin, H. Zeng, Z. Yue, D. Wang, H. Zamani, J. Han, "Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning", in Proceedings of COLM 2025, 2025.
J. Feng, S. Huang, X. Qu, G. Zhang, Y. Qin, B. Zhong, C. Jiang, J. Chi, W. Zhong, "ReTool: Reinforcement Learning for Strategic Tool Use in LLMs", CoRR, vol. abs/2504.11536, 2025.
D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, J. Schmidhuber, "Natural Evolution Strategies", J. Mach. Learn. Res., vol. 15, no. 27, pp. 949--980, 2014.
W. Liu, S. Song, H. Hao, A. Zhou, "EA4LLM: A Gradient-Free Approach to Large Language Model Optimization via Evolutionary Algorithms", CoRR, vol. abs/2510.10603, 2025.
B. Sarkar, M. Fellows, J. A. Duque, A. Letcher, A. L. Villares, A. Sims, C. Wibault, D. Samsonov, D. Cope, J. L. Liesen, K. Li, L. Seier, T. Wolf, U. Berdica, V. Mohl, A. D. Goldie, A. Courville, K. Sevegnani, S. Whiteson, J. N. Foerster, "Evolution Strategies at the Hyperscale", in Proceedings of ICML 2026, 2026.
M. A. Potter, K. A. D. Jong, "Cooperative Coevolution: An Architecture for Evolving Coadapted Subcomponents", Evol. Comput., vol. 8, no. 1, pp. 1--29, 2000.
M. N. Omidvar, X. Li, Y. Mei, X. Yao, "Cooperative Co-Evolution With Differential Grouping for Large Scale Optimization", IEEE Trans. Evol. Comput., vol. 18, no. 3, pp. 378--393, 2014.
P. Yang, H. Zhang, Y. Yu, M. Li, K. Tang, "Evolutionary reinforcement learning via cooperative coevolutionary negatively correlated search", Swarm Evol. Comput., vol. 68, pp. 100974, 2022.
C. Hu, J. Liu, X. Yao, "Evolutionary Reinforcement Learning via Cooperative Coevolution", in Proceedings of ECAI 2024, pp. 3300--3307, 2024.
H. Qian, Y. Wu, R. Qin, X. An, Y. Chen, A. Zhou, "Provable Space Discretization Based Evolutionary Search for Scalable Multi-Objective Security Games", Swarm Evol. Comput., vol. 92, pp. 101770, 2025.
Q. Team, "Qwen3.5: Towards Native Multimodal Agents", 2026.
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, J. Steinhardt, "Measuring Mathematical Problem Solving With the MATH Dataset", in Proceedings of NeurIPS 2021, 2021.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, J. Schulman, "Training Verifiers to Solve Math Word Problems", CoRR, vol. abs/2110.14168, 2021.
H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, K. Cobbe, "Let's Verify Step by Step", in Proceedings of ICLR 2024, 2024.
Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, C. D. Manning, "HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering", in Proceedings of EMNLP 2018, pp. 2369--2380, 2018.
X. Ho, A. D. Nguyen, S. Sugawara, A. Aizawa, "Constructing a Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps", in Proceedings of COLING 2020, pp. 6609--6625, 2020.
H. Trivedi, N. Balasubramanian, T. Khot, A. Sabharwal, "MuSiQue: Multihop Questions via Single-hop Question Composition", Trans. Assoc. Comput. Linguist., vol. 10, pp. 539--554, 2022.
U. Devulapalli, A. Satsangi, A. Narayan, "Fine-Tuning Large Language Models for Structured Clinical Report Generation Using GRPO", Proceedings of the AAAI Symposium Series, vol. 7, no. 1, pp. 496--500, 2025.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, W. Zaremba, "Evaluating Large Language Models Trained on Code", CoRR, vol. abs/2107.03374, 2021.
T. Dao, D. Y. Fu, S. Ermon, A. Rudra, C. R\'e, "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness", in Proceedings of NeurIPS 2022, pp. 16344--16359, 2022.
A. Meurer, C. P. Smith, M. Paprocki, O. \'ik, S. B. Kirpichev, M. Rocklin, A. Kumar, S. Ivanov, J. K. Moore, S. Singh, T. Rathnayake, S. Vig, B. E. Granger, R. P. Muller, F. Bonazzi, H. Gupta, S. Vats, F. Johansson, F. Pedregosa, M. J. Curry, A. R. Terrel, \. Rou, A. Saboo, I. Fernando, S. Kulal, R. Cimrman, A. Scopatz, "SymPy: Symbolic Computing in Python", PeerJ Comput. Sci., vol. 3, pp. e103, 2017.
W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica, "Efficient Memory Management for Large Language Model Serving with PagedAttention", in Proceedings of SOSP 2023, pp. 611--626, 2023.
G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, C. Wu, "HybridFlow: A Flexible and Efficient RLHF Framework", in Proceedings of EuroSys 2025, pp. 1279--1297, 2025.

App. AAdditional Experimental Details

App. A.1Models and Datasets

Models.

We use Qwen3.5-4B [26] as the initial model for every post-training experiment. Qwen3.5-4B without post-training serves as the common reference, while Qwen3.5-9B without post-training is included only as a larger-model reference on the math task.

Math Data.

We construct the post-training data from the 7,500-example training split of MATH [27]. A subject-stratified random split holds out 750 examples for validation. After removing two unusable examples from the remaining data, 6,748 examples are used for post-training. The training and validation data strictly exclude MATH-Test. We evaluate on AIME 2024,^{https://huggingface.co/datasets/HuggingFaceH4/aime_2024} AIME 2025,^{https://huggingface.co/datasets/test-time-compute/aime_2025} GSM8K [28], MATH-500 [29], and MATH-Test. MATH-500 is a curated 500-example subset of the 5,000-example MATH-Test set rather than an independent test set.

QA Data.

For the QA task, we hold out 750 examples from the HotpotQA training split [30] for validation and remove them from the post-training data, leaving 89,697 training examples. Evaluation uses 5,000 randomly selected examples from the HotpotQA validation split, 5,000 randomly selected examples from the 2Wiki development split [31], and the complete 2,417-example MuSiQue development split [32]. Table 5 summarizes the data used in both tasks.

Task	Dataset	Source split	Role	Examples used
Math	MATH	Training	Post-training	6,748
	MATH	Training	Validation	750
	AIME 2024	Test	Evaluation	30
	AIME 2025	Test	Evaluation	30
	GSM8K	Test	Evaluation	1,319
	MATH-500	Test	Evaluation	500
	MATH-Test	Test	Evaluation	5,000
QA	HotpotQA	Training	Post-training	89,697
	HotpotQA	Training	Validation	750
	HotpotQA	Validation	Evaluation	5,000
	2Wiki	Development	Evaluation	5,000
	MuSiQue	Development	Evaluation	2,417

Table 5. Datasets, source splits, and numbers of examples used for post-training, validation, and evaluation.

App. A.2Agent Environment

The two tasks use the same tool-using agent equipped with Local Wiki Search over the Wiki18 corpus [16] and a Python Sandbox. The search tool returns evidence from the local corpus, while the sandbox supports commonly used Python packages for computation and answer verification. For each math problem, the agent may invoke Local Wiki Search at most three times and the Python Sandbox at most five times. The corresponding limits for QA are five search calls and five Python calls. The maximum context length is 128K. Invalid tool calls are retried; if a tool or context limit is reached, an available final answer is still extracted and verified, whereas an interaction without a valid answer is treated as unanswered.

App. A.3Training Rewards and Answer Verification

Answer Extraction.

We inspect assistant outputs in reverse order and first locate the most recent complete answer block, delimited by <answer> and </answer>. Within this block, we use the last boxed or framed value when present; otherwise, we use the entire block content. When no answer block is present, we extract the last boxed value from the full output. If neither form is found, the complete output is used as the fallback prediction. Correctness verification and format scoring are computed separately, so a fallback prediction can still be verified while incurring a format penalty.

Math Reward.

The extracted math answer is normalized with regular-expression-based cleaning and then checked for mathematical equivalence using SymPy [36]. Let \(a_{\mathrm{math}}\in\{0,1\}\) denote the resulting correctness indicator. The training reward is

\[R_{\mathrm{math}}=a_{\mathrm{math}}+r_t+r_f.\](14)

The tool-use bonus is \(r_t=0.1\) when \(a_{\mathrm{math}}=1\) and at least one tool was used, and \(r_t=0\) otherwise.

QA Reward.

The QA base score is the token-overlap F1 defined in Section A.4. Its training reward is

\[R_{\mathrm{QA}}=\mathrm{F1}+r_t+r_f,\](15)

where \(r_t=0.1\) if \(\mathrm{F1}>0\) and at least one tool was used, and \(r_t=0\) otherwise.

Format Reward.

Both tasks use the same format reward. We set \(r_f=0\) when the output contains both an <answer> block and a boxed answer, \(r_f=-0.5\) when either is missing, and \(r_f=-1\) when both are missing.

App. A.4Evaluation Protocol

Math Evaluation.

Validation accuracy is computed on the 750 held-out problems using one generation per problem with temperature \(0\). Test-time decoding uses temperature \(0.3\) and top- \(p=1.0\). For GSM8K, MATH-500, and MATH-Test, we generate 32 outputs per problem and report pass@ \(k\) for \(k=1,\ldots,16\). For AIME 2024 and AIME 2025, we generate 256 outputs per problem and report \(k=1,\ldots,128\).

QA Evaluation.

QA validation uses one generation per problem with temperature \(0\), and the validation metric is mean F1 over the 750 held-out problems. Test-time decoding uses temperature \(0.3\), top- \(p=1.0\), and 32 outputs per problem. We normalize predictions and references by lowercasing, removing English articles, stripping and collapsing whitespace, and applying Unicode case and diacritic normalization; punctuation is retained. Accuracy is one when a normalized ground-truth answer is a contiguous substring of the normalized prediction, exact match (EM) requires equality, and F1 is computed from token overlap. With multiple ground-truth answers, each metric takes its maximum over the references. Each reported Accuracy, EM, or F1 value is first averaged over the 32 outputs for a problem and then averaged across problems.

Pass@ \(k\) and Repeated Sampling.

For a problem with \(n\) generated outputs, of which \(c\) are correct, we use the unbiased estimator [34]

\[\operatorname{pass@}k =1-\frac{\binom{n-c}{k}}{\binom{n}{k}}.\](16)

Math correctness is determined by the verifier described above. The additional QA pass@ \(k\) results in Section E use the binary Accuracy criterion.

App. A.5Training Hyperparameters

Table 6 lists the method-specific settings. Following prior work, we use recommended learning-rate settings for the GRPO variants [7],[33] and established settings for ES-based LLM post-training [12],[19]. We determine \(K=4\) before benchmark test evaluation and use the same method-specific hyperparameters on the math and QA tasks. Standard ES and CoPES use greedy generation with temperature \(0\) during perturbed-model evaluation. All three post-training methods in the QA experiment run for 48 steps. Each reported training configuration is run once. Dataset splitting and evaluation-set subsampling use seed 1088, while the ES-based and GRPO-based methods use training seeds 33 and 42, respectively.

Setting	Value
Prompt batch size	64
Full-parameter GRPO learning rate	\(1\times 10^{-6}\)
LoRA-based GRPO learning rate	\(5\times 10^{-6}\)
GRPO group / mini-batch size	\(8\) / \(32\)
GRPO KL coefficient	\(1\times 10^{-3}\)
GRPO PPO ratio clipping range	\([0.8,1.2]\)
GRPO group-sampling temperature	\(1.0\)
LoRA rank / scaling factor	\(32\) / \(64\)
ES/CoPES population \(N\)	40
ES/CoPES \(\sigma\) / \(\alpha\)	\(1\times 10^{-3}\) / \(5\times 10^{-4}\)
CoPES \(K\) / \(N_{k}\)	\(4\) / \(10\)
CoPES \(\sigma_{k}\)	\(2\times 10^{-3}\)

Table 6. Hyperparameters used in the experiments.

App. A.6Frameworks and Compute Environment

The servers use Intel Xeon Gold 6338 or AMD EPYC 7713 CPUs and run Ubuntu 22.04 with Python 3.11 and CUDA 13. Standard ES and CoPES use vLLM 0.20.2 [37], while both GRPO variants use VeRL 0.8.0 [38].

App. A.7Hardware-Feasibility Configuration

For the hardware-feasibility tests reported in the main paper, we enable all applicable memory optimizations supported by each framework. Full-parameter GRPO uses gradient checkpointing together with parameter, optimizer, and activation offloading. LoRA-based GRPO uses gradient checkpointing and parameter and optimizer offloading; activation offloading is unavailable for this configuration. Fused kernels are unavailable for Qwen3.5-4B in the evaluated VeRL configuration. Standard ES and CoPES use the standard vLLM inference path with KV caching, without additional framework-level GPU-memory optimizations. These settings are enabled in the hardware-feasibility tests and are not algorithmic contributions of this work.

App. A.8Memory-Efficient ES Implementation

Seed Replay and Chunking.

Following OpenAI-ES [10], each perturbation is represented by a random seed and regenerated during the update. CoPES replays each perturbation in parameter chunks, avoiding storage of full perturbation vectors on either GPU or CPU.

CPU Weight Backup.

Rather than restoring the unperturbed model by subtracting a replayed perturbation, CoPES backs up the pre-perturbation weights in CPU memory and directly restores them after each perturbed-model evaluation. This design avoids residual numerical errors caused by the non-reversibility of floating-point addition and subtraction and ensures that every perturbation is evaluated from the same pre-update model.

Partition Replay.

CoPES also stores the random seed used to construct each parameter partition. During the update, it replays the chunk-to-subspace assignment from this seed instead of retaining a parameter-level assignment mask. Seed replay, chunked processing, CPU weight backup, and partition replay support memory-efficient execution; they do not alter the algorithmic update described in the main paper.

Accounted component	Full-parameter GRPO	LoRA-based GRPO	Standard ES / CoPES
GPU-resident model and training states	69.44 GB	18.55 GB	8.68 GB
Parameter gradients	Included	LoRA parameters only	None
Optimizer states	Included	LoRA parameters only	None
Backpropagation activations	Included	Included	None
Logits and entropy-related tensors	Included	Included	Not retained for backward
Context-dependent term	\(3L\) GB	\(3L\) GB	\(0.032L\) GB
Total	\(69.44+3L\) GB	\(18.55+3L\) GB	\(8.68+0.032L\) GB

Table 7. Components included in the theoretical GPU memory accounting. \(L\) denotes context length in K tokens.

App. A.9GPU-Hour Profiling

GPU-hour profiling measures training-step time only. We profile CoPES and full-parameter GRPO for 32 steps on the same \(8\times\) RTX 5880 Ada server as representatives of the ES-based and RL-based method classes, respectively, and extrapolate their average per-step costs to each method's reported training steps. A profiled step includes perturbed-model or policy evaluation, tool execution, reward computation, and the parameter update; time spent on validation is excluded. This class-level extrapolation reflects that perturbed-model or policy evaluation dominates per-step time, while partition replay and parameter updates contribute negligibly to total GPU-hours in our measurements.

App. BTheoretical GPU Memory Accounting

The memory analysis in the main paper estimates algorithm-level GPU memory requirements for Qwen3.5-4B. It accounts for model weights, parameter gradients, optimizer states, backpropagation activations, logits and entropy-related tensors, and inference KV caches. FlashAttention [35] and KV caching are assumed. To isolate the memory implied by each optimization paradigm, the analysis excludes gradient checkpointing, offloading, fused kernels, and other framework-specific memory optimizations. The GRPO estimates cover actor training only and omit a separately resident reference policy and generation engine, making them conservative with respect to the RL-based methods. Standard ES and CoPES have the same theoretical GPU memory requirements.

Fixed States.

Qwen3.5-4B contains \(P=4{,}659{,}865{,}088\) parameters. Full-parameter GRPO retains FP32 model weights, parameter gradients, and two FP32 AdamW states, giving

\[\frac{(4+4+8)P}{2^{30}}=69.44\ \mathrm{GB}.\](17)

For LoRA-based GRPO with rank 32, the frozen FP32 base model occupies 17.36 GB. The \(80{,}150{,}528\) LoRA parameters and their gradients and optimizer states add 1.19 GB, for a fixed total of 18.55 GB. ES-based methods retain only the BF16 model weights on GPU:

\[\frac{2P}{2^{30}}=8.68\ \mathrm{GB}.\](18)

The pre-perturbation backup used by CoPES resides in CPU memory and therefore does not add another GPU-resident model copy.

Context-Dependent Memory.

For both GRPO variants, the dominant context-dependent terms are the full vocabulary logits, entropy/softmax tensors, and activations retained for backpropagation. Their combined lower-bound estimate is approximately 3 GB per K tokens. LoRA reduces trainable parameter states but does not remove the need to backpropagate through the model, so it has essentially the same context-dependent slope. ES and CoPES perform forward-only generation and retain neither parameter gradients nor backpropagation activations. Qwen3.5-4B has eight full-attention layers, four KV heads, and a head dimension of 256, yielding 32 KiB of BF16 KV cache per token, or approximately 0.032 GB per K tokens.

**Figure 4.** Validation accuracy during math post-training. Stars mark the best observed validation checkpoints for full-parameter GRPO, LoRA-based GRPO, and standard ES. The horizontal dashed line marks CoPES validation accuracy at its 16-step fixed-budget endpoint rather than a CoPES training curve.

Conservative Rounding.

The GRPO context-dependent coefficient is rounded near the lower end of its component-wise estimate, whereas the exact ES KV-cache coefficient of 0.03125 GB per K tokens is rounded upward to 0.032. These choices underestimate rather than exaggerate the relative memory advantage of ES. The values are theoretical accounting estimates rather than measured peak GPU memory and intentionally omit runtime-specific overheads.

Pass@k for K\in\{1,2,4,8\}, where K is the number of parameter subspaces. The total population remains N=40, with N_k=N/K and \sigma_k=\sqrt{K}\sigma. — **Figure 5.** Pass@ \(k\) for \(K\in\{1,2,4,8\}\), where \(K\) is the number of parameter subspaces. The total population remains \(N=40\), with \(N_k=N/K\) and \(\sigma_k=\sqrt{K}\sigma\).

Pass@k with joint reward standardization and independent reward standardization. Both variants use K=4 and otherwise identical settings. — **Figure 6.** Pass@ \(k\) with joint reward standardization and independent reward standardization. Both variants use \(K=4\) and otherwise identical settings.

Pass@k for standard ES with \sigma=10^{-3}, standard ES with \sigma=2\times10^{-3}, and CoPES. The second setting matches the per-coordinate perturbation scale used by CoPES. — **Figure 7.** Pass@ \(k\) for standard ES with \(\sigma=10^{-3}\), standard ES with \(\sigma=2\times10^{-3}\), and CoPES. The second setting matches the per-coordinate perturbation scale used by CoPES.

Final Estimates.

The resulting theoretical requirements are

\[\begin{aligned}M_{\mathrm{full}}(L) &= 69.44+3L\ \mathrm{GB}, \\ M_{\mathrm{LoRA}}(L) &= 18.55+3L\ \mathrm{GB}, \\ M_{\mathrm{ES/CoPES}}(L) &= 8.68+0.032L\ \mathrm{GB}.\end{aligned}\](19)

At a context length of 128K, these expressions give 453.44 GB for full-parameter GRPO, 402.55 GB for LoRA-based GRPO, and 12.78 GB for standard ES or CoPES. Thus, even the LoRA-based estimate is more than \(30\times\) the ES estimate, while full-parameter GRPO requires approximately \(35\times\) as much memory.

App. CValidation Curves and Checkpoint Selection

For methods whose training trajectories extend beyond their highest observed validation performance, we report the corresponding best observed checkpoint. For methods run only to a common fixed budget, we report the budget-end checkpoint without characterizing their subsequent training behavior.

Math Validation.

Figure 4 shows the complete validation trajectories used to identify the best observed validation checkpoints reported in the main paper: step 16 for full-parameter GRPO, step 64 for LoRA-based GRPO, and step 112 for standard ES. CoPES is evaluated only within the common 16-step budget, so its horizontal dashed line represents the validation accuracy reached at that endpoint. Standard ES first exceeds this reference level at step 48, requiring three times as many training steps as the CoPES fixed-budget run.

**Figure 8.** Validation F1 during QA post-training. The horizontal dashed line is CoPES validation F1 at its 48-step fixed-budget endpoint rather than a CoPES training curve.

QA Validation.

Figure 8 reports mean validation F1. Full-parameter GRPO reaches its highest observed validation F1 at step 48, which determines the common 48-step QA budget used in the main comparison. The CoPES dashed line marks its validation F1 at this fixed-budget endpoint; its behavior beyond 48 steps was not evaluated. Standard ES reaches the CoPES reference level at step 64; its later checkpoints are shown only to characterize the validation trajectory and are not used in the 48-step main comparison.

Post-Peak Behavior.

The two GRPO curves on math and the full-parameter GRPO curve on QA decline after reaching their highest observed validation performance. We observed repetitive reasoning loops in some later GRPO outputs, which may contribute to this behavior, but this explanation remains a hypothesis rather than an established causal result.

App. DComplete Ablation Results

All ablations use a 16-step fixed budget and a total population of \(N=40\). In the subspace-count ablation, \(N_k=N/K\) and \(\sigma_k=\sqrt{K}\sigma\); the other ablations retain the main CoPES settings except for the factor being evaluated.

Number of Subspaces.

Figure 5 extends the pass@1 ablation in the main paper to the complete pass@ \(k\) curves. Performance generally improves as \(K\) increases from 1 to 4, while \(K=8\) performs substantially worse. At \(K=8\), each subspace receives only \(N_k=5\) perturbations, and the model is divided into more parameter groups. The smaller population per subspace and weaker preservation of parameter interactions may both reduce update quality.

Joint Reward Standardization.

Figure 6 compares the proposed joint standardization with independently standardizing the rewards within each subspace. Joint standardization gives higher pass@1 on all five benchmarks. On GSM8K and MATH-500, the curves approach saturation and cross slightly at larger \(k\); these small pointwise differences do not establish a consistent high- \(k\) advantage for either normalization mode. Overall, the results support joint standardization at the primary pass@1 operating point without claiming uniform dominance at every \(k\).

Perturbation-Scale Control.

CoPES uses \(\sigma_k=2\sigma\) when \(K=4\). Figure 7 therefore compares CoPES with standard ES at both \(\sigma\) and \(2\sigma\) under the same fixed budget. Increasing the standard-ES perturbation scale degrades its performance across the benchmarks rather than reproducing the CoPES improvement. The benefit of CoPES therefore cannot be attributed solely to its larger per-coordinate perturbation scale.

App. EAdditional Results on the QA Task

The main paper reports Accuracy, EM, and F1 averaged over the 32 evaluation outputs per problem. Here, we additionally characterize repeated-sampling behavior with Accuracy-based pass@ \(k\). A generated output is counted as correct when its binary Accuracy equals one, and Equation 16 is applied to the same 32 outputs used to compute the main QA table.

Accuracy-based Pass@k on 2Wiki, HotpotQA, and MuSiQue. Qwen3.5-4B is the reference model without post-training; the other methods use the common 48-step QA budget. — **Figure 9.** Accuracy-based Pass@ \(k\) on 2Wiki, HotpotQA, and MuSiQue. Qwen3.5-4B is the reference model without post-training; the other methods use the common 48-step QA budget.

Figure 9 reports \(k=1,\ldots,16\). By construction, pass@1 exactly matches the Accuracy values in the main QA table. CoPES exceeds standard ES across the three curves, extending its advantage to repeated sampling on the QA task. Compared with full-parameter GRPO, CoPES starts slightly lower at pass@1 but achieves higher pass@ \(k\) for \(k\geq2\) on 2Wiki and MuSiQue. The two methods remain close on HotpotQA, where CoPES also becomes slightly higher as \(k\) increases. These curves supplement rather than replace the Accuracy, EM, and F1 comparisons in the main paper.