CoGR: Co-Evolving Generative Retrieval

Two LLMs, one for queries and one for items, learn to write matching keyword representations. After SFT they are trained with alternating GRPO updates against a single shared retrieval objective, with a counterfactual marginal reward for item-level credit assignment. 10.9–36.1% relative F1 gains over the strongest baseline.

2026 · Apple arXiv paper PDF LLM post-training · GRPO · verl · Qwen3-4B · retrieval

The problem. Dense retrievers compare a query embedding with an item embedding. That works, but the representation is opaque and the two towers are trained once and frozen relative to each other. We wanted retrieval that an LLM can reason about, and that keeps improving on both sides.

+10.9–36.1%
relative F1 over the strongest baseline
SFT → GRPO
alternating query-side and item-side updates
counterfactual
marginal reward for item-level credit

The idea. CoGR trains an LLM to construct a compact keyword representation for a query, and another to construct one for an item. Supervised fine-tuning first establishes a shared keyword space. Then alternating GRPO updates optimize the query-side and item-side generators against the same retrieval objective. The query side is rewarded directly by retrieval F1. The item side gets a counterfactual marginal reward: how much does the query-side retrieval F1 change when this item’s keywords are swapped in, holding everything else fixed? That gives each item its own credit signal instead of a shared, noisy one. Each side is trained against the current state of the other, so the two representations evolve toward each other instead of apart.

Query generator (LLM) query → keywords Item generator (LLM) item → keywords Shared keyword space Retrieval reward does q match the right item? (one objective, both sides) GRPO on retrieval F1, odd rounds GRPO on counterfactual marginal reward, even rounds
The two generators share one reward and are updated in alternation, so each is always optimized against the other's current behavior.

What I did. Co-developed the framework and mentored the intern who led the project.

Why I like it. The reward is the real retrieval task, not a proxy for it. And because the two sides are trained in alternation, each one keeps adapting to what the other has just learned. That makes the setup powerful, and also one you have to watch closely: the evaluation has to tell you whether the two sides are converging on a shared language or collapsing into a degenerate one.