Can an AI agent get better on its first attempt while becoming less likely to solve a task across many tries?
A Meta paper finds that reinforcement learning (RL) post-training improved agents’ first-attempt performance, but reduced their likelihood of finding a solution across 128 attempts. RL is training that uses rewards to guide behaviour.
Two different measures
First-attempt performance asks whether an agent gets the task right immediately. Success across 128 attempts asks whether it finds a solution at any point within that allowance. An improvement in one can coexist with a decline in the other.
The paper’s interpretation: RL mostly sharpens existing skills rather than teaching new ones. That is an interpretation of the reported result, rather than proof that RL cannot teach new skills.
The practical implication: evaluate agents against the retry allowance of the intended workflow. For a process that permits repeated attempts, a stronger first-try score alone does not establish a better chance of eventual success.