How useful is an AI research assistant if it can suggest experiments but struggles to pick the strongest ones?
A study tested all 415 experiments that six frontier models proposed for seven natural language processing (NLP) papers—the field concerned with how computers work with human language.
The study found that only 73 experiments were sound, new and non-obvious. The models’ own rankings and team setups added little.
The distinction matters: proposing an experiment and judging whether it deserves time and resources are separate tasks. This result raises a practical question about the quality of that judgment, beyond the ability to produce ideas.
The evaluation criteria aren’t detailed in the available evidence.
For practitioners, the useful implication is to assess idea generation and idea selection separately. Before delegating research priorities, check whether a model can reliably identify proposals that are sound, new and non-obvious—not merely produce a long list.