We need more science in the judgment of AI

The author argues that LLM capabilities are too often judged from subjectively impressive one-off results rather than pre-registered hypotheses and evaluation criteria, a practice akin to HARKing and p-hacking that is fine for hypothesis generation but not for scientific testing. He notes that impressive feats like one-shot coding often reflect abundant training data for a domain, while areas with sparse data such as Zig, Nextflow and much of biology perform poorly, and that a researcher involved in rewriting bioinformatics software in Rust now uses LLMs to build a deterministic translation tool rather than translating code directly, because too many corner cases are missed.