Nobel Prize-winning physicist P.W. Anderson’s “More Is Different” argues that as the complexity of a system increases, new properties may materialize that cannot (easily or at all) be predicted, even from a precise quantitative under-standing of the system’s microscopic details. As Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo from Stanford University explain in a recently published paper "Emergence has recently gained significant attention in machine learning due to observations that large language models, e.g., GPT, PaLM, LaMDA can exhibit so-called “emergent abilities” across diverse tasks." It has been argued that large language models display emergent abilities not present in smaller-scale models, justifying the huge financial and environmental cost of developing these models.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo "present an alternative explanation for emergent abilities: that for a particular task and model family, when analyzing fixed model outputs, one can choose a metric which leads to the inference of an emergent ability or another metric which does not. Thus, our alternative suggests that existing claims of emergent abilities are creations of the researcher’s analyses, not fundamental changes in model behavior on specific tasks with scale."
Their paper, Are Emergent Abilities of Large Language Models a Mirage?, is quite technical but very well written and important for understanding the debate around AI.
