Why scaling LLMs might work

The theory of evolution is one of humanity’s greatest achievements, helping us understand the past, present, and future. Although evolution typically operates over tens of thousands of years, for the first time, evolutionary thinking might offer insights into what could happen in the coming decades. But before diving into this idea, let’s recall one of the key ingredients necessary for evolution to occur:

Incremental changes with selective advantage

An eye does not evolve in a single generation, nor does a wing appear overnight. Every adaptation requires a series of small, cumulative changes, where each step provides a competitive advantage over the previous one.

You may already see where this is going. If we see the human brain through an evolutionary lens, we can agree that the journey from a single neuron to the complexity of a human brain involved countless small, incremental improvements. Each step built upon the last, offering some advantage over previous generations.

Here’s the first hint, if we allow ourselves to consider the cortical column as a fundamental unit, we can observe how nature scaled up the size of the network by increasing cortical folding over time.

For the second hint, evolution also suggests that larger models require more data for training. For example, in cognitive development, larger brains tend to require longer periods of learning and adaptation. A human baby, for instance, takes months to walk, whereas some species are capable of moving independently almost immediately after birth.

And the final hint come from selective pressure. While biological evolution is driven by survival pressures, the evolution of AI is shaped by economic forces. The selective advantage here isn’t survival in a natural environment but rather market viability. Models that perform better receive more funding, compute, and training, while less effective ones are abandoned. In this sense, LLMs are evolving under artificial selection, where money acts as the primary selective pressure.

So, to conclude, it is possible that in the next couple decades we might just need bigger models, more data and probably a couple tricks to go beyond human intelligence. Of course, these analogies between LLMs and biological brains are not exact, and we shouldn’t take them too literally. However, when pushing the boundaries of human knowledge, small hints and intuition are often all it takes to explore uncharted territory.

Comments URL: https://news.ycombinator.com/item?id=43078705

Points: 1

# Comments: 0