The Shifting Sands of AI Development
Recent developments suggest that the era of simply scaling up large language models (LLMs) may be nearing its end. Innovators are exploring alternative strategies to achieve greater gains in artificial intelligence, moving beyond the challenges of traditional scaling laws.
A New Paradigm in Model Architecture
As the limitations of increasing model size become apparent, researchers are seeking new architectural approaches. Concerns about the diminishing returns of scaling laws, which have been a subject of discussion since late 2023, have prompted a search for models that can boost productivity and performance. Microsoft’s Satya Nadella and Sam Altman, among others, have hinted at a shift in the AI landscape, with Altman even suggesting that the age of oversized models might be waning.
Companies at the Forefront
Several companies are pioneering alternative methods. One such company, Symbolica AI, aims to address the scaling dilemma by creating models based on symbolic representations. Symbolica’s founder, George Morgan, advocates for a move away from simply building larger models toward finding new ways to enhance AI systems.
“The overwhelming thesis that everyone’s taking right now is that we need to build bigger models,” Morgan said in a January interview with Forbes’ Randall Lane at IIA’s Davos event.
Morgan noted the uniformity across major AI players like OpenAI, Anthropic, and Google, all of whom employ similar underlying technologies. This homogeneity, he argues, leads to remarkably similar results, with models plateauing and converging. This is not ideal for consumers or businesses, who constantly seek greater AI capabilities. To achieve these capabilities more affordably and efficiently, a distinct architectural approach is necessary. This new architecture must provide better scaling, be cheaper, and deliver higher performance.
Symbolica, backed by figures like Vinod Khosla and with $33 million in financial runway, is actively researching this new architecture. The company is relocating to Europe to tap into the talent pool in mathematical disciplines.
Beyond the Matrix: A Symbolic Approach
The name ‘Symbolica’ reflects the company’s strategy of incorporating symbols into AI models. The firm’s approach contrasts with the use of large matrix calculations on GPUs, bringing symbols back into the equation in a manner reminiscent of earlier AI approaches that predate neural networks. This is a departure from the current trend of bigger, more complex models.
Accessibility: A New Focus
In a discussion with Forbes, Morgan highlighted how the increasing costs of developing and maintaining models run counter to the historical trend of technology becoming cheaper and more accessible. Current models, which require billions of dollars to train and maintain, are limiting innovation because only a few players can afford to create state-of-the-art systems. Morgan stressed that a reduction in model size could spark more innovation, possibly through smaller models on edge devices.
Beyond the Monolith: Collaborative AI
Beyond Symbolica AI, new strategies are emerging from the realm of Liquid Networks. These networks aim to enhance smaller models by refining how LLMs are structured. In addition to that, the emergence of agentic AI is bringing about collaboration of models, where smaller models can work together to create a complex system that manages multiple tasks.
This collaboration of models suggests a shift away from monolithic AI systems. By using numerous, smaller models working in tandem, the collaborative result can create a powerful, versatile digital intelligence. This concept mirrors Marvin Minsky’s ‘Society of the Mind,’ which describes the human brain as an ensemble of interacting components that contribute to diverse functions.
This new perspective suggests that the best AI potential might lie in systems of smaller components. With each small model playing its role the way it is designed, the results can reach new heights. It also reinforces the idea that collaboration is key for both AI advancement and human progress.
