Liquid AI Challenges Transformer Architecture with Hyena Edge
Liquid AI, a Boston-based AI startup spun out of MIT, is seeking to revolutionize the AI industry by moving beyond the Transformer architecture that underpins most popular large language models. The company has announced “Hyena Edge,” a novel convolution-based, multi-hybrid model designed specifically for smartphones and other edge devices.
A New Architecture for Edge AI
Hyena Edge diverges from traditional attention-heavy designs used in most small models for mobile deployment. Instead, it replaces two-thirds of grouped-query attention (GQA) operators with gated convolutions from the Hyena-Y family. This architecture is the result of Liquid AI’s Synthesis of Tailored Architectures (STAR) framework, which uses evolutionary algorithms to automatically design model backbones.
Outperforming Transformers
In real-world tests on a Samsung Galaxy S24 Ultra smartphone, Hyena Edge outperformed a parameter-matched Transformer++ model in several key areas:
- Up to 30% faster prefill and decode latencies
- Smaller memory footprint
- Better benchmark results
Hyena Edge was trained on 100 billion tokens and evaluated across standard benchmarks for small language models. It either matched or exceeded the performance of the GQA-Transformer++ model in all benchmarks, with notable improvements in perplexity scores and accuracy rates.
Insights into Hyena Edge’s Development
A recent video walkthrough provides a detailed look at Hyena Edge’s evolution, showcasing how key performance metrics improved over successive generations of architecture refinement. The video highlights dynamic changes in the distribution of operator types, offering valuable insights into the model’s architectural design principles.
Open-Source Plans and Future Vision
Liquid AI plans to open-source Hyena Edge and other Liquid foundation models in the coming months. The company’s goal is to develop capable and efficient general-purpose AI systems that can scale from cloud datacenters to personal edge devices, potentially setting a new baseline for edge-optimized AI.
The success of Hyena Edge positions Liquid AI as a significant player in the evolving AI landscape, particularly in the development of alternative architectures to challenge Transformers in practical settings.
