Enhancing AI-Generated Code Accuracy
Researchers from MIT, McGill University, ETH Zurich, Johns Hopkins University, Yale, and the Mila-Quebec Artificial Intelligence Institute have developed a novel method to improve the accuracy and usefulness of AI-generated code. This method, which spans various programming languages, instructs large language models (LLMs) to adhere to the rules of each language, thereby enhancing code quality.
The research team found that by adapting new sampling methods, specifically Sequential Monte Carlo (SMC), AI models can be guided to follow programming language rules more effectively. This approach not only improves the performance of small language models (SLMs), typically used for code generation, but also surpasses that of larger language models. SMC is a family of algorithms that help solve filtering problems.
According to João Loula, co-lead writer of the paper, this method “could improve programming assistants, AI-powered data analysis, and scientific discovery tools.” It also has the potential to cut compute costs and be more efficient than reranking methods. The researchers noted that while AI-generated code can be powerful, it often disregards the semantic rules of programming languages. Their method addresses this by discarding code outputs that may not work early in the process and allocating efforts towards outputs that are more likely to be valid and accurate.
Adapting SMC to Code Generation
The researchers developed an architecture that brings SMC to code generation “under diverse syntactic and semantic constraints.” Key features of this adaptation include proposal distribution guided by cheap constraints, important weights that correct for biases, and resampling that reallocates compute effort towards partial generations.
Model Testing
To validate their theory, the researchers conducted experiments using SMC to engineer more accurate code. These experiments included Python Code Generation on Data Science tasks, Text-to-SQL Generation, Goal Inference in Planning Tasks, and Molecular Synthesis for drug discovery. The results showed that using SMC improved small language models, enhanced accuracy and robustness, and outperformed larger models.
The development of this method is significant as AI models have made engineers work faster and more efficiently, giving rise to new kinds of software engineers. However, concerns over code quality, lack of support for complex coding, and compute costs for simple code generation have been addressed through this innovation. Other companies, such as Together AI and Google, have also explored ways to improve AI-generated code, including releasing DeepCoder-14B and enhancing Code Assist features.
This advancement in AI-generated code accuracy has the potential to make AI-powered coding more useful and enable engineers to trust the code generated by models more. As the field continues to evolve, such methods will be crucial in shaping the future of software development.
