A tool for writing and debugging human-written code: Meta, CodeLlama, CodeWhisperer and AlphaCode

Meta has released a tool called Code Llama, built on top of its Llama 2 large language model, to generate new code and debug human-written work, the company said.

Programmers are using LLMs to assist in a lot of tasks, including writing new software and debugging existing code, according to Meta. The goal is to make developer workflows more efficient so they can concentrate on the most human-centered aspects of their jobs.

Code generators have been helping developers work for a while now. Copilot is powered by GPT-4 and can be used to quickly write and check code. New code can be written by the Copilot. CodeWhisperer is also a part of Amazon’s website. And yes, Google also has a code-writing tool in AlphaCode, but that isn’t out yet.

Copilot is a tool that can reproduce licensed code and is being sued by Microsoft and Openai because of it.

What Code Llama 2 is about: Introducing the Copilot plug-in to improve security in AI coding and using open source software to build applications

“It’s exciting that they’re releasing the weights to the community,” says Deepak Kumar, a postdoctoral researcher at Stanford who has studied AI coding, referring to the parameters of the neural network at the core of the model.

Kumar says the release of Meta’s regular language model Llama 2 led to the formation of communities dedicated to discussing how it behaves and how it can be modified. It gives us a bit more freedom to play with what’s happening under the hood, compared to the closed-sourced models we’ve seen before.

Code Llama will be used to build new kinds of applications. For example, it could be possible to create a programming assistant that performs various additional safety checks before recommending a chunk of code, says Kumar, whose own research has explored how AI assistance can sometimes lead to less secure code. Kumar says the release could inspire the creation of assistants for specific types of coding. He says you can build a number of different things on top of the model.

The Copilot plug-in for coding programs was launched by Microsoft in May 2021, and works with the first line of a user’s comment. Copilot uses a version of Open AI’s GPT, the large language model behind ChatGPT. There are reports that the model is trained further using the code that GitHub stores for developers and that contractors are paid to look at their own code.

Meta is likely to limit the training data to avoid similar problems, since it faces a lawsuit for using open source code in its training data. Copilot costs $10 per month for individuals and $19 per month, per user, for businesses.

Open Chatbots: Why Open Models Need To Be Open? How Free Can You Get? The Case Of Meta Llama 2

The inner workings of a world-famous chatbot remain shrouded in mystery despite making it possible for anyone to play with it.

Efforts to make artificial intelligence more open seem to have gained steam in recent months. In May, someone leaked a model from Meta, called Llama, which gave outsiders access to its underlying code as well as the “weights” that determine how it behaves. Then, this July, Meta chose to make an even more powerful model, called Llama 2, available for anyone to download, modify, and reuse. Meta’s models have since become an extremely popular foundation for many companies, researchers, and hobbyists building tools and applications with ChatGPT-like capabilities.

Llama 2 is free to download, modify, and deploy, but it is not covered by a conventional open source license. Meta does not allow the use of llama 2 to train other language models, and it requires a special license for use in an app or service with more than 700 million daily users.

This level of control means that Llama 2 may provide significant technical and strategic benefits to Meta—for example, by allowing the company to benefit from useful tweaks made by outside developers when it uses the model in its own apps.

The researchers say models that are released under normal open source licenses are more fully open. But it is difficult for such projects to get on an equal footing.

The data required to train advanced models is often kept a secret. Second, software frameworks required to build such models are often controlled by large corporations. The two most popular ones, TensorFlow and Pytorch, are maintained by Google and Meta, respectively. Third, computer power required to train a large model is also beyond the reach of any normal developer or company, typically requiring tens or hundreds of millions of dollars for a single training run. The human labor that is required to improve the models is available only to companies with deep pockets.