Will big language models replace data scientists? | by David Andres | October 2023

There has been a lot of hype lately about large computer programs that can understand and generate human language. These programs are called large language models, LLM for short. Some popular examples are GPT-4, Llama 2 and PaLM 2. They have sparked curiosity about their potential impact on employment and society as a whole.

Disclaimer: Some parts of this article have been enhanced with the help of AI to improve clarity.

The original purpose of these LLMs was to understand the text and be able to extract some information such as sentiment, semantic similarities, summarization‚Ķ However, the researchers found that when these models were trained with large amounts of data, they could also learn about the information. contained in it, or even learn to program or translate into another language! This discovery was a turning point. Companies have begun to train these models using domain texts such as health or medical text to develop a physician assistant; text related to the right to create legal counsel; code in multiple repositories to create a programmer’s assistant… So now these LLMs can do a lot of things beyond just understanding and processing text. They can chat with you, help lawyers, doctors and even assist programmers in writing code.

From general LLM to specific application. Source: Renaissance Rachel

LLM able to code

Some more advanced tools like Copilot and AlphaCode help coders by suggesting sections of code. While the latter have demonstrated their prowess in automating aspects of code development, they have yet to master building complex software or programs from scratch.

However, tools like AutoGPT and LangChain bridge these gaps. It addresses some of the primary limitations of the LLM, including mathematical calculations, Internet connectivity, and performing complex and long-term tasks. These tools skillfully divide challenges into a sequence of sub-tasks and pave the way to achieving a set goal. They are not yet perfect and many times they do not achieve satisfactory results. One…

Leave a Comment