When neural networks are in operation, even the most expert researchers often have no idea what’s going on. This is not a discussion about biology. Rather, it concerns artificial intelligence algorithms, especially those based on deep learning that mimic connections between neurons. These systems operate as black boxes that remain indecipherable to data scientists, the brightest minds in academia, and even recent Nobel Prize-winning engineers at OpenAI and Google.
While the mathematics behind these algorithms is well understood, the behavior produced by the network is not. “We know what data was input into the model and what its output is – the result or prediction – but we cannot clearly explain how this output was obtained.” Verónica Bolón Canedo, an AI researcher at the Information and Communication Technology Research Center, explains. from the University of Coruña, Spain.
This is true for image generators like ChatGPT, Google Gemini, Claude (modeled by startup Anthropic), Llama (from Meta), and DALL-E. It also applies to any system that relies on neural networks, such as facial recognition applications and content recommendation engines.
In contrast, other artificial intelligence algorithms such as decision trees and linear regression, commonly used in fields such as medicine and economics, are easier to interpret. “Their decision-making process is easy to understand and visualize. You can follow the branches of the tree and see exactly how they arrived at a particular outcome,” Boron explains.
This clarity is critical as it brings transparency to the process and provides assurance to the users of the algorithm. In particular, the EU AI law emphasizes the importance of having transparent and accountable systems. Unfortunately, the very architecture of neural networks prevents this transparency. To understand the black box of these algorithms, we need to visualize networks of interconnected neurons or nodes.
“When you enter data into the network, it starts a series of calculations using the values present at the nodes,” explains Juan Antonio, research professor at the Institute of Artificial Intelligence at the Spanish National Research Center (CSIC). Information enters the first node, diffuses, passes as a number to subsequent nodes, and is then relayed to the next node. “Each node calculates a number and sends it to all connections, taking into account the weight (number) of each connection. A new node that receives this information calculates another number,” the researchers added. I did.
It is important to note that current deep learning models consist of thousands or even millions of parameters. These parameters represent the number of nodes and connections in the network after training, and indicate any values that can affect the results of your queries. “Deep neural networks multiply and combine large numbers of elements. You have to imagine this in terms of millions of elements, and it’s impossible to derive meaningful equations from them,” Boron said. says. Volatility is very high.
Some industry sources estimate that GPT-4 has nearly 1.8 trillion parameters. According to this analysis, each language model uses approximately 220 billion parameters. This means that there are 220,000,000,000 variables that can influence the algorithm’s response each time a question is posed.
Efforts to detect bias and other issues
The opacity of these systems makes bias correction increasingly difficult. This lack of transparency fosters mistrust, especially in sensitive areas such as healthcare and the judiciary. “If you understand how a network works, you can analyze it and predict potential errors and problems. It’s a safety issue,” warns Rodríguez Aguilar. “We want to know when something works well and why, and when it doesn’t work well and why.”
Leading AI companies recognize this limitation and are actively working to better understand how their models work. OpenAI’s approach involves using one neural network to observe and analyze the mechanisms of another neural network. Meanwhile, Anthropic (another major startup founded by a former OpenAI founder) is investigating the connections that form between nodes and the circuits that are generated during information propagation. The companies seek to examine factors smaller than nodes, such as node activation patterns and their connections, to analyze network behavior. They aim to use the simplest components first, with the aim of scaling up their findings, but this is no easy task.
“OpenAI and Anthropic are both trying to describe much smaller networks. The GPT-4 network is too large, so OpenAI is focused on understanding the neurons in GPT-2. We need to start with that,” Aguilar explains.
Deciphering this black box will yield significant benefits. In language models, currently the most popular algorithms, such understanding can prevent faulty inferences and reduce the notorious hallucinations. “One problem that could potentially be solved is that systems often return inconsistent answers. Currently, this process is quite empirical. Interpreting the network We don’t know how, so the most thorough training is carried out, and if that training is successful and the tests are passed, the product is launched,” explains Rodríguez Aguilar. However, this process does not always produce the desired result. This is evident when Google Gemini’s first launch incorrectly generated images of Nazis with Asian features and images of black Vikings.
The lack of transparency regarding how the algorithm works is consistent with legislative goals. “European AI law requires developers to provide clear and easy-to-understand explanations of how their AI systems work, especially in high-risk applications,” says Boron. However, he clarified that these systems will continue to be available as long as users are given sufficient explanation of the basis for the system’s decisions.
Rodriguez-Aguilar agrees that there are tools available to unravel the results of algorithms, even if the exact mechanics of the process are still unknown. “But what I’m most concerned about, beyond explainability and transparency, is robustness. Making sure these systems are secure. What we’re trying to do is It’s about identifying circuits in your network that pose a safety risk and can lead to unsafe behavior.
The ultimate goal is to maintain control over AI, especially when it is deployed in sensitive areas. “If we implement AI to suggest hospital treatments, drive self-driving cars, or provide financial advice, we need to make sure it works correctly.” highlights that researchers are keenly focused on understanding what is happening within the inner workings of algorithms. It’s not just a scientific curiosity.
Sign up for our weekly newsletter and get more English news coverage from EL PAÍS USA Edition
