What a folding ruler teaches you about neural networks

Deep neural networks are at the heart of artificial intelligence, ranging from pattern recognition to large-scale languages and inference models like ChatGpt. Principle: During the training phase, the parameters of artificial neurons in the network are optimized to perform specific tasks, such as autonomously discovering objects and distinctive features in the image.

How exactly this works, and why some neural networks are stronger than others, is not easy to understand. The strict mathematical explanations seem out of reach of current techniques. However, such understanding is important if you want to build artificial intelligence while minimizing resources.

A team of researchers led by Dr. Ivan Dokumanic from the Department of Mathematics and Computer Science at the University of Basel has developed an incredibly simple model that replicates key features of deep neural networks and allows for optimizing parameters. They published the results in a physical review letter.

Neural Network Division

A deep neural network consists of several layers of neurons. When learning to classify objects in an image, the network approaches the answer layer by layer. In this step-by-step approach, two classes (e.g., cats” and “dogs” are increasingly clearly distinguished, known as data separation.

“Each layer in a performance network usually contributes equally to data separation, but most of the work can be done by deeper or shallower layers,” says Docmanic.

This depends, among other things, on how you build the network. Neurons simply multiply the data they contain on a specific factor, and which experts would call it “linear”? Or do you want to perform more complicated calculations? In other words, is the network “nonlinear”?

Further considerations: In most cases, the training phase of a neural network also includes elements of randomness or noise. For example, in each training, a random subset of neurons can be simply ignored regardless of the input. Oddly, this noise can improve network performance.

“The interaction of nonlinearity and noise results in extremely complex behavior, making it difficult to understand and predict,” Docmanic said.

“Again, we know that an equalized distribution of data separation between layers will improve network performance.”

So, to make some progress, Dokmanic and his collaborators took inspiration from physical theories and developed a macroscopic mechanical model of the learning process that can be intuitively understood.

Pull and shake the folding ruler

One such model is a folding ruler with individual sections corresponding to layers of neural networks and open at one end. In this case, the nonlinearity is due to mechanical friction between sections. You can add noise by swinging the edges of the folding ruler irregularly during pulling.

The result of this simple experiment: A slow and steadily pulling the ruler unfolds the first section, with the rest still largely closed.

“This corresponds to neural networks where data separation occurs primarily in shallow layers,” explains PhD Chensi. He is a student in the Dokumanic group and the first author of the study. Conversely, if you pull it quickly while shaking a little, the folding ruler will be cleaner and unfold evenly. In a network, this results in uniform data separation.

“We simulated a similar model using spring-connected blocks and analyzed it mathematically. The agreement between the results and the results of the “real” network is almost creepy,” Shi says.

Basel researchers plan to apply the method to large-scale language models soon. In general, such mechanical models can be used in the future to improve training of high-performance deep neural networks without the trial and error approach traditionally used to determine the optimal values of parameters such as noise and nonlinearity.