Recent searches
Search options
Yay! John Hopfield and Geoffrey Hinton won the physics Nobel for their work on neural networks... work that ultimately led to modern-day machine learning.
Some of you are wondering why they got a *physics* Nobel.
In the 1980s, Hopfield invented the 'Hopfield network' based on how atomic spins interact in a chunk of solid matter. Each atom's spin makes it into a tiny magnet. Depending on the material, these spins may tend to line up, or point opposite to their neighbors, or interact in even more complicated ways. No matter what they do, at very low temperatures they tend to minimize energy.
A Hopfield network is a simulation of such a system that's been cleverly set up so that the spins store data, like images. When the Hopfield network is fed a distorted or incomplete image, it keeps updating the spins so that the energy decreases... and works its way toward the saved image that's most like the imperfect one it was fed with.
Later, Geoffrey Hinton and others generalized Hopfield's ideas and developed 'Boltzmann machines'. These exploit a discovery by the famous physicist Boltzmann!
Boltzmann realized that the probability that a chunk of matter in equilibrium has energy E is proportional to
exp(-E/kT)
where E is its energy, T is its temperature and k is a number now called Boltzmann's constant. When the temperature is low, this makes it overwhelmingly probable that the stuff will have low energy. But at higher temperatures this is less true. By exploiting this formula and cleverly adjusting the temperature, we can make neural networks do very useful things.
There's been a vast amount of work since then, refining these ideas. But it started with physics.
@johncarlosbaez That's the causation the wrong way round. Non-physics inspiring breakthroughs in physics, would be physics. But physics inspiring breakthroughs in non-physics is not physics. Or should spin networks win the Nobel prize then? Neural networks have not produced anything meaningful for physics yet, apart from findings without any interest for physicists.
Agree. Physics ideas have entered in so many different fields. Then one can give a Nobel prize in physics for basically anything. I'd be happy if the Nobel Foundation instituted a Nobel prize in computer science instead.
@pglpm @gerenuk - you two seem to have a firm idea of what counts or doesn't count as physics. Your idea doesn't match that of the Nobel prize committee. So, you can argue with them.
But not me. Personally I don't care about divisions between subjects, nor do I care who wins the Nobel prize. It's all just fluff. So to me, this prize is mainly a nice excuse to teach a few people about some ideas from statistical mechanics that underlie machine learning.
@johncarlosbaez @gerenuk Mine is a firm – and of course personal – idea. I don't see any divisions of subjects either, and in principle I don't care about Nobel prize and similar fluff.
But I'm noticing an increasing problem (a problem from my point of view), which I could express this way: there may be fewer and fewer people who want to hear about ideas from statistical mechanics that underlie machine learning. Because, why bother about statistical mechanics? Just put the data in a machine learning algorithm, and let it make the prediction.
I'm noticing this in my university department and others. As a physicist/mathematician, my deepest pleasure comes from trying to understand a phenomenon, in a combination of physical-geometric visualization and maths; hopefully in a way that leads to new physical-geometric ideas and even new maths. But in many new physics projects there is opposition to this: more and more people push to just put measurement data in machine-learning algorithms, train them, make them predict the quantities of interest. And they may well succeed. But what about the physical understanding of the phenomenon?
Maybe the idea of the Nobel Prize committee was to emphasize the importance of physics to computer science. But I think the result will be the opposite, leading to an increase in the problem above.
Maybe this is just a passing phase. Maybe not. And of course there's no "right" or "wrong". Who says how "physics" should be done? Still it makes me deeply sad.
@pglpm wrote: "But I'm noticing an increasing problem (a problem from my point of view), which I could express this way: there may be fewer and fewer people who want to hear about ideas from statistical mechanics that underlie machine learning. Because, why bother about statistical mechanics?"
As a retired mathematical physicist I am blissfully insulated from such ignorant attitudes, and will do my best to remain so. It seems obvious that in the long run, really thinking about things can pay off in ways that mere fiddling about cannot.
@johncarlosbaez @pglpm But when Rome is burning, how do you choose between really thinking about things and fiddling?
Omg, Stephen Grossberg is gonna flip out. 
Honestly, for anybody who knows anything about the field of neural networks, this is a massive joke.
With all due respect, John, saying that neural networks research is physics just because somebody kicked up some dust using some physics terminology is not doing either physics or neuroscience research justice.
@TonyVladusich @johncarlosbaez
how thought and intelligence arise from physical processes is easily one of the oldest and most important questions in science.
It clearly counts as physics in the abstract. It hasn't been something ordinary physicists do day to day, though, because it's been intractable. Now we're getting some traction and the Nobel prize committee is declaring it open season on physical models of intelligence.
@ants_are_everywhere @johncarlosbaez
Where is there any evidence whatsoever that modern statistical sequencing machines have gained any traction on the extant problems of deciphering human intelligence?
Calling this physics disrespects physicists doing wonderful research that will now be less likely to be rewarded.
It also promotes the impression that statistical sequencing is solving problems in understanding human intelligence, which it clearly is not.
@TonyVladusich @johncarlosbaez ANNs are universal function approximators.
https://en.wikipedia.org/wiki/Universal_approximation_theorem
If you feel so convinced they're missing some aspect of intelligence you better have a good proof that intelligence lies in some space of functions that is missed by the approximation.
@ants_are_everywhere @johncarlosbaez
This is a completely nonsensical argument. The onus is on you to provide evidence that a model predicts aspects of human intelligence.
@TonyVladusich @johncarlosbaez
nope
You're the one trying to play moralizing AI skeptic telling the Nobel prize committee what counts as physics and what doesn't.
It doesn't matter to me whether you agree one way or another so I have no proof obligations. But I do think it's a rather strong position for a computational neuroscientist to argue that neural networks have given us no insight into how the mind works. I frankly find that position way too strong to be worthy of serious discussion.
You're a neuroscientist so you have the skills to read papers bridging your field and neural networks. But you'll get nothing out of them if you go in with a chip on your shoulder.
@TonyVladusich - but I didn't say "neural networks research is physics". I say "it all started with physics" - perhaps an exaggeration but I was out of room for qualifiers - and I gave evidence for how Hopfield and Hinton were seriously using ideas from statistical mechanics. Yes, neural network research is now its own subject. I don't think anyone is claiming otherwise.
Wikipedia on Hopfield networks:
"The Sherrington–Kirkpatrick model of spin glass, published in 1975, is the Hopfield network with random initialization. Sherrington and Kirkpatrick found that it is highly likely for the energy function of the SK model to have many local minima. In the 1982 paper, Hopfield applied this recently developed theory to study the Hopfield network with binary activation functions. In a 1984 paper he extended this to continuous activation functions. It became a standard model for the study of neural networks through statistical mechanics."
Wikipedia on Boltzmann machines:
"They are named after the Boltzmann distribution in statistical mechanics, which is used in their sampling function. They were heavily popularized and promoted by Geoffrey Hinton, Terry Sejnowski and Yann LeCun in cognitive sciences communities, particularly in machine learning, as part of "energy-based models" (EBM), because Hamiltonians of spin glasses as energy are used as a starting point to define the learning task."
Fair enough, you could argue it started with physics, although there was a lot of NN research that predated the spin glass and Boltzmann connections. That research was not motivated by physics and was arguably far more advanced than the “physics motivated” work.
But since when do folks receive Nobel prizes for applying existing physics ideas?
@TonyVladusich - You mean like LIGO's detection of gravitational waves? The physics was worked out long before, and we all knew gravitational waves existed - Hulse and Taylor had already won the Nobel for that - so one could argue that all LIGO did was an (magnificent, awesome) application of physics. But apparently the Nobel committee thought that was fine.
Or how about the 1912 physics prize, to Gustaf Dalén, for "for his invention of automatic valves designed to be used in combination with gas accumulators in lighthouses and buoys"? 
As you well know, the LIGO example is not an application of theory but an experiment to test theory. It’s not applied physics it’s physics.
The second example, well what can is say, you’ve a valid counter example.
But I think you’re playing devil’s advocate at this point.
I’m certain you know many physicists more deserving of the award.
@TonyVladusich - I don't really care much who wins Nobel prizes, but yes a lot of my favorite physicists never won Nobels. Heck, Einstein didn't win one for special relativity, and he didn't win one for general relativity - just one for understanding photons, which is fine as far as it goes.
Btw, actually the point of LIGO was not to test our theory of gravitational waves: a billion dollars would be a lot of money to check something we all believed in, and indeed the NSF wouldn't have spent that much if they weren't darn sure it would see something. The real point was to revolutionize astronomy by creating a new kind of telescope: we've seen a lot of unexpected things, like lots of black holes of masses 30-60 times the Sun, and we don't know where they came from. You could call that physics, astrophysics or astronomy... it doesn't really matter to me.
Yep, fair points all!
I also could not really care less who wins prizes of any sort.
My concern in this case is that it seems to me to be part of a politically motivated campaign to legitimise the current AI boom by seeming to ground it in solid science, and what’s more scientifically solid than “physics”?!
@TonyVladusich - I don't know what's motivating this Nobel prize committee. For all I know, you could be right - or they could be trying to prove that physics is still relevant to everyday life, and is worth supporting because it leads to money-making inventions: it's not just Higgs bosons, black holes and such.
Yep, and obviously neither do I. But as somebody who works in both science and technology, it concerns me greatly to see folks in the latter justify their enshitification of the world in terms of the former (and I do see this daily).
@johncarlosbaez it's good that you see it this way, Hinton himself has a different opinion. He humbly stated that the prize is important because it gives him credibility, and shows that Chomsky's school of linguistics is wrong and worthless. Because we know having a Nobel prize in physics makes you an authority, especially in linguistics and neuroscience.
As if there wasn't enough embarassment in this already.
I'm just interested in the part "Chomsky's school of linguistics is wrong and worthless"
Do you have further factual details? Just so as I can put some "exploratory observations" in my notes about Chomsky's grammars
My knowledge is still partial, and that does not help me much to know only that Chomsky has it all wrong.
His universal grammars and variations for example, seems pretty solid to me (and many other people actually).
Or was it humor?
@stphrolland @johncarlosbaez I apologise for the misunderstanding. I am a mathematician, not a linguist. And even if I was one, I wouldn't say such a thing about many people's work.
This was what the mentioned new laureate managed to squeeze in in the reaction, which I find outrageous.
My understanding of the audio recording is that linguists of the Chomsky school seems to say that NN do not / cannot understand Languages the way human do, and it is countered with the argument that NN do understand Languages better by G. Hinton.
Currently I still think that both approaches vaguely overlaps.
Chomsky's grammar heavy on the precision scale. Word embedding and LSTM/Recurrent stuffs more heavy on the imprecision scale.
But I am neither professional mathematician nor linguist. My opinion is still in progess ;-)
@johncarlosbaez @stphrolland what true gibberish that is. I am no great fan of Chomsky, but this is a totally embarrassing opinion - LLMs only give more fuel to Chomsky’s primary criticism of behaviourism about the poverty of stimulus. Also, why would interest in the deep structure of language be a somehow suspect enterprise - our innate ability to recognize the grammaticality of sentences remains something that should give awe. It confirms to me that machine learning people are vapid, anti-intellectual, and overly impressed by their own techniques.
@johncarlosbaez Many ideas in quantitative finance started with physics. Does it also count as physics then? Physics basically inspires all subjects with dynamical and statistical systems. Do we really want to use physics in this broad sense?
It occurs to me that your explanation implies that they treated information as energy. Thus information = energy. Or something like that.
@johncarlosbaez now they have no excuse not giving one to Ed Witten. He took ideas from physics and led to many great advances in mathematics!
@liuyao - the Nobel prize has never been given to purely mathematical results and they probably won't start now. Penrose won a prize for the singularity theorem but only in conjunction with two astronomers who detected the supermassive black hole at the center of our Galaxy. Machine learning, on the other hand, is a practical technique, and Nobel really wanted to give prizes for practical discoveries.
@johncarlosbaez thanks for the explanation! I'm not sure why this specific Nobel choice is getting so much criticism..
@elduvelle - a lot of people don't like AI and they think machine learning researchers have essentially managed to steal the physics Nobel.
Those are two separate categories of people, and I am not sure they actually overlap that much.