Recent searches
Search options
watch a 2 layer neural network learn to separate two classes to the left and right
that was with SGD+momentum. Here's what it looks like with the Adam optimizer
and feature engineering can be important, even for neural networks! Here we feed the points as polar coordinates, rather than as (x, y), and it learns much faster
@matthen2 The struggle is real.
@AlanZucconi so jittery! and this is after exponential moving average
@matthen2 that’s nice Matt.
@matthen2 That’s supercool! Makes you appreciate Adam haha!
@ampanmdagaba adam struggles a lot less!
@matthen2 super cool Matt, thank you!
@matthen2 What does the y dimension represent? Great viz,
@matthen2 is the training loss as shaky as the point movement, or is it smooth as silk? How does the batch size compare to the dataset site? Are these points train or held out?
To many questions, but very cool visualisation idea!
@lb loss is also jittery, but I do use smaller random procedurally sampled batches. In picking hyperparams I optimized a bit for producing a nice animation versus learning efficiently
@matthen2 thanks.
The real reason for my questions is that I was wondering whether this or similar could be used to gain intuition both about the effect of architecture components as well as of optimizer components.
@lb definitely! The type of non linearity also shows interesting differences in the animation. This is relu
@matthen2 comparing different norm layers would be very interesting. Though unclear how to factor out the effect of lr.
Do you plan on publishing the code at some point? Want to know if I need to start working on repro or can just wait :-)
@matthen2 that's fun, it's like it's being stretched under extreme tension
the gibbs phenomenon is apparent here
@matthen2 Hi Matt! You probably get asked this a lot, but can you share some of the code for this animation?
@matthen2 superb visualisation!
@matthen2 That’s so beautiful!
@matthen2 fascinating! Is this code on GitHub?
@matthen2 Way to shake out that nonlinearity!
@matthen2 do you have the code that generated this?
@matthen2 very funky. What are the x and y axes here?
@matthen2 If you want a challenge: Use ML to separate the blood cell types from flow cytometry data. I worked on the electronics years ago, and they had problems doing that with handmade algorithms. One of the companies might be able to supply sample data. The medical/animal labs handle a lot of that.
@matthen2 What if you feed it both? Is it able to leverage that even better? Or maybe the model gets too big and it outweighs the benefit?
@matthen2 feature engineering for the win!
@matthen2 A support vector machine with exponential kernel, say, should be very fast learning this pattern, no?
@mweiss it could learn the middle disk with one support, but would need more for the annular disks
@matthen2 I see, so exponential plus polynomial (quadratic would suffice) to get the nonlinearity. It's been a while but many general kernel should be that flexible.
@matthen2 oy, she's struggling
@matthen2 can the network separate a spiral?