One thing that was really hard for me when I was in academia is how lonely it felt to be working on something that nobody cared about. I was seeing all of these beautiful things and I couldn't share them with anyone.
Sometimes I want to go back and do mathematics "for real" but then I feel sad when I think I won't be able to share it with anyone.
@JordiGH Yeah, it's definitely the suckiest part for me, too. I'd have dropped out long ago if it weren't for the fact that my school hosts a dynamics conference every spring and a nearby school hosts a dynamics conference every fall (with funding for grad students). So I'm basically guaranteed two long weekends/year where I can geek out with a bunch of my peers.
Mathstodon helps, too. :)