This week I read like five language related papers for machine learning from the 2000s or earlier and it's really shocking how much the field has changed these last 20 years.
@jt Even rereading "Attention is All You Need" feels like a trip through time now...
@dlzv That one is on my follow up list haha. I was focusing on Bayesian models because they seem easy to understand and implement without industrial amounts of data.
@jt There are plenty of pretrained models for French! I think it's probably one of the most well-studied languages: https://huggingface.co/models?language=fr&sort=downloads
Thanks for the refs!
@dlzv Thanks, I'm going to give them a try :)
@dlzv Yeah, I kinda dismissed pretrained models because everything is in English and I work in French... besides our data is kinda specific, but I may try at some point in the future.
I think a nice panorama is given in "Two decades of statistical language modeling: Where do we go from here?" and some of the references there, although my favorite is "Latent Dirichlet Allocation" (which came later). Those are two of the ones I read lol.