thoughts

13 march 2025

The field of NLP has been in a bit of a crisis since the arrival of LLMs. Unofficially, the primary goal of the field has always been to produce a system (or systems, plural) that can manipulate, understand, and generate text as effectively as a human (though not necessarily in the same way). My claim is that this goal has been effectively reached (since around GPT-3). Certainly, if you presented someone with no knowledge of LLMs a ChatGPT-generated text, and told them it was produced by a human, there is very little chance they would suspect anything.

Understandably, this has caused a bit of discord among NLP researchers. No scientist actually legitimately hopes the central mysteries of their field will be solved, as they will then be out of a job. For NLP, this has caused a majority of researchers to pivot to LLMs, and a minority to write highly-critical position papers. This, to me, does not seem like a particularly sustainable arrangement.

In fact, the hottest thing in LLM research at the moment—that is, "agentic behavior" and LLM reasoning [1]— has pretty much breached containment and escaped from NLP, instead returning to ancient topics in AI like search and logic. I think this is good evidence for my claim above, suggesting that we have modeled language so well that the only thing left to model is the actual thought that produces language! If you are a linguist who strongly believes that language is a distinct and self-contained psychological process, this should be great news for you.

Okay, back to NLP. NLP is a bit of a weird field in that it simultaneously studies methods for working with natural data but also the data itself. This leads to papers such as the original word2vec paper which both proposes a new method for various NLP tasks (static word embeddings) but also uses said method to make the observation that words with analogical relationships form predictable structure when embedded in these vector spaces. Likewise, NLP conferences tend to be composed of a bimodal distribution of people:

Engineering-minded folks who are interested in optimizing performance on valuable tasks
Science-minded people who are interested in understanding how and why certain methods work

So from the basis of my claim that LLMs effectively solve NLP, what are my predictions? I suspect that continuing to try and hill-climb with more and more accurate models is likely a losing battle, or at least an unexciting one. There will certainly always be room to optimize, but the improvements will only get smaller, and the focus will largely be on improving the speed, efficiency, and portability of models.

On the other hand, I think there are a huge amount of potential for exploring how and why these models work, and using them as models to understand language. My advisor liked to rant that we now have complete models of language that are small enough to run on a laptop, and yet people who study language refuse to use them.

Perhaps moving in this direction would mean less funding from industry, and less hype from the media. On the other hand, it feels like far more interesting and fundamental science.

Of course, I may be as wrong as everyone who declared that physics was solved prior to quantum.

[1] I realize these are distinct topics, but they kind of have the same vibe.

7 march 2025

I suspect that studying a language is a bit like studying a snowflake. Even at a distance, you can see the structure and intentionality, and as you get closer and pull out a magnifying glass you can see more tiny, beautiful details. These details almost seem to go infinitely smaller—a fractal.

Now the expert, you can fill a book with your knowledge about the snowflake. Turns out, others have written their own books on their own snowflakes, and browsing through them you can find patterns that look the same as yours. You might even devote your life to reading these books and sorting the snowflakes into piles based on their patterns, so you can now understand (roughly) all of the snowflakes in the world. However, it's a bit uncertain whether you've really learned much about snowflakes at all!

This is unashamedly a bit of a rant about the current research agenda of (formal) linguistics. The vast majority of garden-variety generative grammar work is endlessly looking closer at snowflakes, hoping that maybe if we look close enough we will suddenly see what snowflakes are. Meanwhile, typology is going around recording data on various snowflakes and making up categorizations, in hopes that maybe a big enough sample will reveal the truth.

The reason we understand what snowflakes are is because we understand how they are formed. It seems to me like that must be how we should understand language. We would need to understand how it is produced and processed in the brain, how it arises within a population, how it spreads and changes, what laws of information theory govern it, and how variation occurs.

Not to stretch the metaphor too far, but I do suspect it is the case that language has a fractal attractor. That is, the formation of a language instance is so highly sensitive to some random starting condition (whatever that means in the context of a population) that it is impossible to predict the full extent of its rules (which have infinite complexity, limited only by hardware restrictions).

Worth noting, this really only applies to morphology/syntax/semantics/and up. Phonology (❤️) is obviously correct and an obvious universal thing with good biological motivation.