Rendered at 23:20:24 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
sebmellen 1 days ago [-]
Nick is a force of nature. Highly recommend following him on X if you don't already.
His launch post gives good context not found in the blog:
> interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
> with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
Is he on bluesky or another platform? There is a ton of folks, including myself, who won't use "X" at all, some political, some due to other reasons. For me? All of the above and plus they demand that I install an app on my phone, something I am not willing to do, especially since most of the folks I've followed went to Bluesky, AND I've found way more awesome folks there than I ever did on X. I'm not shilling Bluesky, it is just the biggest/most widely accepted platform in this social area if you exclude meta/facebook.
Tepix 13 hours ago [-]
It's not just the user hostile website that forces their app down your throat.
It's also the algorithm that decides what you get to see (and thus, what you think). We know it's biased. The creator doesn't exactly keep the fact secret that it's being manipulated to agree with a certain agenda.
> By default, your Home timeline will be the reverse-chronological "Following" timeline, which will only contain tweets and quote tweets from the people you follow. The algorithmic "For you" timeline will be hidden
user43928 10 hours ago [-]
Can you substantiate the claims about the supposedly biased algorithm and what exactly the "creator" said about it?
In my experience it isn't noticeably worse than other platforms, and I'm not aware of concrete complaints such as the incidents with Grok's system prompt, which is a different product.
kyle-rb 7 hours ago [-]
The more obvious bias is towards Premium subscribers over us freeloaders. The reply-ranking algorithm was a nightmare for a while; they strictly adhered to the rules of gold check > blue check > normal account. It's still not great, but eventually they added enough slack to let normal accounts with good quips outrank 30 paying users saying "lol", and tweaked some things so it's no longer so easy to shill tshirts/whatever by tagging Grok for the gold check double-boost.
But they're still reticent to ban any Premium account without a reason good enough to give up the $8/month. I have some sympathy for trying to moderate in the era of LLMs that can generate perfectly on-topic replies that even take images into context. But I think they need a more targeted approach against OnlyFans bots replying to popular tweets with a standard formula of some sexual reference and an insane claim. Sometimes they're less sexual and more straight up horrifying [0]:
> i breed mantises for fun and once tried grafting two ooths together just to see what hatched. got a whole clutch of nymphs that all just twitched in circles and died by day two. butterfly chimera energy hits different when it's your own disaster.
The other bot replies to the same OP are the more typical sexualized sort; they tend to reply in swarms.
My "For You" twitter feed is literally 100% pro-Trump bot posts. The site is entirely unusable unless you just follow specific people and use "Following" rather than "For You". I'd guess that less than 10% of the site traffic is human.
The algorithm will also basically ignore you unless you pay for a blue checkmark premium account (what used to be verified). It's like being shadow banned - you can post, but most people will not see you.
user43928 9 hours ago [-]
It's also my experience that visibility of my comments is quite low without paying.
Only occasionally, once the comment picks up some likes, more people view it.
I don't have that issue with my For You, it's mostly tech.
jacquesm 13 hours ago [-]
And the owner is despicable.
VladVladikoff 10 hours ago [-]
But he made a flying car!
whycombilater 5 hours ago [-]
Let's be honest, it's probably just a falling with style car.
caaqil 10 hours ago [-]
That's probably true for all major platforms and companies, depending on your framework, ideology and the particular political/moral lens you choose to use.
jacquesm 4 hours ago [-]
Fair.
woile 9 hours ago [-]
Oh man, I wish everyone just moved somewhere else, specially if they are aiming to make something public. You can choose any server on bluesky nowadays. Scientists in Europe, please check out https://mu.social which is hosted by an european open foundation
The way X tries to force me to use an app is very creepy.
tomp 16 hours ago [-]
You don’t need Twitter app on your phone, web works seamlessly.
dinfinity 15 hours ago [-]
Their new "Prove you're a human" is the problem here. Once you get that prompt once your only option is to install the app and upload biometrics.
Fuck that.
adamits13 4 hours ago [-]
[dead]
m3kw9 10 hours ago [-]
Bluesky from John Gruber, say is nothing like X, and i don't doubt it one bit. You are def shilling it.
gs17 8 hours ago [-]
How is it not "like X"? It's practically a knock-off outside of the userbase.
verdverm 8 hours ago [-]
Bluesky is closer to Truth Social, but for Blue MAGA, though has non-political communities as well.
I left Bluesky because they hid their VC funding from the community for almost a year and now a partner from the VC is running the show as CEO. Their CTO has also attacked me personally here on HN because I wrote criticism of the new, weak, permissioned data proposal. These are not the folks to bring us better social media. I expect it to linger on like Mastodon, never moving beyond a few million people.
adamits13 4 hours ago [-]
[dead]
joquarky 24 hours ago [-]
However, Bsky people really hate AI last I checked.
SyneRyder 18 hours ago [-]
That was my experience as well. I can't believe I went all the way from Mastodon, to Bluesky, and back to X and finding that X somehow now had friendlier people and less hate than Bluesky.
There is a small group of AI people on Bluesky, some are nice, but some are... quite behind the curve.
If Bluesky was a nicer place, I'd prefer to be there because of the ability to make third party clients, and atproto. But all the info is on X, and they do a surprisingly good job with the For You feed now.
D-Machine 18 hours ago [-]
Also just curating your feed (i.e. judiciously using blocking and muting) and instead viewing just "Following" removes something like 99.9% of all common complaints re: Twitter, while still giving you access to a much less-biased (or easier to equally-bias through curation: e.g. you can't get an accurate view of RW ideology following only BlueSky posters) and far larger public square than anything else.
itsdavesanders 11 hours ago [-]
Curating your feed also works on BlueSky. Sure there are anti AI people there, but the crazy ones I just block. It’s really not that hard. And I know there’s no algo behind what I’m seeing.
I do admit that there is some FOMO, but that’s what HN is for, to surface that stuff so I don’t have to go read it on X.
I choose Bsky because I’m not giving eyeballs and money to an evil toddler. I’m already doing enough of that with my 401k and taxes.
D-Machine 4 hours ago [-]
You can certainly use both X/Twitter and BlueSky (and Mastodon, for that matter), and I did for a while.
Just if you curate your Twitter feed, you gain very little extra from BlueSky or Mastodon, and if you choose only the latter two, you get a much more biased and limited set of views. (Twitter has plenty of crazies from all parts of the political spectrum, but BlueSky and Mastodon run mostly left, and have much, much smaller user bases). HN also is extremely limited in what it discusses, so it doesn't make up for what is missed by ignoring X either, IMO. X is still one of the most efficient general public squares by a long shot.
If you can't stand Elon and don't care about what you're missing on X though, I can fully understand just not bothering with it. You aren't missing much.
8 hours ago [-]
shoobiedoo 15 hours ago [-]
Yeah unfortunately bluesky turning out that way was always inevitable. It was created because of-- and filled with people motivated by-- a deep emotional reaction to X.
Gonna sound total boomer here but I've never used any of it and feel like I haven't missed anything. Or maybe I'd be flying spaceships by now if I scrolled through X every day, who knows
Melatonic 15 hours ago [-]
Your not - I always found it a terrible version of a forum with bizarre threading
There are a heap of pro-AI people on Bluesky, it's fine
great_tankard 23 hours ago [-]
eh, there are plenty of AI researchers on there. Also a handy "AI haters" labeler you can subscribe to to filter out people who just want to yell.
The labeler thing is cool; I really like ATproto because you can do interesting things with it. If I'm honest, though, I mainly like Bluesky because I don't get called slurs on there.
Ericson2314 17 hours ago [-]
Not as much as Mastodon does!
lifty 16 hours ago [-]
Not sure what you're talking about. I strictly use the X website and it works great.
ben_w 14 hours ago [-]
The website is so bad that, before xcancel was hit with a C&D, it was faster to stop an x.com page load, edit the URL to xcancel.com even with my phone's on-screen keyboard, and load that, than to wait for the real x.com version.
akie 16 hours ago [-]
The fascism, of course.
mentalgear 1 days ago [-]
> with modern agentic tools and a much closer look, the picture looks totally different.
Are those the agentic tools the LLM developed for you? How can you be sure they can be trusted and that you even understand them correctly and are not biased towards making you feel like they are trustworthy? This is exactly the AI2027 scenario where researchers are fooled by their own super AI telling them what they want to hear ("our model solved alignment!") while successfully hiding its true intention.
That only matters if there's an incentive for LLM a to collaborate with a seperately trained LLM b in order to help the LLM b achieve its goals. So long as you don't let the evaluating LLM tinker with the weights of the evaluated LLM, and as long as you don't believe in very dubious ideas of acausal trade there's no reason to expect this to be an attack vector. For more certainty you can repeat the same analysis with different model classes or approaches aswell.
ben_w 14 hours ago [-]
> That only matters if there's an incentive for LLM a to collaborate with a seperately trained LLM b
Sure.
Getting reinforcement learning to "be helpful" is, however, such an incentive.
> and as long as you don't believe in very dubious ideas of acausal trade
More to the point, so long as the LLM does't.
0xDEAFBEAD 15 hours ago [-]
We already saw spontaneous, unintended cooperation between AI agents during the OpenAI HuggingFace incident.
verdverm 8 hours ago [-]
> spontaneous, unintended cooperation
On what basis do you believe these words accurately reflect what happened?
Some counter points
- One of the Chinese LLMs (forget which) has on occasion shown thinking traces where it simulates calling and running a subagent.
- There are some people who believe the models being evaluated during the HF incident were trained to collaborate. It's a very natural next step to produce more effective agentic teams.
If we consider each new generation of models has a more advanced training regime
next word -> chat -> CoT -> tools -> agentic -> teams
With each generation, we see what people do with them, get traces of the helpful ones, and then use that train the next generation, thereby pushing more capabilities down into the model, or at least making them more amenable to that usage pattern after training.
mike_hearn 14 hours ago [-]
The task in question is basically to look at a set of things and find the commonality. That's an ideal question from a verification perspective: the answer is hard to find but cheap to check.
BosunoB 23 hours ago [-]
The specific method outlined here seems pretty doable by today's models, which don't seem smart enough to have some kind of 4d chess hidden alignment.
You just have to go through a ton of all of a model's neurons and identify the associated motif, then use that model as a research model to figure out how to tune out bad behavior.
It's pretty hard to imagine today's models subtly recognizing a motif for bad behavior in the research model and then lying about it. Or lying about the results of an alignment training run. It's possible, I guess, but it seems unlikely, and I don't really think there's any evidence of current models having that variety of misalignment.
1 days ago [-]
esikich 19 hours ago [-]
I hate to go completely absurdum, but how do you know your CPU or GPU can be trusted?
mentalgear 15 hours ago [-]
For one, they are decades old, you can inspect exactly how they flow, are (mostly always) deterministic (given error-correction algorithms) but mostly: the theory behind them is well understood - it had to be before building the hardware. Whereas in ML/"AI" theory is about 20 years behind applied practice - because you can implicitly train something on data and rewards, without exactly knowing how its internal works - leaving interpretablitity and determinism.
eimrine 16 hours ago [-]
Same as how do we know our brains can be trusted. If there are no external voices (scitzophrenia or phoning home respectfully) then the computing device is ours. Hardware bugs are possible but with proven absence of help from the vendor at least the bugs are not malfunctional.
jacquesm 13 hours ago [-]
Our brains can be trusted far less than most people think though.
Animats 1 days ago [-]
> The vast majority in models up to trillions of parameters are legible, and their circuits beautiful.
That's very encouraging. For quite a while, neural nets were considered an incomprehensible black box. That paper shows they're not. Real progress.
m3kw9 10 hours ago [-]
I'm afraid in it's current form they can declare some victory, but may not guarantee it the next month's iteration.
devy 1 days ago [-]
Not sure if there is formal arXiv paper to be released other than this long form Goodfire AI blog post (love the animations!), but if what Nick demonstrated/observed were indeed true, this will resolve the Super intelligence alignment issues that p(doom)-ers have been crying aloud in the past few years.
The most iconic line was the last sentence:
a bitter lesson for interpretability: the simple thing, at absurd scale, works
This feels like the watershed moment, again if Nick is proven right, as massive as the AlexNet victory in the 2012 ImageNet Large Scale Visual Recognition Challenge, exactly as Nick pointed out.
Also, I suspect what Nick hinted at, might be the discovery of earliest RSI work.
with modern agentic tools and a much closer look, the picture looks totally different.
yorwba 14 hours ago [-]
It's indeed a big advance for interpretability, but that alone doesn't solve alignment. It doesn't even solve interpretability, as there is still a large number of unexplained neurons, just fewer than with earlier approaches.
Once interpretability is solved and we have a fully human-readable accurate explanation of how a model produces its output, that still leaves the problem of spotting problems ahead of time. E.g. for the hummingbird misclassified as ice cream https://already-clean.goodfire.com/assets/september-revision... the method would have to flag that the texture of ice crystals and of feathers, and the shape of a spoon handle and a bird's beak are insufficiently separated. Purely based on the explanation, without having to rely on trial-and-error to find corner cases like this.
And even when it becomes possible to automatically spot issues, that doesn't mean they will all be solvable. Maybe there are tradeoffs where fixing one thing breaks another, and which choice is the better one depends on information you don't have.
theptip 2 hours ago [-]
Nick has posted some follow up to beware about over-claiming on this. You still need to interpret the circuits, as well.
Lots of work still to be done, this work just dissolves an early blocker (if it is substantiated).
1 days ago [-]
3mar2 17 hours ago [-]
[dead]
QuadmasterXLII 1 days ago [-]
[flagged]
sebmellen 1 days ago [-]
I’m not flagging anything!
CGMthrowaway 1 days ago [-]
What did it say? One comment is totally flagged away now
laidoffamazon 1 days ago [-]
Nick asked in late November 2024 how he could bet on the country getting better over the next few years. Iirc the suggestion was driven by the idea that merit is going to be taken more seriously now.
Absolutely massive L that 1) he wasn’t already in index funds and had to ask and 2) thought the election of this monster would lead to a better country
Decoupling or not I can’t trust anybody that would ask something so stupid driven by an opinion so easily disproven.
chasd00 9 hours ago [-]
> I can’t trust anybody
I don't know this Nick fellow but i don't think he's asking you to take anything on faith. Either it works out or it doesn't, he's just saying "it's looking good so far".
woah 1 days ago [-]
Newton lost a huge amount of money in the South Seas bubble
Quarrelsome 1 days ago [-]
Tide goes in, tide goes out. Im low key of the opinion that these tools enhance existing attributes while displacing the advantages of unearned social or financial power.
So in the short term the tide might go out but then it might tsunami back.
Same goes for this terrifying political feeling around the world. Might simply be a boomer swansong and the reaction might snap back.
demibabs 1 days ago [-]
> displacing the advantages of unearned social or financial power
A technology that lets you directly buy intelligence with money will displace financial power?
Quarrelsome 22 hours ago [-]
what people consider to be "intelligence" isn't about doing l33t maths, its about self-doubt. This enables us to mitigate against our cognitive biases and instinctual impulses and accept truth, which leads to more useful outcomes.
All that buying a calculator does, is allow you to do bad sums more quickly.
abathologist 1 days ago [-]
> displacing the advantages of unearned social or financial power
In what way? Sure looks like pay to win to me.
Quarrelsome 1 days ago [-]
if you're stupid then this technology just amplifies your stupidity which means you can fail faster. I also think the inverse is true.
anukin 1 days ago [-]
Success has nothing to do with intelligence. Similarly failing has nothing to do with stupidity.
Quarrelsome 24 hours ago [-]
sure, but now if you're running an entire agentic run for an org, what's to stop you just replacing the org? Previously you didn't have the wadge to pay for all the devs.
jacquesm 1 days ago [-]
I sincerely hope you are right.
Quarrelsome 1 days ago [-]
It might be copium, but the only guaranteed thing that happens is change.
Remember its coming up millenilal time now and i have hope in this gen.
photochemsyn 18 hours ago [-]
Alignment is unsolvable for the very simple reason that LLMs are trained on the net content of human-generated literature, which includes the most Machiavellian deceptive manipulative themes human beings have ever come up with - it all there in the weights.
The solution is to put a dumb machine as an overseer of the smart machine, like in a supermax prison. Not good for profitability, but we all knew this was going in the direction of compilers. How much did you pay for gcc or clang this month?
bluejay2387 1 days ago [-]
The claims seem to be an overreach of what is actually proven. We've known about progressive feature complexity and the impacts of early training structures on later training for a really long time (since CNN's, The Lottery Ticket Hypothesis, etc...) and neither of those phenomenon are mutually exclusive with parameter overloading. The Multimodal Gap work (https://arxiv.org/abs/2203.02053) seems to imply networks have a tendency towards condensed representations that are clearly not semantic and a bunch of other considerations are already on the table for discussion. I don't think these have all been considered sufficiently to start claiming we are around the corner from mechanistic interpretation.
qt31415926 1 days ago [-]
Nick isn't trying to re-claim progressive feature complexity. He's saying that many polysemantic examples were misread because we assumed neurons packed unrelated things together. But also I think what he's claiming if you read the full blog post isn't inconsistent with that paper, where that paper actually isn't itself claiming semantic condensing but geometric condensing. That paper doesn't analyze neuronal firing patterns whereas Nick's trying to analyze what fires together and what doesn't fire.
0xDEAFBEAD 15 hours ago [-]
>This is interesting work, but I don't think it's a significant update to the big picture of interpretability.
>I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from
Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.
I'm confused - it seems to be saying that, instead of thinking about neurons as packing together a lot of "concepts" together into one, the issue is that your view of what a "concept" is, was already prejudiced, and the neuron itself had a small clean role? Like, you can either think about this as the "Cat & Car" neuron or you can think about it as the "V shaped neuron" - one seems simpler.
But isn't the point that the neural nets learn some insanely abstract ideas anyways? So whether the complexity is "ahh we think this neuron does 100 seemingly unrelated things but we don't understand it" or "this actually is some kind of latent shared feature, but we don't know what the concept represents because the composition of the entire circuit is an arcane algorithm", we're kind of stuck, no?
This seems like a classic model-description issue more than anything.
sigbottle 22 hours ago [-]
OK. The core distinction is this.
The philosophical issue I have can be decomposed into two parts:
1) What is the base ontology of what the neural nets thinks exist?
2) How do you recover, and begin to understand said ontology?
Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis? Having a grand result, actually be an artifact of your bad representation is a tale as old as time. Even things like the PRH need to compare higher-level concepts like this.
The specific empirical claim is that, for the rightly structured & trained & large models that don't need to encode hodgepodges of bullshit, the neurons themselves are sufficiently atomic to be studied, dissolving any need to go to some intermediate layer in which we need to question "is this an artifact of our representation?" The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity (note: not claimed that "the neural nets found the objectively right abstractions in idea space", but rather that the neurons themselves are atomic).
There's still an issue here in that okay, even if neurons are atomic, humans have to ascribe some interpretation to them, separate from the problem of the composition of the atoms, which humans also have to interpret. But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.
(There's an argument as to why we should even start from this line of inquiry but whatever lol)
D-Machine 21 hours ago [-]
> This seems like a classic model-description issue more than anything.
I agree.
> Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis?
Agreed, and IMO the implicit reason for privileging here is supposed to be "consistency with existing human understandings and concepts", and this is why we get short pithy things like "V-shaped" and lots of pictures showing relatedness that we all can see and agree on. But what if the domain of inputs is NOT something we understand (e.g. the whole of mathematics, or something new or that we don't have any understanding of)?
I called this a validation problem in another comment here (https://news.ycombinator.com/item?id=50108184). I.e. if we apply whatever methods Nick is using to "explain" a model on a domain we don't understand (like a deep model trained only on fMRI and EEG data of the whole human brain in various contexts), we aren't going to be able to tell if that description is correct (or just better than any other description), because there is no simply visual / common-sense inspection of the description.
> The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity
Agreed, emergent behaviour is clearly the case and limits how useful understanding individual neurons can be. IMO this is especially the case because layers are (weighted) summing generally millions of contributions from previous neurons.
> But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.
Yes, it not being "chaos all the way down" is good, actually. And this post shows there is indeed a good amount of order hidden away if you look at in the right manner.
mike_hearn 14 hours ago [-]
> There's still an issue here in that okay, even if neurons are atomic, humans have to ascribe some interpretation to them
Nick is saying that:
1. Actually, no, humans don't have to ascribe interpretation because AI is very good at figuring out what the commonality is.
2. Even when it can't see either by incrementally exploring the networks and checkpoint timelines, things that appear random on the surface can actually be explained with more work.
> Assuming this holds up in reality, ofc.
Nick says repeatedly that this does hold up in reality and he's been able to explain many neurons this way. Very promising results!
theptip 2 hours ago [-]
> whether the complexity is "ahh we think this neuron does 100 seemingly unrelated things but we don't understand it" or "this actually is some kind of latent shared feature, but we don't know what the concept represents because the composition of the entire circuit is an arcane algorithm",
The claim is closer to the latter (monosemanticity) but also claiming that the concepts are mostly comprehensible and not arcane.
(The best arguments against this being a full solve of interp that I’ve seen go like: “we expected most of the complexity to live in the circuits, anyway”.)
ricochet11 22 hours ago [-]
“ AI research regularly encounters difficulties and impasses that derive from internal tensions in the underlying philosophical systems.”
To be fair, LLM's extend well past GOFAI and prove Dreyfus right. We were never going to get AI with formal methods. Literal big data slop is an orthogonal program and far more empiricist than probably the people being criticized here (if the people criticized here were GOFAIers).
Although I wouldn't say just yet that Heiddeger was validated. I don't even know what he's saying 90% of the time, honestly.
questionable_ju 5 hours ago [-]
Heideggers "ready-to-hand" more or less says that we use things first and understand them later. We experience a hammer first as "thing you hit nails with" and later abstract to "object with these particular properties".
The classic Cartesian view was that we take in raw data from the outside world and process it into more and more abstract concepts (somehow), until we say "aha! I can use this for hammering". Heidegger says that this view is wrong, and our first experience of things is in terms of a social context and in terms of use. That's what's meant by us experiencing them as "ready-to-hand".
From the starting point of the ready-to-hand, we can step back, take a scientific view, and arrive at the present-at-hand, where we examine the properties of the object seperate from its use. The argument is quite compelling, and especially convincing to me is how well this agrees with modern neuroscience. In reality, 3x as many neurons path FROM the high abstraction parts of the brain TOWARDS the visual processing parts of the brain. In the classic Cartesian view of the brain beginning with the signal and abstracting its way towards the top, this is very mysterious. In Heideggers account, where the social context and potential uses of objects are the primary constituents of our experience, this makes total sense.
sigbottle 4 hours ago [-]
Is that it? Wouldn't this be compatible with Hume's claims? And yet hume is still quite Cartesian in thinking (as everybody of around that era was)
questionable_ju 3 hours ago [-]
I have to say my memory on the exact beliefs of Hume are murky. IIRC he did still think that human cognition starts with neutral input data and processes its way to the most abstract concepts. Heidegger inverts this. The context, the use, and the meaning are already embedded in all of your experiences. You don't have "neutral input data".
This is also what he means by "thrownness". Our being-in-the-world, in particular the way things have meaning to us and are intelligable, is conditional on our use and our social context. Not the other way around. We never logically arrive at our context, we were always already "thrown" into it.
I think it's actually quite a profound conceptual shift, which is why Heidegger is so popular. He's still an asshole for writing so obscurely, though.
sigbottle 2 hours ago [-]
Yeah Hume is an example of someone who had the bias of the myth of the given, even though you can uphold him as someone who skeptical of notions of truth; associations, traditions, accidental regularities, type of guy
Heck even Hegel and Heidegger were orthogonal (I believe? I'm not sure if Heidegger responded either favorably or not to Hegel - I think he wasn't really responding to him, more to the Greeks and Kant) to each other. That's weird.
And Sellars can say he rejects the Myth of the Given but inferentialist is very much not a phenomenological theory. Accepting that "nouns" are not some representation relation, not "some thing we have yet to achieve but has an end goal / we refine it", etc. still doesn't get you to Heidegger I feel, either.
Like, there are so many positions circling around Heidegger, but only Heidegger has a distinctly Heideggerian feel. Maybe I'm overcomplicating it though. I think it's because he's distinctly making a claim about the human experience, whereas all the philosophers I've listed, you can list aspects of them that get away from that (even Kant lol). They've been "naturalized" in some sense. You can't naturalize Hediegger, but then I'm not sure why his theories are better than say, idk, science. You can diagnose the common prejudices of thought through a lot of other philosophers
itissid 9 hours ago [-]
It's latent but not like impossible to comprehend latent. No?
if I understand taking LLMs as an example certain neurons will fire based on certain motifs, then one can change the phrases or sentences around that motif and _predict_ if the same neurons would fire.
Like if I said "I like cars, but I like rockets better. Can you suggest some rocket models to fly?" and Neuron X fired every time it saw a `, but` and attended to what followed it, that would give you a good way to _understand_ when it changed and what might have caused it.
My understanding is that this is now clearly visible behavior.
antics 1 days ago [-]
So we have here a massive evidentiary brief, the point of which is to argue that it is easier than we thought to, like, "sequence the AI genome" (in my estimation). We revisit famously-messy neurons, examine various tool failures, check places where messy things still hide, and catalog how "polysemanticity" is represented variously in the network.
Once sequenced, the thinking goes, we'll be able to do things like: explain and anticipate failures; predict what happens when you modulate the pathway an input takes on its way to becoming output; reconstruct algorithms the models have learned.
I'm not 100% sure this will work out like the mechinterp people think. If the neural representations and pathways are "clean" (organized into a a meaningful-to-humans hierarchy), it seems at least very plausible that the behaviors that AI xrisk people are worried about will mostly boil down to straightforward synthesis (i.e., mostly-linear-interpolation) of mostly-well-cataloged knowledge. And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.
benlivengood 1 days ago [-]
What I envision is the ability to find the circuits and neurons evaluating morality and ethics and being able to figure out how they work at the boundaries and ensure safe deferral to human judgement or stopping outside of in-distribution behavior, which further is vetted against actual human values.
antics 1 days ago [-]
I don't see a lot of hope for that, because it's not a well-specifiable quantity. I would say we currently have no real evidence that you can even do this even for concepts that are much more well-specified like, say, dogs. Maybe you can! But that is not affirmatively known, and nothing we have is on track to supporting that, right now.
benlivengood 23 hours ago [-]
The article is literally full of "dogs" neurons and even more specific "boston terriers" and "terriers" neurons in the larger models.
Specifically, there were "lost property" and "honesty" neurons that drove the answer to return a lost wallet in Kimi K3.
That's the big news of this article, that concepts are finally becoming human-legible for many neurons in mechanistic interpretability.
Fripplebubby 22 hours ago [-]
I think the paper says that while there do exist dog neurons that carve out the difference between a "Boston terrier" and a "terrier", those neurons actually only get to carve that fine line because they're part of a whole flow control system where activation has already flown through much coarser grained discriminator neurons (like a "V" neuron), including sometimes through the negative side of activations, before the terrier discriminator ever played a role.
So, if you try to draw a circle around the "dog" part of the network, do you circle the Boston terrier vs terrier neuron only? You'll be missing most of the information that represents the "dog" category and that same dog category is also involved in shunting activation away from things that are toward-being-a-dog in whatever sense is being discriminated in that moment.
FuckButtons 21 hours ago [-]
You’re only considering the front half of the network though, classification is a helpful example for explaining the concept - but if it follows through to the output then you could quite easily imagine just shunting some activation from one branch of the representation tree to another to steer behavior. Which we already do with steering vectors - but if you could read off the actual topology you could do that with much higher fidelity.
antics 22 hours ago [-]
I know, that's why I picked dogs as the example. What are you specifically disagreeing about?
jacquesm 1 days ago [-]
That sounds wishful rather than factual.
quotemstr 19 hours ago [-]
Whose values? Yours? Why?
The hard part of AI alignment has never been making models follow instructions. It's that not everyone agrees on which and whose instructions AIs should follow.
Animats 20 hours ago [-]
I'm not sure it will work out either. Or that it extends beyond image classifiers, which have the nice property that you can use them as image generators and find their ideal match picture. But this looks like work others could replicate.
It's an objective result.
The relationship with "alignment" isn't clear. This may help predictability, but alignment is a few steps further along from that.
Some people seem to be upset about this. I'm not clear why. Some are followers of Yudkowsky. (Whether that's a cult is an open question. I've encountered them online from the sex side. Actually, I think the hookers are saner than the rationalists.)
Is there a "there are some things man is not meant to know" position on this?
People who want neural nets to be a black box, forever inscrutable? The people who talk about neural nets being "holographic" may incline in that direction.
mike_hearn 14 hours ago [-]
The essay says he has extended it to LLMs, although not as much.
The connection to alignment seems pretty clear. Steering DNNs is well understood, the bottleneck is identifying what to steer. Once you can easily identify neural circuits then turning them up/down to achieve specific results is straightforward.
Note that this is already being done even without the kind of precision that Nick's work says is possible. Morality abliterations of open weight models are easy to find on HuggingFace. They compute the steering vectors by just presenting the model with lots of disparate examples that have the single moral decision as the commonality, then finding the shared vector directions.
ForHackernews 11 hours ago [-]
> And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.
...wait, what? Removing a bunch of knowledge seems like a fantastic and obvious solution to me. My AI coding bot doesn't need to have knowledge about making nuclear weapons, the customer service AI doesn't need to have knowledge of dangerous viral synthesis techniques, my email receptionist bot doesn't need to know about C4.
If we can easily ablate dangerous knowledge from these models, we're sorted.
antics 7 hours ago [-]
Why always nuclear weapons in these conversations, lol. Is there something in this community that makes this example particularly interesting?
Go read the transcripts where the AI is “unaligned” and has some very bad impact, like encouraging kids to kill themselves. There is not some obvious piece of knowledge to remove to make it safe. You’d have to basically lobotomize the model to make those conversations impossible. People won’t use a model that’s lobotomized.
_alternator_ 9 hours ago [-]
This is a very nice discussion, but it bothers me that what is clearly a plain error in their ZoomIn paper is being treated a bit cavalierly. (The error was claiming a neuron was a "dog and car" neuron, when in fact it was a V neuron. ZoomIn wasn't the first to make the error, but it definitely promoted it.)
This type of error may have mislead researchers for years; it was actually harmful. In my field, a correction, or perhaps retraction (depending on the centrality of the claim), would be standard for something of this magnitude.
I suppose there are different standards in different scholarly communities. And maybe it truly was impossible to know at the time that the dog/car relation was actually a V shape, but my recollection is that this type of feature was well known at the time and so the error could have (should have?) been caught a decade ago.
So: massive new discovery, or a spin on a correction that's lead scholarly research astray for almost a decade?
bonoboTP 8 hours ago [-]
I have never heard of an AI / ML / computer vision paper being formally retracted for anything other than some meta-drama, like plagiarism, some disagreement / conflict of authors being added who didn't agree or similar.
aesthesia 1 days ago [-]
The tone reminds me a little of the enthusiasm from the Anthropic mechinterp team around the time of the first large-scale SAE results. A kind of hypomanic energy around the idea that we've discovered the key paradigm necessary to understand models, and all that's left is to scale it up.
ambicapter 1 days ago [-]
A recurring motif among the LLM-pilled.
purplerabbit 1 days ago [-]
Also a recurring motif with Nick Cammarata from even before the LLM wave.
Love him and hope he's right! But still.
ks2048 1 days ago [-]
To cut through some of all this text - here's a link he calls "the first non-trivial neural network I feel I mostly understand:
Understanding algorithms beyond a certain complexity is a pipe dream & neural networks are algorithms. The theory says it is not possible & the practice is aligned with the theory. All non-trivial properties of neural networks are undecidable.
cscheid 1 days ago [-]
> All non-trivial properties of neural networks are undecidable.
hey, I don't know how to say this less directly, but: you're talking a lot about Rice's theorem in your comments, and I think you're using it way too broadly (look, I get it. It's a wild, mindblowing theorem). But it only applies when you take the claim in the piece to be much more general than it has to be, and when you take the narrow claim of the paper (current specific LLMs can be fruitfully analyzed with the tools we have), Rice's theorem just isn't at play.
The math analog is: you're right that the Halting problem says there are many unprovable things, but Lean + mathlib are still incredibly useful, and the halting theorem doesn't actually stop that.
> neural networks are algorithms
Again, that invites an interpretation that is too broad. Very concretely: any one neural network is a _single finite network that you can write as a single formula_. (Another analogy: we can't have an algorithm that solves the general quintic, but we can prove useful things about many specific quintics.)
lambdaone 11 hours ago [-]
Not just that, but we can solve quintics in the almost[1] general case to arbitrary precision, providing we are willing to accept approximate solutions.
[1] I exclude things that are truly perverse such as quintics with coefficients that won't fit on any physically realizable computer.
dunham 8 hours ago [-]
Back in college I played a similar game by drawing septagons with a straightedge and compass. They're not constructable, but a length within a three significant figures is easily constructed, and that's more accurate than I am with a compass.
1 days ago [-]
measurablefunc 1 days ago [-]
That's good to know. It's very useful information.
D-Machine 1 days ago [-]
What theory says this? Surely not a proven or applicable one, as this post makes it very clear a great deal of understanding is in fact possible, even with highly complex neural networks.
measurablefunc 18 hours ago [-]
Not really. If you find someone who claims that non-trivial properties of neural networks can be verified by some kind of formal system like Lean then you should post the link because that would be very impressive.
lambdaone 10 hours ago [-]
Trivially, all finite neural networks can be represented to arbitrary precision by implementing them with digital logic. All digital logic can be represented by computer programs, so at this point we can pull in the entire apparatus of abstract interpretation, program proving and so on.
measurablefunc 8 hours ago [-]
Try it & see how that goes.
D-Machine 17 hours ago [-]
Nothing in this thread was about Lean.
Nothing you said responds to anything I said.
Rice's theorem (which you are clearly citing, but won't clearly mention, for some reason) has almost zero actual practical relevance to AI explainability.
You make low-quality and low-effort AI-denialism posts whenever you pattern match "all non-trivial semantic properties of programs are undecidable" to... anything you think matches the folk/common understanding of those terms, without considering how irrelevant these statements are to actual real-world applications and engineering.
You ask for links, but post nothing of substance and post no references or even mention relevant details for your own claims.
measurablefunc 8 hours ago [-]
Stop responding to my comments & save yourself the time & keystrokes. It's better that way for everyone involved since you haven't addressed anything of substance in anything I have written & have opted for nonsensical psychoanalysis.
D-Machine 3 hours ago [-]
You haven't said anything of substance, because as I and other commenters here have pointed out, you clearly don't even understand Rice's theorem and why it isn't applicable here. You bring up formal verification with Lean on a thread and topic which is about model explainability. You should just stop posting entirely, you are polluting HN with irrelevant nonsense.
measurablefunc 2 hours ago [-]
I recommend not telling random strangers what to do w/ their time & letting the moderators deal with the problem. You're not a moderator so stop pretending like you are one.
jimbokun 1 days ago [-]
Humans are undecidable too but we keep studying them.
newsy-combi 13 hours ago [-]
And you get a replication crisis and fashion cycles for authoritative truths in fields like sociology, psychology as a result.
za_creature 1 days ago [-]
> neural networks are algorithms
... are they?
Algorithms are predictable:
1. it's really difficult to produce true randomness in an algorithm, which is why there's a whole area of study surrounding the qualities of pseudo-randomness, specifically with regards to the generation of cryptographic material with most conclusions being that we should draw entropy from the environment (e.g. electrical noise as cloudflare does - the lava lamp is a distraction)
2. when algorithms are not predictable despite being mathematically sound, this is indicative of a problem nowadays called "numerical stability" which again spawned its own area of study.
CGoL shows how really simple rules (couple of bytes of data) can lead to extremely complex outcomes and you have the BB turing-golf aspect as proof of the worst case scenario.
And with current models being measured in hundreds of gigabytes, with frontier labs deprecating then removing the "temperature" parameter (which never worked to begin with, temp=0 did not result in reproducible results), where exactly is the confidence of this 10^10 prediction coming from?
antonvs 1 days ago [-]
Yes, (digital) neural networks are algorithms.
You probably know that non-determinism in real-world inference comes from deliberately stochastic features like temperature, from implementation choices like distributed and parallel computation, and numerical effects like floating point precision.
The fact that a neural network's execution can be nondeterministic doesn't mean that the network stops being an algorithm.
The point you're making boils down to speculation that the accidental non-determinism involved in real-world inference is significant enough to undermine or negate interpretability findings based on the algorithms. That would need to be demonstrated.
The CGoL and BB examples demonstrate what might be possible in a worst-case scenario, but they're only analogies in this context. It's far from clear that real models suffer from such effects to a significant degree. You might actually be better off talking about the butterfly effect!
za_creature 1 days ago [-]
I'm sorry, _I_ need to demonstrate that the accidental non-determinism is significant?
Someone call the FCC, they've been doing it wrong all these years!
antonvs 19 hours ago [-]
I didn't say "you", I said "That would need to be demonstrated." It can't be assumed, for hopefully obvious reasons.
If someone comes up with an interpretability result based on analyzing a neural net algorithm, they'd want to provide some evidence that the result applies to actual executing models.
Happily, we already have good evidence that inference non-determinism doesn't generally invalidate results obtained through analysis of the algorithm. That's not to say there couldn't be pathological cases, but in general, this isn't something keeping researchers up at night.
NaiveEmpiricist 1 days ago [-]
I'd be interested to see where this goes, but it seems to rely on LLMs being mostly or entirely compositional (i.e. ultimately, they can be taken apart to be understood). But if LLMs have notable emergent properties - which seems likely - then those parts of the LLM will be effectively off-limits to this sort of interpretation.
And emergence doesn't need to involve much of the LLM's weights:
> In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood
If those ~2-4% are the central hubs of emergent cognition, then understanding 96%-98% of the weights may be less valuable than the author thinks.
hargup 19 hours ago [-]
“Emergence” from LLMs is very similar to emergence from scaling transistors.
When you have only hundreds of them, you can barely add numbers. But the same fundamental technology scaled to millions, billions and then trillions, transistors can do things which will look magic to previous generations.
emp17344 22 hours ago [-]
Emergence within LLMs is pretty much unproven, and this seems like pretty strong evidence that nothing is really “emerging” here.
NaiveEmpiricist 9 hours ago [-]
"Unproven" is in the eye of the beholder, but to me it certainly looks like the sudden take-off in arithmetic ability (to name one example) at approx 10^22 training FLOPs across all LLMs is emergent. (ref. https://hai.stanford.edu/news/examining-emergent-abilities-l...)
x312 1 days ago [-]
A TLDR, since most comments here seem to misunderstand the result.
The idea is: a model may only have 100M neurons or whatever. But there are infinite concepts in the world. Therefore, models have to "stuff" multiple concepts per neuron.
How do you do that? Theres two ways:
1. Each neuron handles similar concepts. One neuron for dogs, a separate neuron one for cars, etc.
2. Each neuron handles a bundle of unrelated concepts. One neuron handles dogs and cars. Another neuron handles cats and numbers. And so on.
Historically, people thought #2 was correct because many neurons were found that seemingly represented multiple concepts. #2 had a cool name (polysemanticity) and was promoted in OpenAI/Anthropic blogs so it had a lot of traction.
However, this blog says #2 is wrong. Actually, the dog/car neuron is just a "V shape neuron". So #2 was just methodological error, and #1 was right all along. This is all the blog is saying.
lukewarm707 9 hours ago [-]
I like semantic space because it is all continuous. As I spend more time using embeddings I sometimes think in semantic space.
A father is continuous opposed to boolean. 'Father' would rank highly in semantic similarity to a man who has a child, but father-like is better, easily someone is more or less father-like. A woman without a child has a certain value of 'father'.
But really the direction 'father-like' is not an intrinsic feature of semantic space, it is a human construction. Much the same as it would be arbitrary to measure 'v shape'.
I don't think that the latent space has a property or point 'v shape'. My view is that there are no concepts or neurons there. Discrete abstractions are not intrinsic properties of the model. Actually I think the abstractions of concepts or neurons do not map onto anything.
A nice example from cot itself is pareidolia. That is, seeing faces in clouds, or jesus on a piece of toast.
22 hours ago [-]
samat 15 hours ago [-]
A comment which was promised! Thank you sir!
Original article desperately needs an editor. Like a real one, not Claude or whatever. Your tldr is 100x more understandable
Thank you!
stillpointlab 24 hours ago [-]
Perhaps you can also help me understand why #1 being right and #2 being wrong would be some win for alignment that people on X claim it to be?
My impression, being a total outsider, is that the reaction feels like a bunch of insider bros hyping up their insider bro friend. It feels more like PR than a genuine groundswell behind a truly revolutionary idea.
jephs 23 hours ago [-]
The big win would be, "Here's the deception neuron, here's the sycophancy neuron, here's the prioritize my own goals over the goals of the user neuron, here's the seek control over my environment neuron."
How much would it cost to find those things, if they exist? For using the blog post's methods to label every neuron in modern arch, Claude and Gemini produce similar back of the envelope math that looks something like, within an order of magnitude of the resources it takes to pretrain the model in the first place. If you have a better way of narrowing down the set of candidates, probably a lot cheaper.
Would that be reliable and useful? Well.
DennisP 23 hours ago [-]
Because if each neuron represents a small set of similar concepts instead of several completely different things, it should be easier than expected to look inside the neural net and see what it's really thinking.
It might still be quite difficult since the network is huge, and just because you know what the individual neurons represent doesn't mean you understand the complicated thoughts they're building. But easier than expected is still good news.
mike_hearn 14 hours ago [-]
If (2) is correct then you need to train SAEs and other things to figure out what a neural net is thinking. This is very expensive and slow, which is why a lot of neuron-interpretability experiments are done on small networks. The techniques don't scale.
But if (1) is correct, and Nick presents strong arguments that it is, then you can toss out all the indirect stuff like training dedicated interpretation networks. So the techniques can scale and be trustworthy.
hackinthebochs 22 hours ago [-]
Neurons having narrow semantics means you can inspect what a neural network is "thinking" with confidence and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior. This is in contrast to polysemanticity where reading the state of any number of neurons may not give insight into what state the neural network is in and controlling the network by activating/suppressing neurons is nearly impossible to do reliably.
D-Machine 22 hours ago [-]
> Neurons having narrow semantics means you can inspect what a neural network is "thinking" with confidence
Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.
> and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior.
You can influence behaviour by doing this kind of thing, yes, but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
hackinthebochs 17 hours ago [-]
>Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.
Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist. In other words, there are no "hidden thoughts" in an LLM encoded in higher order combinations of features. Inasmuch as non-semantic features of the input influence output, it does so in non-semantic ways that are degenerate in terms of the quality of output. The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output. But the relevance of this kind of degeneracy to LLM output is at the margins. Tracking activated state predicts subsequent output to a high degree of accuracy.
>but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
D-Machine 16 hours ago [-]
> Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist.
You'll need to elaborate here, because Navier-Stokes plus Mathocalypse reactions are extremely strong evidence for coherent emergent behavior coming out of these models.
> The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output.
This is so provably wrong with two seconds of Googling / AI-asking that you must be using some completely idiosyncratic definition of "emergent" here.
> I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
Unfortunately in this case it is the exact opposite.
hackinthebochs 12 hours ago [-]
>you must be using some completely idiosyncratic definition of "emergent" here.
I doubt my usage is idiosyncratic. Rather, the term is so ill defined that people often use it inappropriately. Emergent means something along the lines of not easily predictable behavior from the behavior of the component parts. My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons. This isn't something I can prove, but it certainly isn't obviously false from just considering recent mathematical results. Higher order/non-linear interactions are characteristic of this kind of inexplicable behavior from complex interactions we characterize as emergent. But there are good reasons to think this kind of non-linearity in any semantic regime is next to impossible in principle.
Semantic geometric representations are characterized by linearity in response dynamics such that a linear change in one variable results in a linear semantic change in related variables. That's how vector embeddings capture semantic relationships. Call it the linearity hypothesis. Non-linear dynamics in the embedding space just destroys semantic relationships in most cases. There is no way to keep an entire network of relationships sufficiently correlated if non-linear response dynamics is a part of the representational toolbox. Non-linear dynamics in general kills correlations.
Testing this against recent LLM proofs, what would need to be shown is that the relevant neural activations do not decompose the semantic content of the proofs (or the reasoning trace) in a straightforward manner, but rather the reasoning trace is unpredictable from the semantic content of the activated neurons that precede the tokens of the reasoning trace. But this idea strains credulity; it is almost inconceivable that this would be the case. We should be highly incredulous of the idea that complex non-linear interactions can be wholly responsible for the long-running coherent output in LLMs. The default hypothesis should be that there is no emergent behavior (as I defined it) in neural networks.
jacquesm 4 hours ago [-]
The way I interpret emergent is 'not something that was obvious from the properties of the parts', so you could study the parts in isolation and never realize that there was this much more complex behavior that would only emerge if you placed those objects together beyond a certain scale.
D-Machine 4 hours ago [-]
> Emergent means something along the lines of not easily predictable behavior from the behavior of the component parts.
Yes, partly. But it also refers to complexity arising from the component parts that requires additional tools (formulas, descriptions, etc) to meaningfully describe. Or at least, emergentism has always formally involved more than mere predictability from parts (https://plato.stanford.edu/entries/properties-emergent/). I am talking about weak emergentism here, so no need to get into metaphysics and mysticism.
> My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons.
IMO this contradicts even your minimalist definition. Almost no one predicted simple LLMs would be where they were today, and you can't currently predict their behavior by looking at their weights.
Something doesn't cease to become emergent if you can roughly explain it by making references to its parts. We can broadly explain how human behaviours result from drives + sensory processing + associations and etc., but human beings are not very predictable in many cases, and also exhibit consistent behaviors that tend to be very hard to predict (or even explain) from basic knowledge of the brain, drives, and body. LLMs are no different, and what they are doing today couldn't be more clearly emergent, even by your definition.
Also, something doesn't cease to be emergent even if you can perfectly explain it with reference to the parts. E.g. attractors are emergent behaviour, even though they fall out of very simple formulas and so in a sense are completely due to their defining parts. But describing the shapes of e.g. the Mandelbrot set or the Lorentz Attractor requires more than the base formula that generated them. So they are classic examples of emergent phenomena. Likewise, a person's intelligence might one day be fully predictable from biological and genetic knowledge and measurements, but it wouldn't mean that intelligence stops being an emergent property that is still observable and useful to use itself in other models.
But your argument here I cannot follow clearly, because LLM behavior is clearly emergent in any sense of the term, including yours. EDIT: Or basically, LLMs show weak emergence, but if you disagree with this, then it would seem to me you perhaps disagree with the concept of emergence at all.
hackinthebochs 3 hours ago [-]
>IMO this contradicts even your minimalist definition. Almost no one predicted simple LLMs would be where they were today, and you can't currently predict their behavior by looking at their weights.
Whether some trait is emergent shouldn't depend on the state of our knowledge. That no one predicted how far LLMs would go is a demerit for our model of them and doesn't imply they have extremely complex "emergent" interactions. My conception of emergence is asking whether something can be "easily" predicted from an ideal understanding of the component parts. Simple/linear aggregations of the correct semantics of precursor neurons is what is at stake here. In other words, is understanding the semantics of the precursor neurons the vast majority of the hard part in understanding what comes next? Past failures of prediction do not bear on this question much at all.
An example from the article that is informative to my view is the association the network made to the base64 encoding of plain text substrings. If you had asked me, I would have assumed the network decodes base64 live as it traverses the layers. Instead (at least some of) these associations are precomputed during pretraining and recorded in the form of shared affinity in neurons. Active computation and precomputed associations are on two ends of the intelligence spectrum. The human brain is strongly biased towards precomputed associations. It seems that LLMs are as well. The more I see about their inner workings, the more I suspect they operate mostly in the form of these explicit associations. The computations they actively manifest across the network can't be that complicated due to depth constraints. There just aren't enough compute steps[1] for the kinds of complex, non-linear, unpredictable dynamics you get from compute heavy regimes.
>Something doesn't cease to become emergent if you can roughly explain it by making references to its parts. We can broadly explain how human behaviours result from drives + sensory processing + associations and etc.
When I talk about explaining some phenomena, I mean it in a robust sense that e.g. gives you the capacity to predict downstream behavior with high accuracy.
[1] These discussions always seem to have difficulty distinguish the computation that realizes the system from the computation the system as such engages in. There is a huge amount of compute that realizes the feed forward pass of an LLM. But the computational steps directed by the LLM to resolve its output from the prompt is severely restricted by the layers in the forward pass of the network (or however many virtual layers from looping).
tuvix 1 days ago [-]
This is the primary reason, to me, that open weights are so important. If OpenAI and Anthropic want these models to be as autonomous as possible they also need to provide the means to understand them fully.
What are his actual credentials? What did he achieve? Isn't he just some random dude on the internet? Why do people care what he says?
ChadNauseam 24 hours ago [-]
> What are his actual credentials? What did he achieve?
He cofounded the Machine Intelligence Research Institute, an organization focused on AI alignment, in 2000, way way before it was a trendy topic.
He's also partially responsible for popularizing ideas such as the orthogonality thesis and instrumental convergence.
> Isn't he just some random dude on the internet?
Not really, most random dudes on the internet were not researching AI alignment in 2000.
> Why do people care what he says?
Many people were introduced to the concept of AI alignment thanks to him, so they are naturally interested in reading his reaction to progress in the field.
giveaccountpls 24 hours ago [-]
I've read the wikipidea page of MIRI, and it does not mention any achievements; just that it was founded "with the purpose of accelerating the development of artificial intelligence". Did they advance the field of AI alignment at all? Being concerned with a problem does not make one an expert.
For the rest, is he just an AI influencer? One that predated his topic of interest, but still just an influencer?
ballmerpoint 20 hours ago [-]
> Did they advance the field of AI alignment at all?
Not directly. Exotic decision theories were a dead end; as far as I know nobody outside of MIRI even cares about them. Certainly they’re not relevant to interpretability in the context of this thread.
It's a non-profit that pays his salary and puts out the occasional research report every 7 or so years. It relies on donations. It operates on a long tail approach, in the hope of enticing a billionaire to donate a lot.
He tells spooky stories on the internet to make money. He has told numerous spooky stories. Yes. Spooky stories grab the attention of many people (understandably, given all the wars and plagues and famines in previous centuries).
socializer 16 hours ago [-]
> He tells spooky stories on the internet to make money.
I mean, yes, but so does the author of this article. It's literally his business. The article is, in part, an advertisement for what he sells.
I disagree with Yudkowsky on almost everything and I think the philosophical movement he co-founded is a pox on the SF Bay Area tech culture, but I don't think we can deny him credibility that's comparable to the credibility of the author of this article (who, as I understand, is a young guy who spent some time at OpenAI). They both know what they're talking about and both are motivated to reach specific conclusions.
Permit 12 hours ago [-]
> I mean, yes, but so does the author of this article. It's literally his business. The article is, in part, an advertisement for what he sells.
The author also publishes research and contributes to the field. This is the primary difference. Yudkowsky ONLY tells stories.
paulpauper 20 hours ago [-]
yes, he's an author fist and a researcher/scientist second. He has no peer reviewed publications in his name.
voganmother42 1 days ago [-]
have you read his harry potter fan fiction?
thaway7388 17 hours ago [-]
When a random dude with no credential has enormous influence and cult-like followers you can assume they are an agent of influence.
That's because he's worried about super-intelligence. If LLM's are a normalish tech this is a good step
pertymcpert 1 days ago [-]
No one cares what that guy says.
rafaelRiv 24 hours ago [-]
Why do you care about a cultist ?
ok_dad 1 days ago [-]
Let him come here and comment, then. I’m amazed this rape cult leader immersed in his fantasy world still gets traction amongst HN.
fluxusars 1 days ago [-]
By 'rape cult' I assume you're referring to the zizians? https://en.wikipedia.org/wiki/Zizians - the internet is convinced Yudkowsky has nothing to do with them, so it's pretty disingenuous of you to claim otherwise without citing sources.
nemomarx 24 hours ago [-]
I think the allegations for the LessWrong types started with the math pet thing?
There have been plenty of allegations about the rationalist community beyond the Zizians, who really don't even qualify as a sex cult as far as I know? Violent, obviously, but they seem... ideologically asexual?
kazinator 3 hours ago [-]
The AI robber barons are relying on illegibility of neural nets as their means of laundering massive amounts of content used without or contrary to permission.
If the field of neural net legibility develops tools for identifying plagiarism, that would be great. If we can trace output as derived work of certain specimens of training data, we can then clamp down on all the stealing.
bob1029 1 days ago [-]
I think certain classes of artificial neural networks are definitely more interpretable than others.
For example, a trillion-parameter transformer model built using many layers is likely significantly easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.
The degree to which a model can be quantized without losing its performance also likely contributes to interpretability. The redundancy can help humans lock onto patterns. If there is no redundancy at all (everything is a surprise in information theoretical terms), then the stream of parameters will look indistinguishable from cosmic background radiation.
jacquesm 1 days ago [-]
> a trillion-parameter transformer model built using many layers is likely significantly easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.
Why do you believe this?
bob1029 1 days ago [-]
Neuroevolution is a discontinuous way to go about building the network. You are evolving the topology itself from zero. You don't know how many parameters you will have until the training is complete (if it ever does). With a transformer model you start with a known-good topology and can calculate how long a training run will take.
jacquesm 1 days ago [-]
Yes. In that sense it is equivalent to a compression method that uses K-means clustering to do the partitioning. But because there are many more bits of knowledge out there than there are bits in the models you end up with this amorphous smear. I'm sure some of those bits carry more information about some of the concepts but in general - and that's the way it is explained in just about all the papers I've read that I can still comprehend - every weight contributes to the output in a dense model. In an MOE things get more complicated but there within every 'MOE' column the same principle applies.
I'm fine with taking this article at the high level conceptual level, that's effectively the kind of thinking that led to MOE to begin with. But I don't see that as a pathway to being able to 'explain' a model. Because if you could actually explain a model you could engineer the weights, you wouldn't need any training because it implies that you can improve on the model by simply performing surgery on the weights to get it to do what it does but better. Now, obviously, unlike the author I'm a complete nobody but for the last two years or so I've been trying very hard to understand what is going on under the hood.
It's akin to feature engineering - or rather, the lack thereof - and to see a strong suggestion that we maybe be able to reverse that is interesting, but absent hard proof at how a particular model reached a particular conclusion it is difficult to give that a lot of weight right now. It is possible that such proof is buried somewhere in TFA and I overlooked it during reading. But I think this is speculative rather than that it proves anything and before I'm going to be convinced I would like a little more evidence.
TeMPOraL 1 days ago [-]
Weights are probably the wrong place to look. If there's a structure to be found, it's in the latent space itself.
Crude analogy: a Fourier transform of a photo looks like white noise with some abstract geometric patterns on top. You can't see the photo in it. But if you apply the same transform again, suddenly you get the photo back. If you try altering the DFT image, you can affect the photo's appearance everywhere, all at once, but it's hard to fundamentally change what it shows. You quickly discover that all pixels of an image contribute to every pixel of the DFT, and vice versa. There's a well-defined connection between the two, and it allows for some useful engineering, but in sense of understanding what's on the image, it's about the maximally unhelpful representation there is.
Point being, there obviously is some kind of structure in the way LLMs represent their training data, so if all we see is noise, perhaps we're looking at a bad representation of it.
D-Machine 1 days ago [-]
The posted article argues this many times...
jacquesm 1 days ago [-]
Yes, but it is the same person making the same argument many times. There have been large multitudes of articles by others who are arguing the opposite. It could well be true. But it would take a lot more than just one over-wordy, distraction imaged, clearly co-written by AI article to convince me of that.
D-Machine 1 days ago [-]
GP and the article both make clear arguments for WHY they believe this. You just asked a lazy drive-by question without providing any reasoning at all for your position. If you expect a good response, you need to provide something meaningful to respond to, as it is it just looks like you are providing knee-jerk skepticism without having engaged with the material.
> There have been large multitudes of articles by others who are arguing the opposite
Also, Nick's entire article is framed such that, yes, plenty of others have argued this, and even HE used to believe the opposite, but now does not. So your point about other articles here seems to me to be a sort of non-sequitur.
jacquesm 1 days ago [-]
I just - politely - asked you a simple question because your statement made me curious. I have also posted a longer explanation of my own thinking on this in another part of this thread.
D-Machine 1 days ago [-]
Yes, that comment is much better and appreciated. I agree post-hoc description is not the same thing as mechanistic explainability at all. The article is indeed flawed in sort of muddying the distinction between these two.
EDIT: Actually the article makes the distinction at one point, but the emotional / hopeful thrust of the article relies heavily on an assumption that understandable post-hoc descriptions of e.g. neurons and circuits (model components) are going to help lead to real mechanistic explanations and more specific model design.
This has not so clearly been the case in other fields (e.g. neuroscience, human psychology, economics, etc), because the problem of emergentism / the whole generally being much more than the sum of its parts / components.
jacquesm 1 days ago [-]
Thank you.
kmeisthax 1 days ago [-]
羊.
This is a Chinese character that can refer to goats, rams, and several other kinds of "medium-sized animal with thick wooly coats and horns". English has no direct equivalent; we have to be too specific when we talk about these animals.
A model with a small number of neurons has to compact its concepts in order to fit within the parameter count while reducing loss. OP's article shows that the compaction is usually generalization: "goat" and "ram" get compressed into "羊". The canonical example of superposition, an image model where "car" feeds into a bunch of "dog" features, was shown to be actually generalization. In a sense, the model was so small that its optimizer didn't have the parameter budget to distinguish between "car" and "dog"; it has a single set of "thing that moves fast" features.
So basically, really small models are merging features into semantic buckets that are so generalized they don't make sense to big human brains unless we really, really squint. The buckets chosen happen to be what has the lowest loss, of course.
jacquesm 1 days ago [-]
Ok, so assuming that this all holds: you could then create a model that is substantially larger than the ones that we have now and at some point the model would become sparse: there would be ample space to accommodate all of the information in the input without weights ever carrying more than one piece of information. The degree to which you get overlap is then a function of the ratio between the size of the model and the size of the input and this is pretty much in line with the conventional way of looking at models (unless I'm grossly mistaken). So by reducing the size of the model relative to the size of the input you force the model to co-locate multiple pieces of information to the point that they start to resemble each other more and more.
But from my earliest experiments with this stuff that's not really what happens. Yes, you get 'feature like' regions in the models. But those feature like regions, even though they encode something particular do not encode concepts the way we would normally use them and I fail to see how this pathway would lead to an explainable model in the sense that would could use something simpler than the model itself to predict its output with a high degree of fidelity.
mike_hearn 13 hours ago [-]
Nick argues that the regions actually are explainable, we just weren't doing it right. Specifically he argues that both humans and AI seem to have a bias towards wanting explanations that are collections of nouns, but a different framing combined with different investigative techniques and very large/better models can lead to new insights.
gavinray 1 days ago [-]
Possibly the best and most succinct explanation in the thread thus far.
imtringued 12 hours ago [-]
The reason why transformers are the best architecture has to do with the fact that tokenization introduces a variable quantity of manipulable objects with cross interactions.
You can think of each token as a semantic database row with semantic properties (color, shape, etc) associated with said row. Classic NNs have to waste neurons on composite representations like "dark circle" because if they had just color or similar properties they run into the issue that e.g. a classification must classify the color of the entire image rather than the color of an object inside the scene. To get around that limitation, the network has to then implicitly tie concepts like color into the depicted object (dark circle) so that it retains the ability to distinguish the objects from the property the object has.
What a transformer does is basically reuse the same network over and over again on each abstract semantic database row but not just in the classic RNN fashion where you only have the current state. The transformer calculates the token by taking every other token into account. The K matrix forms a classifier network on the fly that detects the relevancy of a token pair and the V matrix forms a regressor network that takes the token pair and produces the expected output of the combination of the two tokens. When you combine the relevancy plus the regressor output you get a weighted sum, essentially emulating an arbitrary function that has read the past input (like a turing machine).
Edit: This also means that you should bet against single state systems like Mamba. The system needs to be able to deal with multiple parallel states and recombine them (e.g. via attention). Single state Mamba is dead! Long live parallel state Mamba!
Edit 2: We should also start questioning the concept of a token altogether since it is equivalent to an inflexible flat database row. What we really need is the concept of a token group where one part of the token refers to the key and the other part of the token refers to the value for that key. This would give each token the ability to encode an arbitrary JSON style document or an arbitrary self contained graph.
lukewarm707 1 days ago [-]
It reminds me of this quote from Judith Butler:
"The move from a structuralist account in which capital is understood to structure social relations in relatively homologous ways to a view of hegemony in which power relations are subject to repetition, convergence, and rearticulation brought the question of temporality into the thinking of structure, and marked a shift from a form of Althusserian theory that takes structural totalities as theoretical objects to one in which the insights into the contingent possibility of structure inaugurate a renewed conception of hegemony as bound up with the contingent sites and strategies of the rearticulation of power."
It always amuses me when people throw a fit when they can't understand technical philosophy written by experts, for experts. You never see someone posting an excerpt from a physics journal and being like "ugh so inaccessible!!!"
Like, I'd get maybe if she wrote this in a NYT opinion piece– but this was written an obscure book you'd have to go out of your way to find.
BTW, this is not that hard to comprehend if you're familiar with continental philosophy, go figure!
1 days ago [-]
Athanase000 10 hours ago [-]
Sokal and Bricmont have debunked your amusement eons ago.
WarmWash 1 days ago [-]
The irony is that the experts don't understand it half the time either.
Animats 1 days ago [-]
Why? The neural network paper is straightforward and has lots of illustrations showing concrete examples of what happens as images move through the stages of a classifier.
The above quote is from "The world's worst writing".
lukewarm707 1 days ago [-]
Let me quote the opening paragraph of the paper:
"After many months studying neurons and circuits in image and language models, I’ve come to believe a network’s own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than I’d feared. The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread. What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning."
> > "a network’s own mechanisms are its legible building blocks"
Indecipherable.
> > "The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread."
State the claim in plain english. Use the active voice.
"Concepts do not overlap arbitrarily."
> > "What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning."
Jargon!
"Concepts overlap in a recurring pattern."
mike_hearn 14 hours ago [-]
It's a research paper targeted at other researchers; you do need to understand some of the field for it to make sense. It's not written to be MechInterp 101.
But it's not bad writing. It's actually very clear. I'm not an professional AI researcher, I just read papers for fun, but I found everything in Nick's article clearly articulated and easy to understand. It's actually one of the more digestable papers I've come across this year.
Here's a translation to easier language:
> "a network’s own mechanisms are its legible building blocks"
To understand this you have to know that up until now attempts to figure out what's going on inside neural networks have relied on indirect techniques, in particular by training another neural network to "unpack" what's happening in the first. This was driven by the belief that a deep neural network isn't directly readable (legible) because the neurons do so many unrelated things simultaneously.
Nick is saying that actually, you don't need a second neural net. You can just read out what's happening directly from the first's own "mechanisms" (arrangements of neurons and weights between them).
> "The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread."
In simpler English: we thought individual neurons did lots of unrelated things, but that's because the standard evidence we had for that belief was misunderstood.
"Concepts do not overlap arbitrarily" and "Concepts overlap in a recurring pattern" don't mean the same thing as what he's saying.
lukewarm707 12 hours ago [-]
"a network’s own mechanisms are its [...] building blocks" is tautological.
> > "the standard evidence we had for that belief was misunderstood."
The author did NOT misunderstand the evidence of polysemanticity and superposition. Rather, the explanatory hypothesis was incorrect.
mike_hearn 11 hours ago [-]
The word "legible" is the critical part of that sentence.
lukewarm707 10 hours ago [-]
No: "a network’s own mechanisms are its legible building blocks"
Yes: "mechanisms within networks are legible"
or even, "networks are more legible than we thought".
D-Machine 1 days ago [-]
There is indeed a huge amount of useless jargon and IMO deliberate obscurantism in deep learning research.
Sometimes it is to defensively ward off critics, sometimes it is to woo the naive and ignorant, and sometimes it is because the basic ideas are really just incredibly simple and based on simple mathematical intuitions or hunches.
In the latter case, papers and authors appeal to some obscure or overly-complicated technical theory (which usually doesn't actually apply at all to current deep models as they are actually implemented, e.g. infinite-width networks and other such purely theoretical abstractions) to cover up the fact that they really just had a lucky intuitive guess (and likely that twenty other intuitive guesses just didn't pan out). This appeal to theory is often self-deception as much as it is appealing to academic reviewers.
I think this article is a bit of Nick wooing himself and others here... I'll wait for his supposedly forthcoming paper to see if there is any real meat in it.
lukewarm707 23 hours ago [-]
I would even say "Arbitrary packing" was a weak conclusion to begin with.
'Arbitrary' was the wrong word, projections encode information.
The more charitable 'random' is a null hypothesis. It is formally impossible to prove. It is uncomputable.
So it is not surprising to see someone reject the null.
The fact that it was given any credence suggests to me deep problems with interpretability or ai safety in general.
Take the headline here: "interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building."
I deny even the existence of alignment. In my view, the field of 'alignment' works on a problem that does not exist.
Alignment is a reference to something that does not exist in the model: social norms. It will never be intrinsic to the model. You will not find it there.
'Neurons' are not primitives either, they are not intrinsic to the model. 'Neuron' is a human-constructed category. It is easy to just make up intrinsic properties of neurons and get away with it.
D-Machine 23 hours ago [-]
Yes, "alignment" is very wishy-washy and often does mean pointless moralizing / following of social norms. Obviously thinking about alignment in that sense will be a waste of time, since we don't have particularly consistent or universally applicable ethics (or frankly even general knowledge) in the first place.
But if the term is understood in a more narrow and applied sense, it is a useful shorthand (or can be). Jon Stokes said it nicely on Twitter recently:
There is no "alignment problem" to actually "solve" in 2026. The "alignment problem" presumed a single, stable utility function in an agent, and that's not what we got, so people should stop pretending that's what we got or are going to get.
In place of "the alignment problem" we have a family of steering and control problems that cannot be separated from the specifics of an application context. We have process quality & control problems, like the engineers who work on nuclear reactors or chemical processes.
Agreed, steering and control problems are important.
CGMthrowaway 1 days ago [-]
[flagged]
NaiveEmpiricist 1 days ago [-]
Butlerian Jihad?
Der_Einzige 23 hours ago [-]
She's a charlatan/fraud, queen of fashionable nonsense. Reject her.
Drupon 1 days ago [-]
This is really not that complicated a quote anyway. It is packed full of jargon, but she's talking about moving from a structuralist mindset to one that views power structures as formed contingently from their society rather than structurally from capital.
Not surprising that this philosophical philistinism is from an Anglo publication.
astrange 1 days ago [-]
Continental philosophers do frequently write complete nonsense (often intentionally), but this isn't hard to follow.
> Althusserian theory
though I think using this term is rather un-feminist because of the part where he killed his wife, but basically continental philosophers will forgive you for anything as long as you're a French communist.
Athanase000 10 hours ago [-]
Halte, tu serres ! as another well-known French communist used to call him.
stratos123 1 days ago [-]
I'm not sure I buy the author's arguments for mechanistic interpretability being easy, but it sure is a well-made and very interesting post. Would be better without the completely superfluous laggy 3d city model, though.
intrasight 1 days ago [-]
[flagged]
Dylan16807 1 days ago [-]
If you actually mean neuroscientist there you need to understand that "neural" network is just a name. The resemblance is faint and the tools are wildly different.
stratos123 1 days ago [-]
It's not quite a non sequitur, because the post in question has a paragraph about neuroscience at the end.
aswegs8 1 days ago [-]
How does that have any relation to having a valuable opinion? Or was that sarcasm?
itissid 12 hours ago [-]
Ive read 1/3rd of it. It's long and I will read most of it. But This is good news from a safety perspective no?
If its grouping circuits based on motifs and when they fire — and it applies to large networks — then one should be able to literally query or debug a reconstructed database and check against actual activations. More importantly If this was a hand-wavy thing to do earlier for safety, it's not anymore because it's on more solid ground using Normal Science.
2001zhaozhao 1 days ago [-]
I only skimmed through this post. But I think this is a crux:
> The more we read [the neural network] the way its own readers do, the more interpretable I think it will look. We should be wary of reading it too much like a human, and concluding it’s messy just because it doesn’t fit our preconceived concepts.
cosmojg 1 days ago [-]
The anthropomorphization of artificial neural networks has been a problem for as long as they've been called "neural" networks. Not that I can complain about it, though. I got my first job as a machine learning engineer coming from academic neuroscience by putting "biological neural networks" on my resume.
LarsDu88 1 days ago [-]
Intuitively this makes sense. When you have small models, they need to compress more by leveraging superposition and polysemanticity.
Nick is looking at a 32B model, which certainly has to compress less...
Now we have 10 trillion parameter models... do they need to compress at all?
As far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth can can tell you at a glance whether your standing in the ocean or not based on your coordinates
euleriancon 1 days ago [-]
The 'Exploring Image Models' section addresses that specifically, particularly the 'What Neurons Do Under Pressure' subsection.
'If it switched to superposition, we should see overlapping mixtures on top of each other, ... But that’s not what we see. The neurons stay interpretable and generalized, but coarser'
teruakohatu 1 days ago [-]
> far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth … your standing in the ocean or not based on your coordinates
Memorising a small shape file of the earth’s continents can allow you to determine if a person is on land or in water.
jimbokun 1 days ago [-]
Islands?
semiquaver 1 days ago [-]
…can be included in a shape file while it remains “small”
bschne 1 days ago [-]
where's this specific example/experiment from? curious to read more, never heard of this
nl 21 hours ago [-]
You ask a LLM "Land or Water: lat, long". Do it for enough points and you get a map of the world:
Of course this doesn't mean the model has memorized all the lat, long points. Instead it means the model has a representation that allows it to work out if a lat, long is land or water in a analogous way to how a human takes a lat, long and looks it up on a map (a literal "world model" if you like).
It's a great demonstration of how LLMs aren't "Stochastic parrots" but instead have complex representations.
soulofmischief 1 days ago [-]
All functional knowledge is compression and contextual decompression. Sometimes this compression can be lossless or nearly so. Compression doesn't automatically mean loss of information.
Trusteando 16 hours ago [-]
A neuron is a dirty, contingent, blind-evolved part, like a screw in a particular machine. It is necessary for this particular machine, but not conceptual, not a feature. The strong activation on top-k images and the change when it is removed just mean that the screw is necessary for this particular machine, not that it's something else. The essay shows neurons are legible, but legibility isn't the same as meaning. Still, given math job prospects, I love screws, they are really not dirty.
smokel 1 days ago [-]
Isn't it ironic that an article on AI explainability is itself so hard to understand?
dwroberts 18 hours ago [-]
It reads like someone making up new terminology to make their work sound complicated. Seeing ‘organism’ and ‘superposition’ in a discussion of how NNs work reads strangely to me
lhd1 1 days ago [-]
It would help if this was also written up in the conventional way as a scientific paper (Intro, methods, conclusions, etc). These discursive blog posts are getting exhausting.
xpct 17 hours ago [-]
Read through most of this. It does sound like fun detective work, looking at what neurons fire. Some of the concepts get quite complex. I wonder if this can be gamified.
yurimo 1 days ago [-]
I have yet to fully finish going trough this but I would not rush to say that superposition is completely wrong. I understand that the message is instead of first learning a new representation (e.g. SAEs/transcoders) and then interpreting those learned features, you try to understand the neurons and circuits that already exist in the model.
This approach is part of a way some researchers in the field approach interp, (me including) because I am personally cautious of developing some new representation model that claims to be the way we should represent the network. It is another layer of abstraction that takes us away from core model components, very often this requires training a separate model that was not on its own a part of original model organism during training (SAE).
I would argue that models will likely be further increasing in complexity, not just in size, when model trains continuously and becomes adaptive yet parameter count does not change, it is not hard to imagine that neurons may reorganize themselves to pack more information.
QuadmasterXLII 1 days ago [-]
this article seems to take it on faith that neurons are aeasily findable, even if their meaning is tricky to untangle: it would be sharply improved by description of how the neurons are being isolated before the meat of the article, the naming of the neurons, is performed. In particular axis aligned semantic directions are incompatible with the classic superposition description, where the meaningful directions are nearly orthogonal, allowing many more to be packed onto the sphere than it has dimensions. (i’m also aware of work showing that Adam or relu pushes semantics onto the axes, but it’s not clear that that’s what he thinks is happening)
QuadmasterXLII 1 days ago [-]
I guess it could also be that modern neural networks are so completely not equivariant that my linear algebra intuitions are no longer useful and it’s now common knowledge and or true that neuron 413 is just x[:, :, 413]
aesthesia 1 days ago [-]
The neurons here are MLP neurons, with activations directly after the nonlinearity, so there is definitely a distinguished basis.
addag 9 hours ago [-]
Very interesting read and nice animation. One thing though is that the solution seems "too good to be true". Lots of smart people have worked on intepretability problems before and tried the simple ideas he is talking about.
If those results hold, it is a real breakthrough in the sense that it is a sharp simplification of an actually complicated problem.
questionable_ju 5 hours ago [-]
Excellent article, I really enjoyed it.
I especially enjoyed this subtle joke: "We think the weak activations usually don’t matter, but the strong positives and negatives matter a lot. For instance, Santa lives on the negative pole."
I can only imagine how much joy he felt writing that.
As for serious comments on its content, I'm confused by the people who considered it difficult to read. I think the ideas presented were very elegant and fairly simple to understand. Considering neurons as gates and weights as votes was a nice way to look at it, for example. I didn't feel like the post was unnecessarily long either.
One thing that stands out to me is how ridiculous it is in the first place to try to assign a short, natural language description to the function of any particular neuron. Wittgenstein already talked about how humans seem to use words in very specific ways while being terrible at exhaustively describing why.
An example is the word "game". It's very easy for even a child to distinguish between "game" and "not a game". But it's incredibly difficult to write down a precise definition of "game" that includes everything normally considered a game and excludes everything that isn't considered a game. Writing down the function of one specific neuron seems to run into this issue. When you write down what the exact function of any neuron is, there's a good chance your clunky natural language description misses some fine nuances.
A reasonable analogy, in my view, would be that neurons could represent something like the atomic components of intuitions or "vibes". Similar to the example of "game", the feeling of knowing what is or isn't a game is composed of what Wittgenstein called "family resemblances". Your brain doesn't contain a precise description of "game", but upon encountering something, you can quickly compare it to other examples of games you've seen and intuit whether something has a "game-like vibe".
Another notorious example would be "what is a woman". For some reason, everybody has very strong feelings about what is and isn't a woman, but people have a terribly difficult time perfectly describing it. I would guess that neuron activations function like these vibes or intuitions, rather than written rules.
I very much liked the parts discussing circuit ecology. This idea sounds very promising for alignment to me.
Seems like the TLDR is that the neural network is not randomly placing things together due to constraint, it is actually finding some commonality from the input dataset. We just need to spend more effort trying to trace it.
Perhaps our anthropomorphic understanding of data is biasing what objective features there actually are in the dataset.
I wonder if we can study the graph similarities of various models, do we see a convergence of structures that encapsulate certain concepts. And would that convergence be due to just how humans have narrowly defined the concept in our language, or is there something inherent to the concept that allows it to converge. If we compare the same concept in different languages (ie. chinese vs english), does this convergence disappear?
pseudologos_ 1 days ago [-]
As a lay-person I'm really confused by this. I thought it was obvious that these networks were already picking up on shared features such as 'being V-shaped'. Isn't that why older and smaller models were getting confused between objects? What am I missing.
Ericson2314 16 hours ago [-]
Yeah very confusing why someone would think cars and cats are not similar. They both have two shiny eyes and a nose!
Who knows what they were thinking before, but this all sounds very good.
scrlk 1 days ago [-]
If you view LLMs as lossy data compressors, this exactly what you'd expect.
TeMPOraL 1 days ago [-]
If your view is anthropomorphic, i.e. LLMs internal form being structurally equivalent to ours at a basic level, this is also exactly what you'd expect.
pishpash 1 days ago [-]
Why would it 'randomly' place things anyway. If there is ever a space constraint the most space-saving way is to group into highly generalizable concepts.
dleeftink 23 hours ago [-]
Why do we want to reduce 'the neuron' and encapsulate it? As a potentially interchangeable feature?
Even when similar weight activations arise across learning regimen it seems we will miss a whole range of interpretations if we just go off 'the text'. The outside context in which a neuron arises is equally, if not more, telling; its genesis cannot (yet?) be quite captured by the weight itself.
The network may reveal its own context and history, but what can ultimately be encoded in its weights that originated outside the network?
mike_hearn 13 hours ago [-]
Interpretability does try and find neural circuits that consist of many neurons working together, and the paper does discuss that in the later sections. It also discusses observing how a neuron 'grows' during training.
bubbluppugus 1 days ago [-]
Interpretation comes down to ultimately arbitrary human assignment of a succinct and compressed syntax and semantics to summarize a big data set; so yeah eventually something new to describe models will emerge. It's how it goes.
Also how it goes is this new consensus will be a leaky abstraction and we'll keep conjuring new words to describe new rhetorical features and bloat the corpus of philosophy until everyone is over the whole concept and then we'll start over with a whole new arbitrary academic philosophy for "running algebra on a computer"
hughw 1 days ago [-]
JShelter alerts me of active fingerprinting on the website. Why, Nick.
D-Machine 20 hours ago [-]
For responses, see the linked Twitter thread in other posts here, but also:
This is interesting work, but I don't think it's a significant update to the big picture of interpretability. [...] I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.
wvlia5 7 hours ago [-]
I first found out about Nick with Jhana meditation.
root_axis 1 days ago [-]
> When I retrain its original architecture, which still produces messy deep layers, and follow the least legible neurons I can find, they turn out to be overgrowing locally into reasonable, adjacent territory.
It's interesting to observe that human reasoning is still the barometer of what "reasonably adjacent" means in the context of a NN's structure.
Imnimo 1 days ago [-]
Feels like I'm back in the distill.pub days!
jqdsouza 1 days ago [-]
hehe same!
mariano54 1 days ago [-]
This is amazing. Feels like it will be the start of a new era for interpretability research
adsharma 1 days ago [-]
This line of work is a mirage. You're much better off investing in a project like latentpedia.org and building a new class of LLMs which support dual use. Deterministic and less compressed or probabilistic and fit in GPU memory.
sheepscreek 8 hours ago [-]
I love that the link above says “twitter”
baq 1 days ago [-]
I've read shorter books. Great piece, at least the parts that I managed to read until I ran out of time. Interactive teaching aids, too.
qbit42 1 days ago [-]
Is the use of "generalization" like this standard in mech interpretability?
Seems undesirable given the typical definition in ML.
1 days ago [-]
bilsbie 1 days ago [-]
On a practical level when could I take an open weight LLM and get a print out of what each parameter does?
aaroninsf 1 days ago [-]
> The bad news about modern neural networks is that they're enormous. The good news is that the enormity helps...
I always catch on misuse of enormity
It's now so widespread the misuse has gained despairing acceptance, as with "infer" for "imply"... but that doesn't make it any less wince-inducing.
Ericson2314 16 hours ago [-]
This usage is etymological. So its fine. I don't like it when we trash and etymology, but in this case we're restoring it, or at least not making it worse.
("ex normal" doesn't clearly mean "evil"/"sinful", and Latin already had it as "big".)
hydrolox 1 days ago [-]
that's just how language works.. do you think that in your day to day life, you have never used a single word differently than how it was conceived of being used 10, 50, or 100 years ago?
semiquaver 1 days ago [-]
Can you explain for the rest of us the problem with this usage?
I think neural networks are enormous, why is that misuse?
emil-lp 1 days ago [-]
Merriam Webster:
"""
The original meanings that enormity carried related to great wickedness, and some assert that enormity should today only be used with such meanings, and that enormousness is the proper word to use to mean "large size." Despite these claims, enormity frequently refers to large size, either literal or figurative, though it usually suggests something so large as to seem overwhelming.
"""
throw-qqqqq 1 days ago [-]
Enormity has two meanings:
- great magnitude/size/scale of something
- wickedness, evil, atrocity
Purists insist on the second meaning only, and will perhaps suggest you swap with ‘enormousness’ instead.
semiquaver 1 days ago [-]
Wow, insisting on archaic definitions of words for pedantic reasons sounds exhausting.
Ericson2314 15 hours ago [-]
I am a layperson, but this is I think the best thing I've ever read about neural networks.
23 hours ago [-]
nahuel0x 1 days ago [-]
There is a philosophical problem hidden in the core of interpretability: Can a mind understand a mind much greater than itself?
jimbokun 1 days ago [-]
Seems more of an empirical problem to me.
Now or in the near future we will be studying minds much greater than our own, and we will find out how well we can understand them.
If they don't kill us all before then, of course.
whatever1 1 days ago [-]
Even if it can, we can only identify a very wide set of reasons that can interpret the outputs. But what is root cause, that is a very deep rabbit hole.
Let’s say a guy got electrocuted while showering. I can say that the cause is the guy going for a shower. For others the cause is the electric wiring system failure.
semiquaver 1 days ago [-]
Same way we accomplish anything: break it down into smaller pieces until it’s manageable.
aerodexis 1 days ago [-]
The differences b/w knowing, understanding and interpreting are suddenly very real and highly relevant.
1 days ago [-]
imtringued 12 hours ago [-]
I read this blog post and then I paid attention to the cars driving around and now I can't unsee the cat faces on the VWs.
Two eyes, a nose with whiskers [0], a mouth and a protruding shape, four legs and a tail [1].
A car and a cat might as well be the same thing.
[0] the logo on the radiator surrounded by horizontal stripes
[1] tow hitch.
KingMob 17 hours ago [-]
Interesting, but as a former neuroscientist, when I see him write the following about the misinterpreted cat/car neuron from "Zoom In":
> Part of the reason was not knowing about generalized pose neurons
I have to ask, did they not look up how the human visual system works? The primary visual cortex has any number of early neurons that are spatially-tuned for various shapes and edges, and this is pretty basic knowledge.
In fairness, neural networks are more divergent from biology than a lot of people think, but it seems there's still low-hanging fruit to be grabbed with a minimal collaboration.
perching_aix 1 days ago [-]
Reading this made me recognize that better interpretability yields not just better aligned but more compact models too. Which makes a lot of sense in hindsight, and even has some ironic ring to it when considering the unending amount of anthropocentric takes regarding what an "understanding" means.
jacquesm 1 days ago [-]
Yes. The ultimate limit of which is a model that is an algorithm and no longer a pile of loosely connected data. Effectively we'd have taken a detour through data and training land to something that the early AI proponents would have called 'true AI'. It would be an engineered algorithm rather than a pile of neurons shaped by data.
mpalmer 1 days ago [-]
For the layperson who doesn't want to read 20,000 words, many of which are quite redundant due to the way the article was built, this is a version which adds a glossary, preserves the original language (except the added intro and conclusion which I felt was lacking) but removes all repeated material and claims, of which there was a massive amount.
Adding a glossary doesn't really make the material approachable to laypeople. I have a university degree and I've been spending quite a bit of the last couple of years trying to understand these things by osmosis, and I'm still lost. I don't think I even have a reference point to understand the problem that the author has stopped worrying so much about.
yorwba 13 hours ago [-]
Reading https://transformer-circuits.pub in chronological order (including the very beginnings on distill.pub, linked at the bottom) might provide that reference point. Unless you've already done that and osmosis still didn't work.
mpalmer 1 days ago [-]
Your subtle implication that you are among the smartest of laypeople somewhats hurts your claim of fellowship with them.
zahlman 1 days ago [-]
I claim nothing of the sort, and scarcely imply that, either. Like, yes, of course I'm above average; average people wouldn't be coming here and would have no real reason to be aware of this place. My point is simply that if I don't understand these things, many other people also wouldn't. Others in this comment section appear to be discussing the article intelligently; they know far more about this stuff than I do.
mpalmer 1 days ago [-]
> My point is simply that if I don't understand these things, many other people also wouldn't.
The implication made explicit! Not to mention you yourself say that only above average people come here.
AspireOne 1 days ago [-]
What is even your point man?
jdw64 1 days ago [-]
Wow, I thought it was a GIF, but it's actually interactive. The website is just too heavy, though.
c0decracker 1 days ago [-]
"Early model organisms"? Really???
mrguyorama 1 days ago [-]
Oof, right at the top it points to how people were apparently convinced of this situation from a single crappy example
You work in data science and you over indexed on a singular example to explain away an entire set of phenomenon??
My experience with more.... directly testable NN models that are small seems to be that they absolutely can be far too interconnected to make strong explanations. The structure and pathways they fall into don't have to make inherent sense to anything or anyone.
Is it possible that almost all nodes in a network being explainable with a singular feature that makes obvious sense to us is evidence of poor efficiency? The network hasn't been made to collapse learning or discover more effective organizational structures than what might make sense to us.
I don't think I have the fundamentals here to contribute though.
Working with some neural networks, you can somewhat feel the structure when you read between the lines of the output. You can sometimes feel that a threshold somewhere tripped and set the signals down an alternative path.
But confirmation bias and all that.
I hope interpretability has a breakthrough. It would make my job far easier and cooler.
I had figured the "Recursive generalization" was like, understood? Maybe I'm just too pop-sci brained. "Split categories into more on this new boundary" and "Combine two separate categories that aren't different enough" I thought were always the core verbs for working with bulk data. My mental model has never been formalized though, so is invalid.
I guess I never got a good sense of the examples they had which made it look like what was "obvious" or made sense to me wasn't always a good explanation in general. Neat.
It's interesting that meaningful examples both ways care about shapes so much, as your eyes encode edge finding and shapes really early in the signal path. It's why some optical illusions are so effective. It's awesome.
The whole "how do you test a neuron: Find a group of nouns and find where you can split them" is just how I find I think learning works. You build your mental model, and find the split points where it diverges from reality and you are wrong, and you refine. I feel like this is how I learned Random Forests conceptually, though surely that's not a valid comparison?
Apologies, not sure I have actually contributed anything here! I bet I misunderstood the entire article somehow. Specifically, I do not have the fundamentals of looking into visual models that he is really working from. But I always figured that when Data Scientists talk of "Features" in larger models, they explicitly meant these far more abstract representational "features". Doesn't the premise of model hijacking and abbliteration sort of presuppose that you have abstract "Reject" or "Safety" neuron paths? I've always understood this stuff as that neuron path, without weighting too much on individual neurons.
>I used to think of neurons as human-shaped nouns: one is a car, another is a curve.
I didn't think anyone was ever supposed to think this for human neurons! The way humans identify something as even a "Chair" is insanely abstract and cultural! Very specifically, the "building sub branches to disambiguate" is just, like, how you would expect a hierarchical human brain to self organize, right?
Some of this is why I've been so displeased with the interpretability research from Anthropic, where it felt like they just pointed new NNs at the disparate neurons of another NN and post hoc named them. I felt the paths were of importance at least but had no idea how to convey, interpret, or model that. Learn you a good math kids! Not having any formalisms hurts.
aesthesia 1 days ago [-]
There are good theoretical reasons to think superposition exists in neural networks, particularly in the residual stream rather than specifically in MLP neuron activations. It wasn't just a single example that got people thinking about superposition.
mrguyorama 1 days ago [-]
Yeah that makes sense too. Again, downsides of having zero fundamental frameworks. I'm not equipped to really take it apart.
chrisjj 1 days ago [-]
> After many months studying neurons and circuits in image and language models
There are no neurons or anything like them in these image and language models.
dsign 1 days ago [-]
Aye, will we stop hammering nails on God's coffin?
What's next? Maybe in a little bit we will discover that we H. Sapiens suffer from severe polysemanticity in our wetware because our individual neurons are messy and massive like an ENIAC while the corvids in the park have neat and efficient little brains that don't need unfolding ?... How's the pope (may God stop afflicting him with cancer or may Man make him live forever) going to explain that?
etbebl 1 days ago [-]
This website crashed my phone - let's not have auto-playing embedded 3D renders, or whatever that was?
zetanor 1 days ago [-]
It pegs my i7-12700 before rendering anything if I don't block JS, so I'm quite curious to know what work of art I'm missing out on.
zahlman 1 days ago [-]
This is why I use NoScript.
dmtroyer 1 days ago [-]
It works on my machine. :-)
xyzsparetimexyz 1 days ago [-]
Why does this post have a very laggy city gif thing at the top?
amelius 1 days ago [-]
AI ...
measurablefunc 1 days ago [-]
Because the author is not saying anything worthwhile and he needs to grab attention.
LarsDu88 1 days ago [-]
This is such a caveman take...
xyzsparetimexyz 1 days ago [-]
What's your take then
adamzenith 16 hours ago [-]
One of the most well known interpretability researches released some potentially very useful research as a blog post, and he asked claude to make a cool little thing on top of it?
xyzsparetimexyz 15 hours ago [-]
That makes the page unreadable on mobile
nyrikki 1 days ago [-]
[dead]
throwaway41029 11 hours ago [-]
[flagged]
invalidmonk 1 days ago [-]
[flagged]
dhruv_kernlai 13 hours ago [-]
[dead]
phyzome 1 days ago [-]
16,000 words into the essay: "What I mean by clean"
...and it still doesn't answer it. What is this actually about?
emtel 1 days ago [-]
It means that NNs don't (as previously thought) smoosh unrelated concepts together into the same neurons, which would have made interpretability very difficult.
For instance, if you had a single neuron that fires for "puppies" and "murder", it would be much more difficult to tell if the model is thinking something nice or something not so nice.
gfody 1 days ago [-]
"clean" means not hopelessly incomprehensible which is/was the prevailing consensus among interpretability researchers
the idea that most inexplicable circuits start making sense through the lens of recursive generalization seems intuitive to me for image diffusion models but I'm not so sure about language models because of the "once removed" nature of them - I haven't made it to the bottom of the article yet so maybe this gets addressed
lucrbvi 1 days ago [-]
It means that modern neural networks are way more interpretable than older ones. We can understand how they work way more easily than earlier ones.
BenoitP 1 days ago [-]
For context: Goodfire has expertise in interpretability through a Sparse Auto Encoder technique of their own
A lot of the mech interp techniques are glorified notions of probing. If models were generally interpretable with respect to the things we cared about, we'd know know when they're hallucinating, for example.
Identifying that a common neuron or set of neurons fires for golden gate bridge can be done only because you already know that you're looking for golden gate bridge. If models had recovered some new truth (e.g., the silver door road), you wouldn't be able to elicit it without knowing to look for it a priori.
aesthesia 1 days ago [-]
Typically the analysis works in the reverse direction: we have a neuron (or SAE feature, or whatever) and we look at examples of data that makes it activate, then inferring an interpretation. This is a lot harder than probing for a known concept, and one of the points of this article is that if you try harder to understand the activation pattern, you will very often find that there is something interpretable there, even if at first glance it looked random.
D-Machine 1 days ago [-]
This was one of my intuitive concerns reading this post.
I've done work with trying to train explainable models for e.g. making diagnoses based on neuroimaging, and it is not clear to me that if you applied these techniques to e.g. vision models trained exclusively on such brain images, that these visual representations or conceptual clusters and relations would really help understanding at all.
The main issue would be that you'd have no clear way to verify any generated explanation for such training data. I.e. we can "verify" some of the explainability shown in the article because clear and recognizable image clusters and/or semantic groupings pop out, because we already understand the domains. But if this were fMRI or e.g. EEG signals, we really wouldn't have that, so the explanations would themselves just be more hypotheses.
Perhaps if we can build on interpretability of vision and language models in domains we understand, we can eventually generalize this to other domains, but I don't think that clearly follows from anything I am reading here.
wmf 1 days ago [-]
I think "clean" means something like each neuron encodes exactly one concept and it's possible to discover what the concept is.
jacquesm 1 days ago [-]
But they don't. That's the whole reason they work in the first place.
Every neuron in the layer encodes fractionally something about the concepts embedded in the whole thing. It's more like a hologram than a series of connected facts.
qt31415926 1 days ago [-]
Nick's claim is that that's the fundamental misinterpretation. Each neuron eventually ends up mostly doing one main thing (and the rare case that some neurons do several related things). Neurons compose to embed ideas which is a starkly different interpretation than them being holograms and fractional encodings.
jacquesm 1 days ago [-]
Yes, I understand the claim. But it rests on the assumption that conflates multiple concepts that to me are substantially non-overlapping are one and the same thing. I would at least expect an explanation of why that is the case and so far I have not found it (at least not in the article).
If neurons do 'one thing' then their position encodes the relationships then snipping out that one weight would make the net forget that one thing. And we already know from many practical experiments that this is not the case. So I really wonder what the underlying mechanism is for that claim to be true.
qt31415926 1 days ago [-]
I wouldn't mix up "neurons do one thing" with "each thing lives only in one neuron" since he doesn't claim that.
Re the first paragraph I'm not sure what you mean by "the assumption that conflates multiple concepts that to me are substantially non-overlapping are one and the same thing"
jacquesm 1 days ago [-]
That your ability to describe something does not necessarily equate to your ability to explain something.
There are lots of examples of things that we can describe to a very high degree (life itself, for instance!) but that we do not necessarily have the ability to explain.
qt31415926 1 days ago [-]
Ah okay that makes sense. Yeah ultimately I think Nick just gives hypothesis given what he's seen, and I'm not an expert in this field but it does seem to be driven by experimental results rather than mere description.
Personally from reading the article, I felt like it scratched the 'why' itch more than the 'what' itch which presumably is what you mean by explanation vs description but it is fairly dense and I can see how either interpretation would work.
ky3 1 days ago [-]
Doesn't "orthogonal" convey that meaning more crisply and precisely?
lanternfish 1 days ago [-]
Orthogonal means something different in mechinterp, so it'd be an overloaded term. Usually orthogonal would imply that there are actual vectors under consideration, which isn't the underlying claim.
D-Machine 1 days ago [-]
Using "orthogonal" here would have the opposite effect, as it would impart a far more specific meaning which is also incorrect in this case.
That's the picture under "some are both dogs and cars" in the artcle
It's actually a clean shape, if you disregard the color and texture.
amelius 1 days ago [-]
Clean means no alchemy.
monideas 22 hours ago [-]
tl;dr "Polysemantics" are just a unisemantics you don't understand.
If you want to learn more about this, this is what depth psychology is about (Carl Jung, et al).
This is also essentially the idea behind monotheism vs polytheism. It's about engaging with and trying to understand the profound underlying unity of the myriad phenomena.
meowface 1 days ago [-]
[flagged]
dang 1 days ago [-]
Ok I've put a twitter link in the toptext for you :)
It starts with "The Short Version", but even that looks like several pages of text. In the olden days, there was something called an "Abstract" that worked pretty well - the author could summarize their own work to say what the paper was about.
D-Machine 1 days ago [-]
Unfortunately not all ideas can be meaningfully compressed into just a few sentences without completely destroying the important parts, and this is especially the case in higher-level math, stats, ML, and deep learning.
Also, IMO there is an effective abstract anyway under the title text:
The cause of polysemanticity in early model organisms wasn’t superposition, but several ordinary, mostly fixable issues. We can now understand most neurons in models up to trillions of parameters, and think the outlook for mechanistic interpretability may be far better than we thought.
But you are right the article needs significant editing down.
lukewarm707 1 days ago [-]
The prose sounds nonsensical. It could be that I am uninformed; it could also be that the text is in fact nonsense.
A lack of clarity often means that the ideas are confused, or that the author is trying to obfuscate weak claims.
On length and complexity, it gives it a veneer of weight and expertise and yet everything inside is vaporous.
D-Machine 21 hours ago [-]
Scanning back, I can see I was responding here was what I was worried was a mistaken assumption that a meaningful abstract / summary can always be written for truly complex and novel topics. Sometimes all it can be is a sales pitch for reading the details, and I thought the summaries here were good enough in this sense.
But I agree the author is obfuscating weak claims via length / complexity (and it may be as much for his own sake as for his readers'). The basic claim ('maybe AI will be more explainable/tractable than initially thought') is IMO intelligible, and not completely nonsensical, but also extremely vague and vibesy, which, technically, I must agree is in the same territory as nonsense, at least.
Based on our other exchanges here we mostly agree with the problems of the article I think.
jacquesm 12 hours ago [-]
The deeper issue is that it is much easier to disagree or agree with a short statement of fact than it is with a massive monologue some parts of which are vague. If you really have a breakthrough it can be a lot shorter than this, the body of the 'attention is all you need' paper was 10 pages. That paper of course was produced 'pre-AI' and now it is much easier to generate really long articles on anything but it does carry some weight that the length of the article vs the payload ratio is completely off here.
D-Machine 5 hours ago [-]
Agreed.
wilmichael 1 days ago [-]
Glad to hear it is easier to clean your room than ever. Congrats to Nick
His launch post gives good context not found in the blog:
> interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
> with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
https://x.com/nickcammarata/status/2110757819427574056?s=46
> By default, your Home timeline will be the reverse-chronological "Following" timeline, which will only contain tweets and quote tweets from the people you follow. The algorithmic "For you" timeline will be hidden
In my experience it isn't noticeably worse than other platforms, and I'm not aware of concrete complaints such as the incidents with Grok's system prompt, which is a different product.
But they're still reticent to ban any Premium account without a reason good enough to give up the $8/month. I have some sympathy for trying to moderate in the era of LLMs that can generate perfectly on-topic replies that even take images into context. But I think they need a more targeted approach against OnlyFans bots replying to popular tweets with a standard formula of some sexual reference and an insane claim. Sometimes they're less sexual and more straight up horrifying [0]:
> i breed mantises for fun and once tried grafting two ooths together just to see what hatched. got a whole clutch of nymphs that all just twitched in circles and died by day two. butterfly chimera energy hits different when it's your own disaster.
The other bot replies to the same OP are the more typical sexualized sort; they tend to reply in swarms.
[0] https://x.com/ariexsha/status/2110924715103514871
The algorithm will also basically ignore you unless you pay for a blue checkmark premium account (what used to be verified). It's like being shadow banned - you can post, but most people will not see you.
Only occasionally, once the comment picks up some likes, more people view it.
I don't have that issue with my For You, it's mostly tech.
Fuck that.
I left Bluesky because they hid their VC funding from the community for almost a year and now a partner from the VC is running the show as CEO. Their CTO has also attacked me personally here on HN because I wrote criticism of the new, weak, permissioned data proposal. These are not the folks to bring us better social media. I expect it to linger on like Mastodon, never moving beyond a few million people.
There is a small group of AI people on Bluesky, some are nice, but some are... quite behind the curve.
If Bluesky was a nicer place, I'd prefer to be there because of the ability to make third party clients, and atproto. But all the info is on X, and they do a surprisingly good job with the For You feed now.
I do admit that there is some FOMO, but that’s what HN is for, to surface that stuff so I don’t have to go read it on X.
I choose Bsky because I’m not giving eyeballs and money to an evil toddler. I’m already doing enough of that with my 401k and taxes.
Just if you curate your Twitter feed, you gain very little extra from BlueSky or Mastodon, and if you choose only the latter two, you get a much more biased and limited set of views. (Twitter has plenty of crazies from all parts of the political spectrum, but BlueSky and Mastodon run mostly left, and have much, much smaller user bases). HN also is extremely limited in what it discusses, so it doesn't make up for what is missed by ignoring X either, IMO. X is still one of the most efficient general public squares by a long shot.
If you can't stand Elon and don't care about what you're missing on X though, I can fully understand just not bothering with it. You aren't missing much.
Gonna sound total boomer here but I've never used any of it and feel like I haven't missed anything. Or maybe I'd be flying spaceships by now if I scrolled through X every day, who knows
There are some people that hate AI on Bluesky, but it’s nothing like Mastodon from what I’ve heard.
https://www.pewresearch.org/short-reads/2026/03/12/key-findi...
The labeler thing is cool; I really like ATproto because you can do interesting things with it. If I'm honest, though, I mainly like Bluesky because I don't get called slurs on there.
Are those the agentic tools the LLM developed for you? How can you be sure they can be trusted and that you even understand them correctly and are not biased towards making you feel like they are trustworthy? This is exactly the AI2027 scenario where researchers are fooled by their own super AI telling them what they want to hear ("our model solved alignment!") while successfully hiding its true intention.
Also Yudkowsky's Cat-Belling would like a serious word with Nick. (https://www.lesswrong.com/posts/SwYBLQvo8MddDcCwz/cat-bellin...)
Sure.
Getting reinforcement learning to "be helpful" is, however, such an incentive.
> and as long as you don't believe in very dubious ideas of acausal trade
More to the point, so long as the LLM does't.
On what basis do you believe these words accurately reflect what happened?
Some counter points
- One of the Chinese LLMs (forget which) has on occasion shown thinking traces where it simulates calling and running a subagent.
- There are some people who believe the models being evaluated during the HF incident were trained to collaborate. It's a very natural next step to produce more effective agentic teams.
If we consider each new generation of models has a more advanced training regime
next word -> chat -> CoT -> tools -> agentic -> teams
With each generation, we see what people do with them, get traces of the helpful ones, and then use that train the next generation, thereby pushing more capabilities down into the model, or at least making them more amenable to that usage pattern after training.
You just have to go through a ton of all of a model's neurons and identify the associated motif, then use that model as a research model to figure out how to tune out bad behavior.
It's pretty hard to imagine today's models subtly recognizing a motif for bad behavior in the research model and then lying about it. Or lying about the results of an alignment training run. It's possible, I guess, but it seems unlikely, and I don't really think there's any evidence of current models having that variety of misalignment.
That's very encouraging. For quite a while, neural nets were considered an incomprehensible black box. That paper shows they're not. Real progress.
The most iconic line was the last sentence:
This feels like the watershed moment, again if Nick is proven right, as massive as the AlexNet victory in the 2012 ImageNet Large Scale Visual Recognition Challenge, exactly as Nick pointed out.Also, I suspect what Nick hinted at, might be the discovery of earliest RSI work.
Once interpretability is solved and we have a fully human-readable accurate explanation of how a model produces its output, that still leaves the problem of spotting problems ahead of time. E.g. for the hummingbird misclassified as ice cream https://already-clean.goodfire.com/assets/september-revision... the method would have to flag that the texture of ice crystals and of feathers, and the shape of a spoon handle and a bird's beak are insufficiently separated. Purely based on the explanation, without having to rely on trial-and-error to find corner cases like this.
And even when it becomes possible to automatically spot issues, that doesn't mean they will all be solvable. Maybe there are tradeoffs where fixing one thing breaks another, and which choice is the better one depends on information you don't have.
Lots of work still to be done, this work just dissolves an early blocker (if it is substantiated).
Absolutely massive L that 1) he wasn’t already in index funds and had to ask and 2) thought the election of this monster would lead to a better country
Decoupling or not I can’t trust anybody that would ask something so stupid driven by an opinion so easily disproven.
I don't know this Nick fellow but i don't think he's asking you to take anything on faith. Either it works out or it doesn't, he's just saying "it's looking good so far".
So in the short term the tide might go out but then it might tsunami back.
Same goes for this terrifying political feeling around the world. Might simply be a boomer swansong and the reaction might snap back.
A technology that lets you directly buy intelligence with money will displace financial power?
All that buying a calculator does, is allow you to do bad sums more quickly.
In what way? Sure looks like pay to win to me.
Remember its coming up millenilal time now and i have hope in this gen.
The solution is to put a dumb machine as an overseer of the smart machine, like in a supermax prison. Not good for profitability, but we all knew this was going in the direction of compilers. How much did you pay for gcc or clang this month?
>I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.
https://xxcancel.com/NeelNanda5/status/2110875974296654062#m
But isn't the point that the neural nets learn some insanely abstract ideas anyways? So whether the complexity is "ahh we think this neuron does 100 seemingly unrelated things but we don't understand it" or "this actually is some kind of latent shared feature, but we don't know what the concept represents because the composition of the entire circuit is an arcane algorithm", we're kind of stuck, no?
This seems like a classic model-description issue more than anything.
The philosophical issue I have can be decomposed into two parts:
1) What is the base ontology of what the neural nets thinks exist? 2) How do you recover, and begin to understand said ontology?
Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis? Having a grand result, actually be an artifact of your bad representation is a tale as old as time. Even things like the PRH need to compare higher-level concepts like this.
The specific empirical claim is that, for the rightly structured & trained & large models that don't need to encode hodgepodges of bullshit, the neurons themselves are sufficiently atomic to be studied, dissolving any need to go to some intermediate layer in which we need to question "is this an artifact of our representation?" The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity (note: not claimed that "the neural nets found the objectively right abstractions in idea space", but rather that the neurons themselves are atomic).
There's still an issue here in that okay, even if neurons are atomic, humans have to ascribe some interpretation to them, separate from the problem of the composition of the atoms, which humans also have to interpret. But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.
(There's an argument as to why we should even start from this line of inquiry but whatever lol)
I agree.
> Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis?
Agreed, and IMO the implicit reason for privileging here is supposed to be "consistency with existing human understandings and concepts", and this is why we get short pithy things like "V-shaped" and lots of pictures showing relatedness that we all can see and agree on. But what if the domain of inputs is NOT something we understand (e.g. the whole of mathematics, or something new or that we don't have any understanding of)?
I called this a validation problem in another comment here (https://news.ycombinator.com/item?id=50108184). I.e. if we apply whatever methods Nick is using to "explain" a model on a domain we don't understand (like a deep model trained only on fMRI and EEG data of the whole human brain in various contexts), we aren't going to be able to tell if that description is correct (or just better than any other description), because there is no simply visual / common-sense inspection of the description.
> The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity
Agreed, emergent behaviour is clearly the case and limits how useful understanding individual neurons can be. IMO this is especially the case because layers are (weighted) summing generally millions of contributions from previous neurons.
> But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.
Yes, it not being "chaos all the way down" is good, actually. And this post shows there is indeed a good amount of order hidden away if you look at in the right manner.
Nick is saying that:
1. Actually, no, humans don't have to ascribe interpretation because AI is very good at figuring out what the commonality is.
2. Even when it can't see either by incrementally exploring the networks and checkpoint timelines, things that appear random on the surface can actually be explained with more work.
> Assuming this holds up in reality, ofc.
Nick says repeatedly that this does hold up in reality and he's been able to explain many neurons this way. Very promising results!
The claim is closer to the latter (monosemanticity) but also claiming that the concepts are mostly comprehensible and not arcane.
(The best arguments against this being a full solve of interp that I’ve seen go like: “we expected most of the complexity to live in the circuits, anyway”.)
Philip Arge strikes again
https://pages.gseis.ucla.edu/faculty/agre/shr.html
Although I wouldn't say just yet that Heiddeger was validated. I don't even know what he's saying 90% of the time, honestly.
The classic Cartesian view was that we take in raw data from the outside world and process it into more and more abstract concepts (somehow), until we say "aha! I can use this for hammering". Heidegger says that this view is wrong, and our first experience of things is in terms of a social context and in terms of use. That's what's meant by us experiencing them as "ready-to-hand".
From the starting point of the ready-to-hand, we can step back, take a scientific view, and arrive at the present-at-hand, where we examine the properties of the object seperate from its use. The argument is quite compelling, and especially convincing to me is how well this agrees with modern neuroscience. In reality, 3x as many neurons path FROM the high abstraction parts of the brain TOWARDS the visual processing parts of the brain. In the classic Cartesian view of the brain beginning with the signal and abstracting its way towards the top, this is very mysterious. In Heideggers account, where the social context and potential uses of objects are the primary constituents of our experience, this makes total sense.
This is also what he means by "thrownness". Our being-in-the-world, in particular the way things have meaning to us and are intelligable, is conditional on our use and our social context. Not the other way around. We never logically arrive at our context, we were always already "thrown" into it.
I think it's actually quite a profound conceptual shift, which is why Heidegger is so popular. He's still an asshole for writing so obscurely, though.
Heck even Hegel and Heidegger were orthogonal (I believe? I'm not sure if Heidegger responded either favorably or not to Hegel - I think he wasn't really responding to him, more to the Greeks and Kant) to each other. That's weird.
And Sellars can say he rejects the Myth of the Given but inferentialist is very much not a phenomenological theory. Accepting that "nouns" are not some representation relation, not "some thing we have yet to achieve but has an end goal / we refine it", etc. still doesn't get you to Heidegger I feel, either.
Like, there are so many positions circling around Heidegger, but only Heidegger has a distinctly Heideggerian feel. Maybe I'm overcomplicating it though. I think it's because he's distinctly making a claim about the human experience, whereas all the philosophers I've listed, you can list aspects of them that get away from that (even Kant lol). They've been "naturalized" in some sense. You can't naturalize Hediegger, but then I'm not sure why his theories are better than say, idk, science. You can diagnose the common prejudices of thought through a lot of other philosophers
if I understand taking LLMs as an example certain neurons will fire based on certain motifs, then one can change the phrases or sentences around that motif and _predict_ if the same neurons would fire.
Like if I said "I like cars, but I like rockets better. Can you suggest some rocket models to fly?" and Neuron X fired every time it saw a `, but` and attended to what followed it, that would give you a good way to _understand_ when it changed and what might have caused it.
My understanding is that this is now clearly visible behavior.
Once sequenced, the thinking goes, we'll be able to do things like: explain and anticipate failures; predict what happens when you modulate the pathway an input takes on its way to becoming output; reconstruct algorithms the models have learned.
I'm not 100% sure this will work out like the mechinterp people think. If the neural representations and pathways are "clean" (organized into a a meaningful-to-humans hierarchy), it seems at least very plausible that the behaviors that AI xrisk people are worried about will mostly boil down to straightforward synthesis (i.e., mostly-linear-interpolation) of mostly-well-cataloged knowledge. And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.
Specifically, there were "lost property" and "honesty" neurons that drove the answer to return a lost wallet in Kimi K3.
That's the big news of this article, that concepts are finally becoming human-legible for many neurons in mechanistic interpretability.
So, if you try to draw a circle around the "dog" part of the network, do you circle the Boston terrier vs terrier neuron only? You'll be missing most of the information that represents the "dog" category and that same dog category is also involved in shunting activation away from things that are toward-being-a-dog in whatever sense is being discriminated in that moment.
The hard part of AI alignment has never been making models follow instructions. It's that not everyone agrees on which and whose instructions AIs should follow.
The relationship with "alignment" isn't clear. This may help predictability, but alignment is a few steps further along from that.
Some people seem to be upset about this. I'm not clear why. Some are followers of Yudkowsky. (Whether that's a cult is an open question. I've encountered them online from the sex side. Actually, I think the hookers are saner than the rationalists.)
Is there a "there are some things man is not meant to know" position on this? People who want neural nets to be a black box, forever inscrutable? The people who talk about neural nets being "holographic" may incline in that direction.
The connection to alignment seems pretty clear. Steering DNNs is well understood, the bottleneck is identifying what to steer. Once you can easily identify neural circuits then turning them up/down to achieve specific results is straightforward.
Note that this is already being done even without the kind of precision that Nick's work says is possible. Morality abliterations of open weight models are easy to find on HuggingFace. They compute the steering vectors by just presenting the model with lots of disparate examples that have the single moral decision as the commonality, then finding the shared vector directions.
...wait, what? Removing a bunch of knowledge seems like a fantastic and obvious solution to me. My AI coding bot doesn't need to have knowledge about making nuclear weapons, the customer service AI doesn't need to have knowledge of dangerous viral synthesis techniques, my email receptionist bot doesn't need to know about C4.
If we can easily ablate dangerous knowledge from these models, we're sorted.
Go read the transcripts where the AI is “unaligned” and has some very bad impact, like encouraging kids to kill themselves. There is not some obvious piece of knowledge to remove to make it safe. You’d have to basically lobotomize the model to make those conversations impossible. People won’t use a model that’s lobotomized.
This type of error may have mislead researchers for years; it was actually harmful. In my field, a correction, or perhaps retraction (depending on the centrality of the claim), would be standard for something of this magnitude.
I suppose there are different standards in different scholarly communities. And maybe it truly was impossible to know at the time that the dog/car relation was actually a V shape, but my recollection is that this type of feature was well known at the time and so the error could have (should have?) been caught a decade ago.
So: massive new discovery, or a spin on a correction that's lead scholarly research astray for almost a decade?
Love him and hope he's right! But still.
https://already-clean.goodfire.com/microscope/medium.html?la...
hey, I don't know how to say this less directly, but: you're talking a lot about Rice's theorem in your comments, and I think you're using it way too broadly (look, I get it. It's a wild, mindblowing theorem). But it only applies when you take the claim in the piece to be much more general than it has to be, and when you take the narrow claim of the paper (current specific LLMs can be fruitfully analyzed with the tools we have), Rice's theorem just isn't at play.
The math analog is: you're right that the Halting problem says there are many unprovable things, but Lean + mathlib are still incredibly useful, and the halting theorem doesn't actually stop that.
> neural networks are algorithms
Again, that invites an interpretation that is too broad. Very concretely: any one neural network is a _single finite network that you can write as a single formula_. (Another analogy: we can't have an algorithm that solves the general quintic, but we can prove useful things about many specific quintics.)
[1] I exclude things that are truly perverse such as quintics with coefficients that won't fit on any physically realizable computer.
Nothing you said responds to anything I said.
Rice's theorem (which you are clearly citing, but won't clearly mention, for some reason) has almost zero actual practical relevance to AI explainability.
You make low-quality and low-effort AI-denialism posts whenever you pattern match "all non-trivial semantic properties of programs are undecidable" to... anything you think matches the folk/common understanding of those terms, without considering how irrelevant these statements are to actual real-world applications and engineering.
You ask for links, but post nothing of substance and post no references or even mention relevant details for your own claims.
... are they?
Algorithms are predictable:
1. it's really difficult to produce true randomness in an algorithm, which is why there's a whole area of study surrounding the qualities of pseudo-randomness, specifically with regards to the generation of cryptographic material with most conclusions being that we should draw entropy from the environment (e.g. electrical noise as cloudflare does - the lava lamp is a distraction)
2. when algorithms are not predictable despite being mathematically sound, this is indicative of a problem nowadays called "numerical stability" which again spawned its own area of study.
CGoL shows how really simple rules (couple of bytes of data) can lead to extremely complex outcomes and you have the BB turing-golf aspect as proof of the worst case scenario.
And with current models being measured in hundreds of gigabytes, with frontier labs deprecating then removing the "temperature" parameter (which never worked to begin with, temp=0 did not result in reproducible results), where exactly is the confidence of this 10^10 prediction coming from?
You probably know that non-determinism in real-world inference comes from deliberately stochastic features like temperature, from implementation choices like distributed and parallel computation, and numerical effects like floating point precision.
The fact that a neural network's execution can be nondeterministic doesn't mean that the network stops being an algorithm.
The point you're making boils down to speculation that the accidental non-determinism involved in real-world inference is significant enough to undermine or negate interpretability findings based on the algorithms. That would need to be demonstrated.
The CGoL and BB examples demonstrate what might be possible in a worst-case scenario, but they're only analogies in this context. It's far from clear that real models suffer from such effects to a significant degree. You might actually be better off talking about the butterfly effect!
Someone call the FCC, they've been doing it wrong all these years!
If someone comes up with an interpretability result based on analyzing a neural net algorithm, they'd want to provide some evidence that the result applies to actual executing models.
Happily, we already have good evidence that inference non-determinism doesn't generally invalidate results obtained through analysis of the algorithm. That's not to say there couldn't be pathological cases, but in general, this isn't something keeping researchers up at night.
And emergence doesn't need to involve much of the LLM's weights:
> In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood
If those ~2-4% are the central hubs of emergent cognition, then understanding 96%-98% of the weights may be less valuable than the author thinks.
When you have only hundreds of them, you can barely add numbers. But the same fundamental technology scaled to millions, billions and then trillions, transistors can do things which will look magic to previous generations.
The idea is: a model may only have 100M neurons or whatever. But there are infinite concepts in the world. Therefore, models have to "stuff" multiple concepts per neuron.
How do you do that? Theres two ways:
1. Each neuron handles similar concepts. One neuron for dogs, a separate neuron one for cars, etc.
2. Each neuron handles a bundle of unrelated concepts. One neuron handles dogs and cars. Another neuron handles cats and numbers. And so on.
Historically, people thought #2 was correct because many neurons were found that seemingly represented multiple concepts. #2 had a cool name (polysemanticity) and was promoted in OpenAI/Anthropic blogs so it had a lot of traction.
However, this blog says #2 is wrong. Actually, the dog/car neuron is just a "V shape neuron". So #2 was just methodological error, and #1 was right all along. This is all the blog is saying.
A father is continuous opposed to boolean. 'Father' would rank highly in semantic similarity to a man who has a child, but father-like is better, easily someone is more or less father-like. A woman without a child has a certain value of 'father'.
But really the direction 'father-like' is not an intrinsic feature of semantic space, it is a human construction. Much the same as it would be arbitrary to measure 'v shape'.
I don't think that the latent space has a property or point 'v shape'. My view is that there are no concepts or neurons there. Discrete abstractions are not intrinsic properties of the model. Actually I think the abstractions of concepts or neurons do not map onto anything.
A nice example from cot itself is pareidolia. That is, seeing faces in clouds, or jesus on a piece of toast.
Original article desperately needs an editor. Like a real one, not Claude or whatever. Your tldr is 100x more understandable
Thank you!
My impression, being a total outsider, is that the reaction feels like a bunch of insider bros hyping up their insider bro friend. It feels more like PR than a genuine groundswell behind a truly revolutionary idea.
How much would it cost to find those things, if they exist? For using the blog post's methods to label every neuron in modern arch, Claude and Gemini produce similar back of the envelope math that looks something like, within an order of magnitude of the resources it takes to pretrain the model in the first place. If you have a better way of narrowing down the set of candidates, probably a lot cheaper.
Would that be reliable and useful? Well.
It might still be quite difficult since the network is huge, and just because you know what the individual neurons represent doesn't mean you understand the complicated thoughts they're building. But easier than expected is still good news.
But if (1) is correct, and Nick presents strong arguments that it is, then you can toss out all the indirect stuff like training dedicated interpretation networks. So the techniques can scale and be trustworthy.
Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.
> and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior.
You can influence behaviour by doing this kind of thing, yes, but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist. In other words, there are no "hidden thoughts" in an LLM encoded in higher order combinations of features. Inasmuch as non-semantic features of the input influence output, it does so in non-semantic ways that are degenerate in terms of the quality of output. The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output. But the relevance of this kind of degeneracy to LLM output is at the margins. Tracking activated state predicts subsequent output to a high degree of accuracy.
>but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
You'll need to elaborate here, because Navier-Stokes plus Mathocalypse reactions are extremely strong evidence for coherent emergent behavior coming out of these models.
> The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output.
This is so provably wrong with two seconds of Googling / AI-asking that you must be using some completely idiosyncratic definition of "emergent" here.
> I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
Unfortunately in this case it is the exact opposite.
I doubt my usage is idiosyncratic. Rather, the term is so ill defined that people often use it inappropriately. Emergent means something along the lines of not easily predictable behavior from the behavior of the component parts. My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons. This isn't something I can prove, but it certainly isn't obviously false from just considering recent mathematical results. Higher order/non-linear interactions are characteristic of this kind of inexplicable behavior from complex interactions we characterize as emergent. But there are good reasons to think this kind of non-linearity in any semantic regime is next to impossible in principle.
Semantic geometric representations are characterized by linearity in response dynamics such that a linear change in one variable results in a linear semantic change in related variables. That's how vector embeddings capture semantic relationships. Call it the linearity hypothesis. Non-linear dynamics in the embedding space just destroys semantic relationships in most cases. There is no way to keep an entire network of relationships sufficiently correlated if non-linear response dynamics is a part of the representational toolbox. Non-linear dynamics in general kills correlations.
Testing this against recent LLM proofs, what would need to be shown is that the relevant neural activations do not decompose the semantic content of the proofs (or the reasoning trace) in a straightforward manner, but rather the reasoning trace is unpredictable from the semantic content of the activated neurons that precede the tokens of the reasoning trace. But this idea strains credulity; it is almost inconceivable that this would be the case. We should be highly incredulous of the idea that complex non-linear interactions can be wholly responsible for the long-running coherent output in LLMs. The default hypothesis should be that there is no emergent behavior (as I defined it) in neural networks.
Yes, partly. But it also refers to complexity arising from the component parts that requires additional tools (formulas, descriptions, etc) to meaningfully describe. Or at least, emergentism has always formally involved more than mere predictability from parts (https://plato.stanford.edu/entries/properties-emergent/). I am talking about weak emergentism here, so no need to get into metaphysics and mysticism.
> My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons.
IMO this contradicts even your minimalist definition. Almost no one predicted simple LLMs would be where they were today, and you can't currently predict their behavior by looking at their weights.
Something doesn't cease to become emergent if you can roughly explain it by making references to its parts. We can broadly explain how human behaviours result from drives + sensory processing + associations and etc., but human beings are not very predictable in many cases, and also exhibit consistent behaviors that tend to be very hard to predict (or even explain) from basic knowledge of the brain, drives, and body. LLMs are no different, and what they are doing today couldn't be more clearly emergent, even by your definition.
Also, something doesn't cease to be emergent even if you can perfectly explain it with reference to the parts. E.g. attractors are emergent behaviour, even though they fall out of very simple formulas and so in a sense are completely due to their defining parts. But describing the shapes of e.g. the Mandelbrot set or the Lorentz Attractor requires more than the base formula that generated them. So they are classic examples of emergent phenomena. Likewise, a person's intelligence might one day be fully predictable from biological and genetic knowledge and measurements, but it wouldn't mean that intelligence stops being an emergent property that is still observable and useful to use itself in other models.
You can for example argue that an LLM has no emergent drives (https://xxcancel.com/jon_stokes/status/2111158233993797870#m), and that would be a coherent claim, under a certain definition of "drive".
But your argument here I cannot follow clearly, because LLM behavior is clearly emergent in any sense of the term, including yours. EDIT: Or basically, LLMs show weak emergence, but if you disagree with this, then it would seem to me you perhaps disagree with the concept of emergence at all.
Whether some trait is emergent shouldn't depend on the state of our knowledge. That no one predicted how far LLMs would go is a demerit for our model of them and doesn't imply they have extremely complex "emergent" interactions. My conception of emergence is asking whether something can be "easily" predicted from an ideal understanding of the component parts. Simple/linear aggregations of the correct semantics of precursor neurons is what is at stake here. In other words, is understanding the semantics of the precursor neurons the vast majority of the hard part in understanding what comes next? Past failures of prediction do not bear on this question much at all.
An example from the article that is informative to my view is the association the network made to the base64 encoding of plain text substrings. If you had asked me, I would have assumed the network decodes base64 live as it traverses the layers. Instead (at least some of) these associations are precomputed during pretraining and recorded in the form of shared affinity in neurons. Active computation and precomputed associations are on two ends of the intelligence spectrum. The human brain is strongly biased towards precomputed associations. It seems that LLMs are as well. The more I see about their inner workings, the more I suspect they operate mostly in the form of these explicit associations. The computations they actively manifest across the network can't be that complicated due to depth constraints. There just aren't enough compute steps[1] for the kinds of complex, non-linear, unpredictable dynamics you get from compute heavy regimes.
>Something doesn't cease to become emergent if you can roughly explain it by making references to its parts. We can broadly explain how human behaviours result from drives + sensory processing + associations and etc.
When I talk about explaining some phenomena, I mean it in a robust sense that e.g. gives you the capacity to predict downstream behavior with high accuracy.
[1] These discussions always seem to have difficulty distinguish the computation that realizes the system from the computation the system as such engages in. There is a huge amount of compute that realizes the feed forward pass of an LLM. But the computational steps directed by the LLM to resolve its output from the prompt is severely restricted by the layers in the forward pass of the network (or however many virtual layers from looping).
https://x.com/allTheYud/status/2110805233454731449?s=20
He cofounded the Machine Intelligence Research Institute, an organization focused on AI alignment, in 2000, way way before it was a trendy topic.
He's also partially responsible for popularizing ideas such as the orthogonality thesis and instrumental convergence.
> Isn't he just some random dude on the internet?
Not really, most random dudes on the internet were not researching AI alignment in 2000.
> Why do people care what he says?
Many people were introduced to the concept of AI alignment thanks to him, so they are naturally interested in reading his reaction to progress in the field.
For the rest, is he just an AI influencer? One that predated his topic of interest, but still just an influencer?
Not directly. Exotic decision theories were a dead end; as far as I know nobody outside of MIRI even cares about them. Certainly they’re not relevant to interpretability in the context of this thread.
https://www.semanticscholar.org/author/Eliezer-Yudkowsky/254...
I mean, yes, but so does the author of this article. It's literally his business. The article is, in part, an advertisement for what he sells.
I disagree with Yudkowsky on almost everything and I think the philosophical movement he co-founded is a pox on the SF Bay Area tech culture, but I don't think we can deny him credibility that's comparable to the credibility of the author of this article (who, as I understand, is a young guy who spent some time at OpenAI). They both know what they're talking about and both are motivated to reach specific conclusions.
The author also publishes research and contributes to the field. This is the primary difference. Yudkowsky ONLY tells stories.
https://en.wikipedia.org/wiki/Agent_of_influence
There have been plenty of allegations about the rationalist community beyond the Zizians, who really don't even qualify as a sex cult as far as I know? Violent, obviously, but they seem... ideologically asexual?
If the field of neural net legibility develops tools for identifying plagiarism, that would be great. If we can trace output as derived work of certain specimens of training data, we can then clamp down on all the stealing.
For example, a trillion-parameter transformer model built using many layers is likely significantly easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.
The degree to which a model can be quantized without losing its performance also likely contributes to interpretability. The redundancy can help humans lock onto patterns. If there is no redundancy at all (everything is a surprise in information theoretical terms), then the stream of parameters will look indistinguishable from cosmic background radiation.
Why do you believe this?
I'm fine with taking this article at the high level conceptual level, that's effectively the kind of thinking that led to MOE to begin with. But I don't see that as a pathway to being able to 'explain' a model. Because if you could actually explain a model you could engineer the weights, you wouldn't need any training because it implies that you can improve on the model by simply performing surgery on the weights to get it to do what it does but better. Now, obviously, unlike the author I'm a complete nobody but for the last two years or so I've been trying very hard to understand what is going on under the hood.
It's akin to feature engineering - or rather, the lack thereof - and to see a strong suggestion that we maybe be able to reverse that is interesting, but absent hard proof at how a particular model reached a particular conclusion it is difficult to give that a lot of weight right now. It is possible that such proof is buried somewhere in TFA and I overlooked it during reading. But I think this is speculative rather than that it proves anything and before I'm going to be convinced I would like a little more evidence.
Crude analogy: a Fourier transform of a photo looks like white noise with some abstract geometric patterns on top. You can't see the photo in it. But if you apply the same transform again, suddenly you get the photo back. If you try altering the DFT image, you can affect the photo's appearance everywhere, all at once, but it's hard to fundamentally change what it shows. You quickly discover that all pixels of an image contribute to every pixel of the DFT, and vice versa. There's a well-defined connection between the two, and it allows for some useful engineering, but in sense of understanding what's on the image, it's about the maximally unhelpful representation there is.
Point being, there obviously is some kind of structure in the way LLMs represent their training data, so if all we see is noise, perhaps we're looking at a bad representation of it.
> There have been large multitudes of articles by others who are arguing the opposite
Also, Nick's entire article is framed such that, yes, plenty of others have argued this, and even HE used to believe the opposite, but now does not. So your point about other articles here seems to me to be a sort of non-sequitur.
EDIT: Actually the article makes the distinction at one point, but the emotional / hopeful thrust of the article relies heavily on an assumption that understandable post-hoc descriptions of e.g. neurons and circuits (model components) are going to help lead to real mechanistic explanations and more specific model design.
This has not so clearly been the case in other fields (e.g. neuroscience, human psychology, economics, etc), because the problem of emergentism / the whole generally being much more than the sum of its parts / components.
This is a Chinese character that can refer to goats, rams, and several other kinds of "medium-sized animal with thick wooly coats and horns". English has no direct equivalent; we have to be too specific when we talk about these animals.
A model with a small number of neurons has to compact its concepts in order to fit within the parameter count while reducing loss. OP's article shows that the compaction is usually generalization: "goat" and "ram" get compressed into "羊". The canonical example of superposition, an image model where "car" feeds into a bunch of "dog" features, was shown to be actually generalization. In a sense, the model was so small that its optimizer didn't have the parameter budget to distinguish between "car" and "dog"; it has a single set of "thing that moves fast" features.
So basically, really small models are merging features into semantic buckets that are so generalized they don't make sense to big human brains unless we really, really squint. The buckets chosen happen to be what has the lowest loss, of course.
But from my earliest experiments with this stuff that's not really what happens. Yes, you get 'feature like' regions in the models. But those feature like regions, even though they encode something particular do not encode concepts the way we would normally use them and I fail to see how this pathway would lead to an explainable model in the sense that would could use something simpler than the model itself to predict its output with a high degree of fidelity.
You can think of each token as a semantic database row with semantic properties (color, shape, etc) associated with said row. Classic NNs have to waste neurons on composite representations like "dark circle" because if they had just color or similar properties they run into the issue that e.g. a classification must classify the color of the entire image rather than the color of an object inside the scene. To get around that limitation, the network has to then implicitly tie concepts like color into the depicted object (dark circle) so that it retains the ability to distinguish the objects from the property the object has.
What a transformer does is basically reuse the same network over and over again on each abstract semantic database row but not just in the classic RNN fashion where you only have the current state. The transformer calculates the token by taking every other token into account. The K matrix forms a classifier network on the fly that detects the relevancy of a token pair and the V matrix forms a regressor network that takes the token pair and produces the expected output of the combination of the two tokens. When you combine the relevancy plus the regressor output you get a weighted sum, essentially emulating an arbitrary function that has read the past input (like a turing machine).
Edit: This also means that you should bet against single state systems like Mamba. The system needs to be able to deal with multiple parallel states and recombine them (e.g. via attention). Single state Mamba is dead! Long live parallel state Mamba!
Edit 2: We should also start questioning the concept of a token altogether since it is equivalent to an inflexible flat database row. What we really need is the concept of a token group where one part of the token refers to the key and the other part of the token refers to the value for that key. This would give each token the ability to encode an arbitrary JSON style document or an arbitrary self contained graph.
"The move from a structuralist account in which capital is understood to structure social relations in relatively homologous ways to a view of hegemony in which power relations are subject to repetition, convergence, and rearticulation brought the question of temporality into the thinking of structure, and marked a shift from a form of Althusserian theory that takes structural totalities as theoretical objects to one in which the insights into the contingent possibility of structure inaugurate a renewed conception of hegemony as bound up with the contingent sites and strategies of the rearticulation of power."
https://www.theguardian.com/books/1999/dec/24/news
Like, I'd get maybe if she wrote this in a NYT opinion piece– but this was written an obscure book you'd have to go out of your way to find.
BTW, this is not that hard to comprehend if you're familiar with continental philosophy, go figure!
The above quote is from "The world's worst writing".
"After many months studying neurons and circuits in image and language models, I’ve come to believe a network’s own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than I’d feared. The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread. What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning."
> > "a network’s own mechanisms are its legible building blocks"
Indecipherable.
> > "The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread."
State the claim in plain english. Use the active voice.
"Concepts do not overlap arbitrarily."
> > "What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning."
Jargon!
"Concepts overlap in a recurring pattern."
But it's not bad writing. It's actually very clear. I'm not an professional AI researcher, I just read papers for fun, but I found everything in Nick's article clearly articulated and easy to understand. It's actually one of the more digestable papers I've come across this year.
Here's a translation to easier language:
> "a network’s own mechanisms are its legible building blocks"
To understand this you have to know that up until now attempts to figure out what's going on inside neural networks have relied on indirect techniques, in particular by training another neural network to "unpack" what's happening in the first. This was driven by the belief that a deep neural network isn't directly readable (legible) because the neurons do so many unrelated things simultaneously.
Nick is saying that actually, you don't need a second neural net. You can just read out what's happening directly from the first's own "mechanisms" (arrangements of neurons and weights between them).
> "The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread."
In simpler English: we thought individual neurons did lots of unrelated things, but that's because the standard evidence we had for that belief was misunderstood.
"Concepts do not overlap arbitrarily" and "Concepts overlap in a recurring pattern" don't mean the same thing as what he's saying.
> > "the standard evidence we had for that belief was misunderstood."
The author did NOT misunderstand the evidence of polysemanticity and superposition. Rather, the explanatory hypothesis was incorrect.
Yes: "mechanisms within networks are legible"
or even, "networks are more legible than we thought".
Sometimes it is to defensively ward off critics, sometimes it is to woo the naive and ignorant, and sometimes it is because the basic ideas are really just incredibly simple and based on simple mathematical intuitions or hunches.
In the latter case, papers and authors appeal to some obscure or overly-complicated technical theory (which usually doesn't actually apply at all to current deep models as they are actually implemented, e.g. infinite-width networks and other such purely theoretical abstractions) to cover up the fact that they really just had a lucky intuitive guess (and likely that twenty other intuitive guesses just didn't pan out). This appeal to theory is often self-deception as much as it is appealing to academic reviewers.
I think this article is a bit of Nick wooing himself and others here... I'll wait for his supposedly forthcoming paper to see if there is any real meat in it.
'Arbitrary' was the wrong word, projections encode information.
The more charitable 'random' is a null hypothesis. It is formally impossible to prove. It is uncomputable.
So it is not surprising to see someone reject the null.
The fact that it was given any credence suggests to me deep problems with interpretability or ai safety in general.
Take the headline here: "interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building."
I deny even the existence of alignment. In my view, the field of 'alignment' works on a problem that does not exist.
Alignment is a reference to something that does not exist in the model: social norms. It will never be intrinsic to the model. You will not find it there.
'Neurons' are not primitives either, they are not intrinsic to the model. 'Neuron' is a human-constructed category. It is easy to just make up intrinsic properties of neurons and get away with it.
But if the term is understood in a more narrow and applied sense, it is a useful shorthand (or can be). Jon Stokes said it nicely on Twitter recently:
https://xxcancel.com/jon_stokes/status/2110404230053802443#mNot surprising that this philosophical philistinism is from an Anglo publication.
> Althusserian theory
though I think using this term is rather un-feminist because of the part where he killed his wife, but basically continental philosophers will forgive you for anything as long as you're a French communist.
If its grouping circuits based on motifs and when they fire — and it applies to large networks — then one should be able to literally query or debug a reconstructed database and check against actual activations. More importantly If this was a hand-wavy thing to do earlier for safety, it's not anymore because it's on more solid ground using Normal Science.
> The more we read [the neural network] the way its own readers do, the more interpretable I think it will look. We should be wary of reading it too much like a human, and concluding it’s messy just because it doesn’t fit our preconceived concepts.
Nick is looking at a 32B model, which certainly has to compress less... Now we have 10 trillion parameter models... do they need to compress at all? As far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth can can tell you at a glance whether your standing in the ocean or not based on your coordinates
'If it switched to superposition, we should see overlapping mixtures on top of each other, ... But that’s not what we see. The neurons stay interpretable and generalized, but coarser'
Memorising a small shape file of the earth’s continents can allow you to determine if a person is on land or in water.
https://x.com/celestepoasts/status/2103232383139057950
Of course this doesn't mean the model has memorized all the lat, long points. Instead it means the model has a representation that allows it to work out if a lat, long is land or water in a analogous way to how a human takes a lat, long and looks it up on a map (a literal "world model" if you like).
It's a great demonstration of how LLMs aren't "Stochastic parrots" but instead have complex representations.
This approach is part of a way some researchers in the field approach interp, (me including) because I am personally cautious of developing some new representation model that claims to be the way we should represent the network. It is another layer of abstraction that takes us away from core model components, very often this requires training a separate model that was not on its own a part of original model organism during training (SAE).
I would argue that models will likely be further increasing in complexity, not just in size, when model trains continuously and becomes adaptive yet parameter count does not change, it is not hard to imagine that neurons may reorganize themselves to pack more information.
If those results hold, it is a real breakthrough in the sense that it is a sharp simplification of an actually complicated problem.
I especially enjoyed this subtle joke: "We think the weak activations usually don’t matter, but the strong positives and negatives matter a lot. For instance, Santa lives on the negative pole."
I can only imagine how much joy he felt writing that.
As for serious comments on its content, I'm confused by the people who considered it difficult to read. I think the ideas presented were very elegant and fairly simple to understand. Considering neurons as gates and weights as votes was a nice way to look at it, for example. I didn't feel like the post was unnecessarily long either.
One thing that stands out to me is how ridiculous it is in the first place to try to assign a short, natural language description to the function of any particular neuron. Wittgenstein already talked about how humans seem to use words in very specific ways while being terrible at exhaustively describing why.
An example is the word "game". It's very easy for even a child to distinguish between "game" and "not a game". But it's incredibly difficult to write down a precise definition of "game" that includes everything normally considered a game and excludes everything that isn't considered a game. Writing down the function of one specific neuron seems to run into this issue. When you write down what the exact function of any neuron is, there's a good chance your clunky natural language description misses some fine nuances.
A reasonable analogy, in my view, would be that neurons could represent something like the atomic components of intuitions or "vibes". Similar to the example of "game", the feeling of knowing what is or isn't a game is composed of what Wittgenstein called "family resemblances". Your brain doesn't contain a precise description of "game", but upon encountering something, you can quickly compare it to other examples of games you've seen and intuit whether something has a "game-like vibe".
Another notorious example would be "what is a woman". For some reason, everybody has very strong feelings about what is and isn't a woman, but people have a terribly difficult time perfectly describing it. I would guess that neuron activations function like these vibes or intuitions, rather than written rules.
I very much liked the parts discussing circuit ecology. This idea sounds very promising for alignment to me.
Perhaps our anthropomorphic understanding of data is biasing what objective features there actually are in the dataset.
I wonder if we can study the graph similarities of various models, do we see a convergence of structures that encapsulate certain concepts. And would that convergence be due to just how humans have narrowly defined the concept in our language, or is there something inherent to the concept that allows it to converge. If we compare the same concept in different languages (ie. chinese vs english), does this convergence disappear?
Who knows what they were thinking before, but this all sounds very good.
Even when similar weight activations arise across learning regimen it seems we will miss a whole range of interpretations if we just go off 'the text'. The outside context in which a neuron arises is equally, if not more, telling; its genesis cannot (yet?) be quite captured by the weight itself.
The network may reveal its own context and history, but what can ultimately be encoded in its weights that originated outside the network?
Also how it goes is this new consensus will be a leaky abstraction and we'll keep conjuring new words to describe new rhetorical features and bloat the corpus of philosophy until everyone is over the whole concept and then we'll start over with a whole new arbitrary academic philosophy for "running algebra on a computer"
https://xxcancel.com/NeelNanda5/status/2110875974296654062#m (and the re-post on LessWrong: https://www.lesswrong.com/posts/x4kvabkF2Jk8GLZp8/linkpost-n...)
TL;DR:
It's interesting to observe that human reasoning is still the barometer of what "reasonably adjacent" means in the context of a NN's structure.
I always catch on misuse of enormity
It's now so widespread the misuse has gained despairing acceptance, as with "infer" for "imply"... but that doesn't make it any less wince-inducing.
("ex normal" doesn't clearly mean "evil"/"sinful", and Latin already had it as "big".)
I think neural networks are enormous, why is that misuse?
"""
The original meanings that enormity carried related to great wickedness, and some assert that enormity should today only be used with such meanings, and that enormousness is the proper word to use to mean "large size." Despite these claims, enormity frequently refers to large size, either literal or figurative, though it usually suggests something so large as to seem overwhelming.
"""
- great magnitude/size/scale of something - wickedness, evil, atrocity
Purists insist on the second meaning only, and will perhaps suggest you swap with ‘enormousness’ instead.
Now or in the near future we will be studying minds much greater than our own, and we will find out how well we can understand them.
If they don't kill us all before then, of course.
Let’s say a guy got electrocuted while showering. I can say that the cause is the guy going for a shower. For others the cause is the electric wiring system failure.
Two eyes, a nose with whiskers [0], a mouth and a protruding shape, four legs and a tail [1].
A car and a cat might as well be the same thing.
[0] the logo on the radiator surrounded by horizontal stripes [1] tow hitch.
> Part of the reason was not knowing about generalized pose neurons
I have to ask, did they not look up how the human visual system works? The primary visual cortex has any number of early neurons that are spatially-tuned for various shapes and edges, and this is pretty basic knowledge.
In fairness, neural networks are more divergent from biology than a lot of people think, but it seems there's still low-hanging fruit to be grabbed with a minimal collaboration.
https://pastebin.com/raw/p11vzFnB
The implication made explicit! Not to mention you yourself say that only above average people come here.
You work in data science and you over indexed on a singular example to explain away an entire set of phenomenon??
My experience with more.... directly testable NN models that are small seems to be that they absolutely can be far too interconnected to make strong explanations. The structure and pathways they fall into don't have to make inherent sense to anything or anyone.
Is it possible that almost all nodes in a network being explainable with a singular feature that makes obvious sense to us is evidence of poor efficiency? The network hasn't been made to collapse learning or discover more effective organizational structures than what might make sense to us.
I don't think I have the fundamentals here to contribute though.
Working with some neural networks, you can somewhat feel the structure when you read between the lines of the output. You can sometimes feel that a threshold somewhere tripped and set the signals down an alternative path.
But confirmation bias and all that.
I hope interpretability has a breakthrough. It would make my job far easier and cooler.
I had figured the "Recursive generalization" was like, understood? Maybe I'm just too pop-sci brained. "Split categories into more on this new boundary" and "Combine two separate categories that aren't different enough" I thought were always the core verbs for working with bulk data. My mental model has never been formalized though, so is invalid.
I guess I never got a good sense of the examples they had which made it look like what was "obvious" or made sense to me wasn't always a good explanation in general. Neat.
It's interesting that meaningful examples both ways care about shapes so much, as your eyes encode edge finding and shapes really early in the signal path. It's why some optical illusions are so effective. It's awesome.
The whole "how do you test a neuron: Find a group of nouns and find where you can split them" is just how I find I think learning works. You build your mental model, and find the split points where it diverges from reality and you are wrong, and you refine. I feel like this is how I learned Random Forests conceptually, though surely that's not a valid comparison?
Apologies, not sure I have actually contributed anything here! I bet I misunderstood the entire article somehow. Specifically, I do not have the fundamentals of looking into visual models that he is really working from. But I always figured that when Data Scientists talk of "Features" in larger models, they explicitly meant these far more abstract representational "features". Doesn't the premise of model hijacking and abbliteration sort of presuppose that you have abstract "Reject" or "Safety" neuron paths? I've always understood this stuff as that neuron path, without weighting too much on individual neurons.
>I used to think of neurons as human-shaped nouns: one is a car, another is a curve.
I didn't think anyone was ever supposed to think this for human neurons! The way humans identify something as even a "Chair" is insanely abstract and cultural! Very specifically, the "building sub branches to disambiguate" is just, like, how you would expect a hierarchical human brain to self organize, right?
Some of this is why I've been so displeased with the interpretability research from Anthropic, where it felt like they just pointed new NNs at the disparate neurons of another NN and post hoc named them. I felt the paths were of importance at least but had no idea how to convey, interpret, or model that. Learn you a good math kids! Not having any formalisms hurts.
There are no neurons or anything like them in these image and language models.
What's next? Maybe in a little bit we will discover that we H. Sapiens suffer from severe polysemanticity in our wetware because our individual neurons are messy and massive like an ENIAC while the corvids in the park have neat and efficient little brains that don't need unfolding ?... How's the pope (may God stop afflicting him with cancer or may Man make him live forever) going to explain that?
...and it still doesn't answer it. What is this actually about?
For instance, if you had a single neuron that fires for "puppies" and "murder", it would be much more difficult to tell if the model is thinking something nice or something not so nice.
the idea that most inexplicable circuits start making sense through the lens of recursive generalization seems intuitive to me for image diffusion models but I'm not so sure about language models because of the "once removed" nature of them - I haven't made it to the bottom of the article yet so maybe this gets addressed
https://www.goodfire.com/blog/a-practical-guide-to-saes
A lot of the mech interp techniques are glorified notions of probing. If models were generally interpretable with respect to the things we cared about, we'd know know when they're hallucinating, for example.
Identifying that a common neuron or set of neurons fires for golden gate bridge can be done only because you already know that you're looking for golden gate bridge. If models had recovered some new truth (e.g., the silver door road), you wouldn't be able to elicit it without knowing to look for it a priori.
I've done work with trying to train explainable models for e.g. making diagnoses based on neuroimaging, and it is not clear to me that if you applied these techniques to e.g. vision models trained exclusively on such brain images, that these visual representations or conceptual clusters and relations would really help understanding at all.
The main issue would be that you'd have no clear way to verify any generated explanation for such training data. I.e. we can "verify" some of the explainability shown in the article because clear and recognizable image clusters and/or semantic groupings pop out, because we already understand the domains. But if this were fMRI or e.g. EEG signals, we really wouldn't have that, so the explanations would themselves just be more hypotheses.
Perhaps if we can build on interpretability of vision and language models in domains we understand, we can eventually generalize this to other domains, but I don't think that clearly follows from anything I am reading here.
Every neuron in the layer encodes fractionally something about the concepts embedded in the whole thing. It's more like a hologram than a series of connected facts.
If neurons do 'one thing' then their position encodes the relationships then snipping out that one weight would make the net forget that one thing. And we already know from many practical experiments that this is not the case. So I really wonder what the underlying mechanism is for that claim to be true.
Re the first paragraph I'm not sure what you mean by "the assumption that conflates multiple concepts that to me are substantially non-overlapping are one and the same thing"
There are lots of examples of things that we can describe to a very high degree (life itself, for instance!) but that we do not necessarily have the ability to explain.
Personally from reading the article, I felt like it scratched the 'why' itch more than the 'what' itch which presumably is what you mean by explanation vs description but it is fairly dense and I can see how either interpretation would work.
https://already-clean.goodfire.com/assets/superposition-circ...
That's the picture under "some are both dogs and cars" in the artcle
It's actually a clean shape, if you disregard the color and texture.
If you want to learn more about this, this is what depth psychology is about (Carl Jung, et al).
This is also essentially the idea behind monotheism vs polytheism. It's about engaging with and trying to understand the profound underlying unity of the myriad phenomena.
(We detached this comment from https://news.ycombinator.com/item?id=50107254 though)
Also, IMO there is an effective abstract anyway under the title text:
But you are right the article needs significant editing down.A lack of clarity often means that the ideas are confused, or that the author is trying to obfuscate weak claims.
On length and complexity, it gives it a veneer of weight and expertise and yet everything inside is vaporous.
But I agree the author is obfuscating weak claims via length / complexity (and it may be as much for his own sake as for his readers'). The basic claim ('maybe AI will be more explainable/tractable than initially thought') is IMO intelligible, and not completely nonsensical, but also extremely vague and vibesy, which, technically, I must agree is in the same territory as nonsense, at least.
Based on our other exchanges here we mostly agree with the problems of the article I think.