CNSP: keynote from
CNSP: Micah Heilbroth.
CNSP: So Micah is a brand new professor
CNSP: assistant professor. Let me get on to my
CNSP: slide show there brand new assistant professor of Cognitive AI at the University of Amsterdam.
CNSP: and his is interested in how brains and artificial systems understand language and make sense to the world
CNSP: who strained at the Dawners Institute and the Max Blanc Institute for Psycho Linguistics, under the supervision of 2 highly regarded scientists, Flora Stalong and Peter Haggart, and wrote a provocatively titled thesis. At least, I think maybe not so much provocative. But
CNSP: interesting
CNSP: we have.
CNSP: which is here. II like the the idea of the the bi-directional window into things. So I really, I wanted to put that up on the screen.
CNSP: Mica also published one of my labs favorite, or at least Ads Labs, favorite papers last year, and not just because they rail an analyze some of our labs data which happens to be available on the Cnsb website. But in this paper he analyzed word level predictions from a large language model and showed multiple hierarchical predictions. So maybe getting at some of those questions that we're asking. Well, yeah, then there,
CNSP: that were dissociated in neural activity
CNSP: which we just thought was really exciting that he could pull that out of our data and some other data. So I'm really excited to introduce our early career. Keynote to be given by Micah Heibron. So join us
CNSP: and welcoming him
Micha Heilbron: alright. So if everything should be.
CNSP: where can you be able to see my slides. Now is that correct? See your slides and hear you. Yes.
Micha Heilbron: perfect. Thank you so much. Well, thanks for the very kind introduction, and for the invitation. Of course it's it's an honor, and it's a lot of fun to be here.
Micha Heilbron: It's also very nice and cool speaking just after marlies, because our work is very similar in in some ways, at least, this is some of what I will be presenting. But I'll be
Micha Heilbron: coming from a pretty different theoretical background, but the methods are in many aspects quite similar. So I guess that's also very nice and pedagogical for all the attendees. They can see very similar ways of working and drawing quite different conclusions, or learning different aspects of the same types of analyses.
Micha Heilbron: But rights.
Micha Heilbron: So today, what I want to talk about is a sort of more high level framework for studying the brain. And we can use both the types of naturalistic data analyses that we've been that we are discussing in these 2 days, and modern AI developments.
Micha Heilbron: and how we can combine them. So now I hope that my clicker will work
Micha Heilbron: all right cool. So the theoretical backdrop here is the idea of the predictive brain
Micha Heilbron: and the predictive brain is this sort of very high level term
Micha Heilbron: for a family of models and theories really? That all revolve around the idea that the brain neuroprocessing in the brain relies on continuous prediction.
Micha Heilbron: and mechanistically, this is formalized in most detail in vision, in particular, around mechanistic ideas, like predictive coding.
Micha Heilbron: And here it is typically formalized as this bi directional processing cascade, where the feed forward component works something akin to a con ofnnet in the sense that it extracts increasingly abstract features, and this is then complemented by top down component which conveys predictions from a generative model.
Micha Heilbron: And so these predictions are constantly compared to what is coming in. And by making this comparison the brain computes a prediction error which is used both for inference at short time skills to try to make sense what is out there, and like longer time skills to learn to perform self-supervised learning.
Micha Heilbron: And importantly, this is possible to be a canonical cortical computation, which is, I guess, a fancy way of saying that it's something really quite general and something fundamental and something that is found all over the brain.
Micha Heilbron: So this should be a fundamental principle that applies not just to to vision, but also to hearing and to language. And what have you
Micha Heilbron: now? There is quite some
Micha Heilbron: evidence
Micha Heilbron: that is in line with this idea.
Micha Heilbron: and that has made the framework pretty influential over the past years.
Micha Heilbron: So, for instance, in the domain of visual perception, very influential set of paradigms is, for instance, statistical learning. So participants would then be exposed over and over to one stimulus, predicting another stimulus, such as here a jukebox predicting a clock.
Micha Heilbron: and what is observed is that as participants are exposed to these regularities, a jukebox clock, jukebox, clock, jukebox clock over and over, as one stimulus becomes predictable to hit the clock.
Micha Heilbron: You see that the brain responds differently. much more weekly than if it's unpredictable.
Micha Heilbron: And you see this in many domains. So, for instance, in language, Marlise already referred to the N. 400. So when a sentence ends expectedly, so I drink my coffee with cream and sugar.
Micha Heilbron: the brain responds very differently, and much more weakly than if the sentence ends unexpectedly like I drink my coffee with cream and dog.
Micha Heilbron: and so time and again, in all these domains. But all these types of evidence seem to point to is this predictive brain idea where the brain is seemingly constantly comparing what is going in
Micha Heilbron: to its own expectations.
Micha Heilbron: Now.
Micha Heilbron: all this foundational work using, let's say, classical cognural methods is, of course, very important, and it has taught us a lot.
Micha Heilbron: And it's the reason that I got interested in the predictive brain in the first place.
Micha Heilbron: But there are some limitations of this sort of classical approach. and the first is generalization so. But we now know very well that the brain indeed, starts predicting in these specific paradigms, like jukebox clock jukebox clock.
Micha Heilbron: What is not so clear is whether the findings in those jukebox clock scenarios would also generalize the perception more broadly. And this is actually very important for the predictive brain, because for the predictive brain. The idea is not that the brain can predict.
Micha Heilbron: but that it always does
Micha Heilbron: right, so does it also apply to a more naturalistic scenario like when you're listening to music, or you're watching movies. Or perhaps you're just seeing complex images.
Micha Heilbron: And the second question is, what is meant by a prediction, because this predictive brain theory is a very general theory. So according to the predictive brain, a prediction can be anything. So if you're predicting a clock, for instance, you might predict some high level idea, or some visual prototype, or perhaps even the very specific, low, level features of that exact clock.
Micha Heilbron: Now, of course, nearly these different options, these different scenarios would imply a very different known mechanism. But it's often not so clear in these sort of classical methods. What types of predictions are manipulated, and therefore what is going on.
Micha Heilbron: Now, there are 2 recent sort of developments in neuroscience and AI machine learning
Micha Heilbron: that allow addressing both these issues.
Micha Heilbron: and the first is the rise of generative AI. So for a long time this notion of a generative model that predicted brain theories talked about was quite elusive, and people could write down the equations, and it was formulated in a Bayesian brain framework. But no one could actually implement the types of predictions that we thought the brain would be making.
Micha Heilbron: And this is not changing, since generative models are actually working so. But, for instance, in language, one sort of very striking example in GPT. 4, and all other sort of more sophisticated generative models. So these are models that can predict stimuli, and you can. And then they can generate text. For instance, if the same is true for for videos. So they predict video frames. And they could generate video.
Micha Heilbron: Now, the second development is sort of the rise of a neural data science and open science more broadly. So we now have these large data sets of very high quality, naturalistic images of naturalistic stimuli that is both sort of in the mouse or rodents, like invasive electrophysiology, and also the types of data sets we've been talking about today.
Micha Heilbron: So what I've been doing is sort of combining these 2 trends.
Micha Heilbron: And then I call it the generative AI framework
Micha Heilbron: for studying the predictive brain. So how does that work?
Micha Heilbron: So instead of so, yeah, the set, the setup in general is quite simple.
Micha Heilbron: We start with a stimulus, often a naturalistic stimulus and not a jukebox clock stimulus, but, for instance, in language could be an audiobook or natural music, or simply natural images.
Micha Heilbron: and these are simply shown to the brain and the the brain activity is recorded.
Micha Heilbron: At the same time the same naturalistic images are analyzed with a neural network, and then specifically a generative model. So a model that can predict, for instance, the next word, or it can predict
Micha Heilbron: the next note in music, and under these predictions can be compared to the actual stimuli to compute the predictability.
Micha Heilbron: and then we perform a regression to test. If, indeed, the brain response is different. When this, the stimulus is predictable versus unpredictable.
Micha Heilbron: and this regression can be depending on the the situation. So if it's a time, continuous naturalistic stimulus, you would need deconvolution approaches like implemented in the Mtrf toolbox. It can also be a non deconvolution if there's no temporal overlap. But the the important aspect is that it's multiple regression approach. So we can include all the baseline variables like Marlis already talked about, such as acoustics or invasion can be local contrast.
Micha Heilbron: And on top of that we add the expectedness estimates to test if the brand responds differently
Micha Heilbron: to unexpectedness while controlling for low-level conference.
Micha Heilbron: Now this has 2 sort of main advantages. And the first is generalization, right? So we can move beyond the jukebox, gloves, clock scenario and test. If even when you're simply watching a movie, if the brain responds differently. When a stimulus.
Micha Heilbron: it's it's predictable.
Micha Heilbron: And the second, perhaps more interesting aspect, is that because we model the predictions instead of manipulating them. You can make the concept of prediction as precise as you want.
Micha Heilbron: so you can, for instance, compute the overall predictability of a word. But you can also separately model the predictability at different levels of abstraction, like the predictability of the meaning
Micha Heilbron: or the predictability of the exact sound.
Micha Heilbron: and so on. So here. You're only limited by your own modeling skills and imagination, and you can make this concept of prediction arbitrarily precise. So you can have as I like to go. As I like to God, you can have an unlimited conceptual resolution.
Micha Heilbron: So if we're using really powerful and beautiful open data sets, we have very nice temporal resolution. Very nice spatial resolution, and
Micha Heilbron: only limited by your own modeling. You can have very strong conceptual resolution.
Micha Heilbron: Alright. So I so that's still very high level now, the first sort of study where I was approaching this framework and I've worked out in detail was a study for my Ph. D. Which was on language.
Micha Heilbron: And here we analyzed and
Micha Heilbron: people listening to audiobooks. And one of the data sets was actually the data set recorded by, I think, Jeff, funny. And it was yeah graciously shared by the Lalore lab
Micha Heilbron: and another data set. They say, EG. The MEG. Data set that we be granted ourselves at the dumbbells.
Micha Heilbron: and to model the predictions we use. GPT. 2, which is this large generative model which can predict the upcoming word.
Micha Heilbron: Now from Gp. 2. What you get are the high level predictabilities of words. So, for instance, this is one in the sentence from one of the audiobooks. It made the boy sad, too.
Micha Heilbron: And then what you get from the model is a probability distribution over every potential next word.
Micha Heilbron: and what we came up with in this, in this paper was a method to then disentangle this overall prediction into different levels of prediction.
Micha Heilbron: So, for instance, how does that work.
Micha Heilbron: You can see here that it made the bystand to thing to see, to note, to hear, to say, to B. If you analyze all these predictions. You see, even though GPT. 2 is not exactly sure about the identity of the next word.
Micha Heilbron: All these are verbs, so it's pretty damn sure that the next word will be a verb. You can do the same for us, for semantics and for speech sounds
Micha Heilbron: so you can compute not just how predictable a word is overall, but also at what level? So syntactically, lexically, or phonologically.
Micha Heilbron: So. The striking thing of this paper was that we found that the brain responds not just to word unexpectedness, but it responds differently to each type of unexpectedness.
Micha Heilbron: So here you can see the Trfs.
Micha Heilbron: and you see that, for instance, the syntactic predictability leads to a very early frontal effect, whereas semantic predictability
Micha Heilbron: results in a much later and especially much more distributed effect, whereas phonological or phoning predictability into this early effect.
Micha Heilbron: And here I'm showing the Crfs. But we also did a analysis like Marlise described where you look at. If the model. If each of these variables explained unique cross-validated variants over and above the the baseline variables. And so we found
Micha Heilbron: independent, dissociable, and sort of unique contributions of predictions at each, like each level of abstraction.
Micha Heilbron: So given that it seems that when you simply listen to speech, the brain predicts at multiple levels of abstraction. The next question we asked was.
Micha Heilbron: How do different levels interact? And we already touched on this a bit during Q&A with malice. One way to look at that
Micha Heilbron: is, to look at different ways of doing phoneme prediction
Micha Heilbron: so normally how this works phonium predictions are sort of formalized by the idea of
Micha Heilbron: incremental word recognition. So as phonemes come in
Micha Heilbron: the cohort of potential electrical candidates that are consistent with the funding so far shrinks
Micha Heilbron: until the word is incrementally recognized. And you can cast this as a process of probabilistic phony and prediction by grouping each of the candidate's next words by their identifying next funny.
Micha Heilbron: So facts. Family fat would be at false 4. Fortunate would be all figure fin.
Micha Heilbron: finish fishing over here, etc.
Micha Heilbron: If you then attach a weight to each of these candidates you can do the math and derive a probability distribution over phonemes for each phone. Now, there are 2 ways of assigning the weights.
Micha Heilbron: and one way is to use this simple dilexical frequency or the unogram probability.
Micha Heilbron: So here you would take the probability of of an overall word overall in English. So, for instance, the word 4, it's very frequent. So we would have very high a priori likelihood.
Micha Heilbron: and the overall probability for the next phoneme is they're not highly biased by these high frequency words.
Micha Heilbron: Now, alternative possibility is that it's the the weight is not determined by the overall word frequency or word frequency, but by the contextual probability that we take from GPT. 2.
Micha Heilbron: And here the distribution would be quite different. Because if we look at this context. They say his father was a
Micha Heilbron: the word, for even though its very high frequency is not very likely because it wouldn't be even chromatical, they say its father was a. For whereas fishermen, even though over all, it's a very unlikely word
Micha Heilbron: in this context, it's very likely, because this is from the Altman and the sea. I don't know if you've read it, but it's all about fishing and and and fishermen. So Gbt, too, understood that, and knew that it was indeed very likely.
Micha Heilbron: so the overall probability distribution looks very different.
Micha Heilbron: Importantly.
Micha Heilbron: While the this frequency model is a single level model in the sense that it uses a short sequences of context so short within word, sequences of phonemes as a context plus a fixed prior, namely, the the overall probability of each word in the English language.
Micha Heilbron: The contextual model is hierarchical
Micha Heilbron: in the sense that you use not just short intra word frequencies of prior phonemes, but the prediction is also based on the longer sequences of prior words. So it's hierarchical in this way that the concept of
Micha Heilbron: hierarchical graphical models in Bayesian inference, for instance, you use 2 levels of context.
Micha Heilbron: And so what we did is we compared the sort of cross, validated prediction performance of both models and compared it to the baseline model, and what we found both in the Eeg and in the Meg participants.
Micha Heilbron: that the the models that include word phoneme unexpectedness were better than the baseline, but the models that include hierarchical unexpectedness work better still.
Micha Heilbron: and this is indeed very much in line with the sort of concurrent paper from Christian broadback.
Micha Heilbron: and where they actually found they asked a different question, and they found that sync the single level and hierarchical model seem to work in parallel. So we did. A very rough is one better than the other analysis.
Micha Heilbron: and it seems that the brain, indeed performs hierarchical computation. But there may, it may also perform single level computations.
Micha Heilbron: So the story may be a bit more subtle there.
Micha Heilbron: But overall, what we found in language is that when people listen to natural language. The brain constantly predicts, it predicts at multiple levels of abstraction. And we find this very suggestive evidence that at least in the domain of phonemes, higher level context can inform lower level context, which is exactly what you would expect from this predictive brain idea.
Micha Heilbron: And with a master student we performed a similar study in music. So the setup was quite similar. We again did one Mvg. Study at the Donders, and graciously, thankfully used the data set provided by Giovanni, which is EG.
Micha Heilbron: And the setup was again similar. So people listened to Midi, which is converted to to audio.
Micha Heilbron: and then, at the same time, we have a transformer very similar to Gpd. 2 which analyzes the media, and then we ask if the brain responses to the musical notes or not words but hear. Musical modes are different as a function of the unexpectedness of the musical notes.
Micha Heilbron: And again, we found, as well as in the Meg Eeg as in the Meti data, we find these characteristic Trfs and we find that the brain response is indeed modulated by the unexpectedness of the time.
Micha Heilbron: also similar to what Giovanni and others found. The twist of this study was
Micha Heilbron: not the Trf. Itself, but was what type of information is used in the prediction.
Micha Heilbron: So here we use not just one neural network, but we use actually many neural networks which each use a different amount of context. And what we found first is that the neural network that used the most context
Micha Heilbron: could predict the music best.
Micha Heilbron: And this is not surprising, but it was an important sanity check. So the more information we gave to the transformer the better it became at predicting the next node.
Micha Heilbron: But when we did the same analysis
Micha Heilbron: for the the data, for the for, the, not the ability to predict the music, but the ability to predict the brain. Using the unexpectedness of the music, you find a very different curve.
Micha Heilbron: So here we found that the the models that best predict the musical surprisal response was actually not the model that used the most context, but that used one of the fewest context. So context window of maybe only 5 to 10 words. So this is a very short.
Micha Heilbron: just a few seconds, just one or 2 musical motifs
Micha Heilbron: so contrary to language. What we found is that the musical surprise response actually seems to reflect very low level in the sense of short-range acoustic predictions.
Micha Heilbron: And yeah, this study was, by the way, very much
Micha Heilbron: spearheaded by Piers, who is now starting a Ph. D. And I wanted to mention him specifically
Micha Heilbron: alright. So in the last study, I wanted to highlight.
Micha Heilbron: I wanna take this idea to the visual domain.
Micha Heilbron: And here it's just the the visual perception of naturalistic images. And here the situation is quite different.
Micha Heilbron: because when you simply see a natural scene, the complex image like this, there's no temporal prediction.
Micha Heilbron: however, one form of prediction that might be there, and that is very important in sort of classical, predictive coding theories is spatial prediction or pattern completion.
Micha Heilbron: Now pattern completion is pretty easy to think about in very extreme cases, like a checkerboard, where, if you perceive only a small part of the stimulus, you can perfectly pat and complete and predict the rest.
Micha Heilbron: But how can you probe this in more naturalistic images? Now here we took an idea inspired by a paper from last year from neuron, which is very nice paper.
Micha Heilbron: And the key idea is that it relies on receptive fields. So just to make sure that everyone understand what I mean by that. The the visual cortex is sensitive to only these small apertures, small parts of the visual world.
Micha Heilbron: and if you have a dataset where you know the receptive field for each voxel or for each neuron, like in the Allen Institute's visual coding. Dataset of the Mouse visual cortex.
Micha Heilbron: What you can do is that you can take out. You can mask out the receptive field from the perceived image.
Micha Heilbron: Take a generative model that is trained for filling in, for, for impending, for reconstructing, to reconstruct and to predict the content of the receptive field. and then compare the predicted receptive field to the actual receptive field to compute the visual predictability.
Micha Heilbron: And then you regress that again on the brain response by taking into account lower level visual information like
Micha Heilbron: the acoustic kind of the visual contrast. importantly like before, you can compute not just the overall predictability. But you can compute this predictability at multiple levels of abstraction.
Micha Heilbron: So if you take your actual receptive field patch and your predicted receptive field patch, what you can do is extract the hierarchy of features.
Micha Heilbron: Here. We did that with a convolutional neural network you could theoretically also use something else.
Micha Heilbron: And in CNN. The early layers extract simple patterns like edges and lines, whereas later layers extract things like textures and object parts some more high level features. If you do the comparison at each level, you can compute not just how unexpected or how unpredictable the visual information is overall, but also at what level of abstraction, like at the level of simple edges or textures or shapes.
Micha Heilbron: etc.
Micha Heilbron: And so, just to make sure they understand what's going on. The data set here is a data of mice viewing the large full field images, all for 250 ms at the same time.
Micha Heilbron: and each image is repeated 50 times in total. That's like 6,000 trials. And you have thousands of neurons.
Micha Heilbron: All right. So if we look at V one, if we, the average response looks like this, it's the blue line.
Micha Heilbron: We see that when a natural image comes online, the the the fry roommate starts to peak.
Micha Heilbron: And so the red line here is sort of analogous to the Trf. Although we didn't use a Trf and method, but just linear regression. But it's the effect of the the average response to a natural image plus one Sd of the unpredictability.
Micha Heilbron: and just for comparison, the the gray line here is the effect of the average response, plus that of local contrast in the preferred spatial orientation and spatial frequency.
Micha Heilbron: So it's quite interesting, as you can see, that unpredictability modulates the response, but it modulates the response only later in line with the more recurrent process, whereas a low-level contrast modulates the immediate feed-forward sweep, and it has also stronger effects.
Micha Heilbron: If you do that for each convolutional layer. So for each level of abstraction. You see that
Micha Heilbron: overall each layer seems sort of the unexpectedness of each convolutional layer modulates the brain response.
Micha Heilbron: but especially the highest convolutional layers. So if we now summarize over time and look at the effects, the level of abstraction in terms of their effect. Size.
Micha Heilbron: But we find is that indeed the later layers of Alec Smith have the largest effects. and if you're a vision scientist, you may not be a bit puzzled, because this is indeed visually early. This is v. One.
Micha Heilbron: So we see that the the most most abstract feature on predictability has the largest effect.
Micha Heilbron: We know from v. One that v. One itself is sensitive to low, level features.
Micha Heilbron: So indeed, if you do, the classical sort of decarlo encoding approach where you so we're not thinking about unpredictability now, but just the sensitivity to the visual features we see, we see exactly the opposite curve. So v. One neuron neuro responses are best predicted by low, level features.
Micha Heilbron: But high level unpredictability.
Micha Heilbron: So you see exactly the opposite June. which is kind of interesting.
Micha Heilbron: Oh, yeah. And then, sort of as a last tidbit. All this work, by the way, is still very much in progress, so it may still change a bit
Micha Heilbron: as we do some additional analyses. But one thing that I didn't want to ignored today is the laminar analysis.
Micha Heilbron: So in this data set, the myas have these neural pixels probes in their visual cortex, and they measure not just
Micha Heilbron: at one visual and one cortical location, but they measure all throughout the cortex. So what you can do is look up for every unit that you record where they are in cortex, and then sort of
Micha Heilbron: very roughly subdivided them into superficial or deep cortical layers.
Micha Heilbron: And if you do that, you find that the effect is much stronger in the neurons that are classified as superficial than those in deep. But this is only for the feature, unpredictability.
Micha Heilbron: We don't find this for contrast.
Micha Heilbron: So it seems that this unpredictability effect is not just tuned inversely to what you might expect from v. One. But it's also strongest for the superficial layers compared to the deep, which is exactly in line with many of these theories about the predictive brain.
Micha Heilbron: Alright. So if you come back to that question, so generalization. do we find that indeed the brain always predicts
Micha Heilbron: so what did these studies so far show, including others that in language, yes, in music, yes, in vision. Yes, we find predictability effects and and continuous prediction in all these domains.
Micha Heilbron: If we don't ask the second question.
Micha Heilbron: what is actually in this prediction we see that in language we find this hierarchy of predictions where the brain is constantly predicting at multiple levels of abstraction and higher levels seem to be able to inform lower level predictions.
Micha Heilbron: whereas in music we find strikingly low level in the sense of short range predictions.
Micha Heilbron: And then finally, envision, we find these higher level predictions that are then fed back to V one. Although I have to say that yeah, this visual, this vision project is still work in progress that the exact conclusions may still change
Micha Heilbron: alright. So, summing up
Micha Heilbron: the rise of generative Iai AI enables stronger and more precise tests of the predictive brain hypothesis.
Micha Heilbron: They so far reveal ubiquitous prediction in many domains and across levels of representation. And to me, at least, this really showcases the power of AI to change the way we do cognitive neuroscience.
Micha Heilbron: And yeah, but the last point is just,
Micha Heilbron: it's my own personal view. But I think this is really the future for the study of the predicted break.
Micha Heilbron: So with that, I just wanted to thank you, of course, for your attention and all my co-authors, and especially the people who provided all the open source, software and data with special mention here, Giovanni and the Lol lower lab more generally.
Micha Heilbron: because without you it would really would it definitely possible. But I've been doing over the past years.
Micha Heilbron: So with that, thank you very much.
CNSP: Well, cool talk. Really like it. I'll anyone have any questions we have.
Micha Heilbron: Should I stop sharing?
CNSP: Oh, you can. You can keep it up in in case you have slides that could possibly answer questions.
CNSP: So we have one for one of the attendees. I'll we'll let attendees ask questions first, and then, you know, I have a question. Giovanni has a question, so if there's no questions, we'll
CNSP: we'll ask some questions. But there's an attendee who has a question feel free to unmute if you'd like.
danna pinto: So I have a question about the language model.
danna pinto: Yeah.
danna pinto: So you talked about how? Ii mean. I know you use it as an example. But in the old man and the C example it sort of use the context I I'm assuming I don't know so much about the Gpt models. But I'm assuming it uses context that it finds online to sort of make the predictions. Is that
danna pinto: correct assumption?
Micha Heilbron: Yeah, definitely.
danna pinto: So would then understanding the human brain like, wouldn't that be a big jump? Because
danna pinto: the human brain doesn't have all of the information that.
danna pinto: like the the model, would have so like context, clues, would be more in terms of like the meaning of the sentence. But then, when you look at explosives that include all of the literature you know ever written.
Micha Heilbron: Wouldn't that be like a not equal comparison? Yeah.
Micha Heilbron: yeah, no, I'm really happy. You'll ask this question.
Micha Heilbron: because, yeah, I first had a slide on sort of clarifications and common objections, but I removed it for the sake of time. But one of these things it was actually really happy. You ask, because it's a question that comes up a lot.
Micha Heilbron: So in this
Micha Heilbron: approached where you use a generative model to quantify the statistics that you hope that you assume that the brain is tracking.
Micha Heilbron: It doesn't require assuming that the exact statistics computed by Gbd are exactly what is tracked by the brain. and what I often
Micha Heilbron: do is make a comparison to something like how we compute word frequency, for instance.
Micha Heilbron: and so for word frequency. We often have these algorithms that that we count the word occurrences in a huge corporate. and they are often much larger than what you read
Micha Heilbron: in your entire lifetime.
Micha Heilbron: But so this is not meant to be an exact statistic of what you exactly have in mind, but what people tend to find is that the better you objectively approximate this statistic on average, the better that you would track the specific predictions in the brain. And that's also what you find in language.
Micha Heilbron: Some are least mentioned before that she couldn't rarely find significant effects
Micha Heilbron: differences between different models like an Rnn. And an ngram. But in reading time, in the reading time literature, people have done this a lot, and what people always find is that the better the language model becomes so going from an Ngram via an Rnn.
Micha Heilbron: Via S. It's simple transformers up to like GPT. There's this monotonic, almost linear relationship between the quality of the language model and the quality of the fit. And that's probably not because the human brain is exactly making that prediction. Probably the human brain is doing something less sophisticated. It has also read read less and learned less, but the more it matches the objective statistics.
Micha Heilbron: the more, on average, it will be closer to the idiosyncratic statistics on individual participants. So that was a long-winded explanation.
Micha Heilbron: But I hope it makes it clear.
danna pinto: Yeah, it makes more sense. Thank you. Thank you.
CNSP: Hi, Mike.
EDMUND LALOR: this is Ed Laura here I had a question on them.
EDMUND LALOR: If I just trying to understand the you had a kind of an X shape for these predictions versus low, level contrast that the was sensitive to is that is that right?
EDMUND LALOR: So I'm I'm maybe you could just talk us through that again, because I was trying to understand how
EDMUND LALOR: what V one might care about in terms of low, level features. So if if you know, if if I think about predictive coding theory, surely it should be getting predictions
EDMUND LALOR: that are of predicted low, level features, so maybe you could talk me through this curve again. Oh, no, no, you you didn't misunderstand it at all, and it is also surprising, and it's also what I expected, even though this particular neuron paper already pointed a bit in this direction.
Micha Heilbron: It was also what I expected. So I was. I was expecting to find that early level, early visual cortex is sensitive to predictability of lower level features, whereas later visual cortex would be sensitive to the predictability of high high, level features, and that would indeed be in line with
Micha Heilbron: very strict or sort of cartoonist view of predictive coding. But what we find is the opposite.
Micha Heilbron: So I think you understood it perfectly. We find this inverted
Micha Heilbron: tuning functions, where, in terms of the load level features. V. One is indeed sensitive to the lowest levels of here you quantify its via the encoding performance of Alex Net.
Micha Heilbron: But in terms of the unpredictability, it's exactly the other way around. And
Micha Heilbron: and yeah.
Micha Heilbron: I can speculate why, that may be
Micha Heilbron: But in in terms of, if you've understood it correctly, I think you understood it perfectly.
Umhm. Okay, yeah, thanks.
Micha Heilbron: not yet. If there's not another question, I can also speculate a bit. Why, this may be the case.
Micha Heilbron: One thing that I find myself especially interesting is that there are now results in
Micha Heilbron: in, in, in machine learning.
Micha Heilbron: where people are impressed by sort of the performance of GPT. For instance. And they, they think, okay, if we can learn everything there is about language, just from predicting words. Can we also learn everything there is to learn about a vision just from predicting pixels. So they've trained these large models to predict pixels based on their current.
Micha Heilbron: And what they find is actually that models that have become really good at predicting pixels are really not so useful for high level visual tasks, and the only methods that work very well in terms of this self, supervised pre training our
Micha Heilbron: methods that explicitly, in some way or another, discourage the prediction of low level features and anchorage, the prediction of high level features. So what you see from machine learning is that the thing that's really useful in self, supervised pre-training is prediction of high, level features. And then what we see in the brain is that the thing that really drives early visual cortex is the predictability of high, level, visual features.
Micha Heilbron: So that's to me at least a striking parallel. And to me, what that may suggest is that this may so this may reflect a learning signal which we know from machine learning to work really well.
Micha Heilbron: but that's, of course, speculation. You could also come up with other explanations. But it's my best theory for why we may find this opposite tuning.
EDMUND LALOR: That's very interesting. Thank you.
CNSP: Joe. Did you want to ask a question, or should I?
Giovanni Di Liberto: Yeah, have a have a quick one. I don't
CNSP: go for it.
Giovanni Di Liberto: Music, hierarchy
Giovanni Di Liberto: have you tried to look into? You know your second point in the conclusions was that yeah, there is the hierarchy in speech.
Giovanni Di Liberto: and but in music you find this slow. small, you know short context to be the optimal. But have you tried to combine short, long context?
Micha Heilbron: No, no. So the you don't know. And so the yeah
Micha Heilbron: the answer is, no, I also really wouldn't know how you can do that actually, that this is why I like working with language so much, because we have this large inventory, and we know that sort of words exist. At least, if you believe words exist in the mind, we have syllables and phones, and we can really dissect stimuli at these multiple levels, and then explicitly. Look how one level informs the other. I wouldn't really know. So I know what a note is.
Micha Heilbron: but I wouldn't. I don't have the conceptual apparatus to know what a word would be in in in music. So if we could, II wasn't smart enough. Let Le let's move like that to to perform the same types of analysis. But what do you think? How can you test this music? What's the right level of currency?
Giovanni Di Liberto: Yeah, I mean, I'm just thinking, I mean, from a purely methodological point of view, you could have multiple features fit with different context lines. That that's the boring thing. The other thing is, is inspired from what Ed said to me years ago on classical music, that the first time you listen to it, maybe you just focus on the on on patterns that are short term. And then you start getting the the, you know, different levels of
Giovanni Di Liberto: of of context. And so maybe maybe modulating that on the participant
Giovanni Di Liberto: getting multiple repetitions of complex patterns, and see whether that curve that you have there changes, you know, having the same identical method you have there, but with different conditions or different. Yeah.
Micha Heilbron: yeah, no, that's true. It would require yeah. And and we wouldn't be able to use your download. Get your data and use such an open ended data set. But indeed, yeah, I agree with sort of very clever experimental design. I think you you can go there.
Micha Heilbron: And what did you mean with the first boring one you meant like a decoding analysis
Micha Heilbron: and then sort of comparing the decoding performance on a time result basis and relating that to expectedness or something. Or, Yeah, sorry, Adam, this is not as quick as I thought, and a bit like what you're doing language that would include multiple features.
Giovanni Di Liberto: multiple surprise vectors that were fit using different context lengths. You know what I mean.
Micha Heilbron: Reds you you could. You could either simply use that model and have different context lengths if they're independent of each other.
Giovanni Di Liberto: Sorry. And I mean they could be both. It just may. Maybe that's not. Maybe not with that model. I don't know. Maybe the other models.
Micha Heilbron: Yeah, we're not.
Micha Heilbron: I don't know. II do like the this effect, because I, initially pious, came up with the idea. And I was like, Are you sure, Piers, because I think the better the model is, the better it will fit the brain. Do you think we can have an interesting effect, and then piece like well, we don't know blah blah blah! And then we did it. And we found this really striking dissociation. And I never thought we we would have
Micha Heilbron: so yeah.
Giovanni Di Liberto: good. Thanks.
Micha Heilbron: thanks a lot.