See my screen.
Stephanie Haro: Yes.
Marlies: okay, great.
Marlies: So in the next couple of 20 min I want to give you a brief introduction. About linguistic speech tracking by the brain. And I will start with a very, very general introduction
Marlies: to even just neural speech tracking. I know there are a lot of experts in the audience, and but just to make sure we're all on the same page.
Marlies: So whenever you have an issue with hearing
Marlies: the the audiologist, which will test how much speech you can understand can rely on an expensive test battery which
Marlies: Holds, for example, pure to an audiometry or sentence recall tests.
Marlies: and this test they have some clear disadvantages. Of course, these pure tone audiometries. They work with tones and tones you cannot understand. And these sentence recall tests.
Marlies: They actually work with really standardized and control sentences, which is nothing like real listening environments.
Marlies: Additionally, if you fail on these tests or cannot perform these tests?
Marlies: Then the outcome does not allow you to pinpoint the origin of the deficit. Is it situated more peripherally, and can the patient be help to the hearing aid? Or is it situated more cortically? Where maybe language treatment is a better option
Marlies: mit ctl, and additionally, there are also, or the another drawback of this test is that they are not inclusive. Not everyone can perform this task. You need to have attention, and you need to be able to give feedback.
Marlies: But sometimes this is not possible. Think about working with very young children or patients with with coma, for example.
Marlies: So that's why our research group and many others, and try to go towards a more objective way of testing speech, understanding, meaning that we look at brain signals to see or to try and see whether or not someone understand speech, and to do so we rely on the phenomenon of neural checking
Marlies: so very quickly. II know you've heard this yesterday, but when you, whenever you listen to speech, and you would meanwhile record brain signals, then you can actually trace certain characteristics of the speech back in the brain signals
Marlies: and previous studies showed that, using specific features of the speech, such as acoustics, it can really give you insight in how much speech you can understand. So if neural tracking to the acoustic feature of the speech is higher, the speech is better understood.
Marlies: but then becomes the issue. Does neural checking to acoustics means
Marlies: speech understanding?
Marlies: Well, that's not necessarily the case, because we also observe like significant acoustic neural checking. Whenever you listen to music or whenever you listen to the ignored speaker in a cocktail paradigm.
Marlies: so you cannot say that if you have acoustic tracking, that there is a successful speech understanding. It might be that acoustic speech tracking is necessary in order to understand speech, but it's not a causality.
Marlies: So before we delve into. The linguistic tracking stuff that I'm really eager and to talk about. I want to first take, take a step back and go fairly, conceptually, conceptually, about the methods that we use
Marlies: and how this method actually works.
Marlies: So basically we have a participant coming to our lab and we need them listen to stories or podcasts.
Marlies: And in a first step, we estimate a brain model.
Marlies: So we accept some speech and representations think about the envelope and acoustic speech representation here shown in the pinkish line.
Marlies: This can also be suplexical features, so as the lexical features denote the onset of a word or a phoneme, and those are indicated by the the pulses. So whenever, for example, here, whenever you have a phoneme, there's a pulse, and disposers are always of the same height.
Marlies: and then you have the linguistic features. So basically, these are also features with a lot of puzzles. But this time the amplitude changes, depending on how much information the following carries, but more details on those linguistic features later.
Marlies: So whenever you have extracted these speech representations, you have, of course, also the recorded brain responses. And then in in the next step, you basically link those speech features with the recorded brain signals 250,
Marlies: and that's to obtain this temporal response function
Marlies: in a second step. You actually have this estimated brain model. You will use this trf, is sample of response function
Marlies: to in order to extract a measure of neural tracking. So first again you extract speech, features, and then you do a convolution with the temporal response function.
Marlies: a temporal sorry. A convolution is basically a multiplication over time. Just as a quick recap. So we have a feature here. We have a Trf. This multiplication over time or convolution works as follows, so
Marlies: whenever there's an increase, you see the wave form of the temporal response function.
Marlies: Now, whenever the signals become more complex. You can predict more complex signals. For example, here you have 2 impulses next to each other, so the 2 Trfs are very close to each other, so the outcome of the multiplication over time, or this convolution will be a summation of the 2 signals.
Marlies: On the other hand, these waveforms can also be out of phase. So there's like a cancellation of this way here.
Marlies: Now, if you also vary the impulses, heights, or amplitudes, then you can also. Yeah, this also has an effect on your predicted signal.
Marlies: So
Marlies: using this multiplication over time, we have actually our our speech feature that we extracted, we have the temporal response function which tells you how the brain responds to the speech feed feature. Using this multiplication over time. We're basically giving or predicting a brain response
Marlies: and evoked by the speech.
Marlies: we, we not only have the predicted brain signal, but also the actual brain signal, and that allows us to calculate a prediction accuracy which is a measure of neural tracking.
Marlies: It's basically a correlation between each channel and of the predicted signal with the actual recorded signal.
Marlies: I hope.
Marlies: this very brief introduction. Was brief enough for the experts, but introductionary re enough for all the beginners or for people who are unfamiliar with this method. The takeaway is, you have 2 important outcomes. You have this simple response version which tells you how the brain response to the speech features. And you have this prediction, accuracy which tells you how well the brain
Marlies: tracks the speech. It's the measure of neural tracking, and this you have for every
Marlies: or both measures you have for every sensors that you use.
Marlies: So, using this approach.
Marlies: I will. I tried to answer a couple of questions. And I will cover these 3 questions during the talk. So the first question is, which
Marlies: speech feature should we actually use whenever you want to investigate linguistic processing by the brain.
Marlies: and then the the second, and also the third, is whether these speech features really reflect speech understanding, because we tested this via 2 different paradigms, and I would like to go into them both.
Marlies: So let's start with this first question.
Marlies: For this we tested, and 29 young normal healing participants. We listened to a really long story presented in silence
Marlies: at the beginning of my Ph. D. The first and studies were published using linguistic speech, features.
Marlies: So, as a starting point, we we made like a scheme of all the speech features that we were aware of. That used linguistic speech features for other features.
Marlies: and we wanted to test
Marlies: whether these linguistics speech features at different levels. So you have at the phoning level. So which characterizes the information at the level of a phoning at the word, or even at the level of the context.
Marlies: Whether these linguistic speech features actually
Marlies: explains more variance in the Eeg signal, in the neural signal on top of acoustical and suplexical features.
Marlies: Now, why is this important?
Marlies: If you look at speech features. They're all derived from exactly the same signal, and this causes that these speech features are highly correlated.
Marlies: So whenever you have a burst of
Marlies: Acoustic energy.
Marlies: You know that the envelope is going to be like a high waveform
Marlies: symbolizes this first. but at the beginning of this burst
Marlies: there there is phoneme on such, and also an impulse of how much information this phoneme
Marlies: Gary's.
Marlies: So if you would only test a linguistic feature. it is still highly correlated with the response to, for example, the envelope, or purely the acoustics of signal.
Marlies: So to make sure that we estimated linguistic tracking
Marlies: in a way that's it's not biased by acoustic responses. We used. A specific framework.
Marlies: II will call it the subtraction approach.
Marlies: So basically, we calculated 2 types of models we complete model. This complete model
Marlies: contains acoustic speech, features, suplexical speech, features, sort of phonemic word on sets as well as these linguistic speech features, and then we subtracted the baseline model which actually included all control variables. So these acoustic features and the suplexable features, and by taking the difference
Marlies: we could investigate the added value of these linguistic features. So
Marlies: how much does a model improve in predicting the neural response? Whenever we add these linguistic features on top of the acoustics and suplexical features
Marlies: this way, and although it's not the perfect of of approach, of course.
Marlies: this way, we try to minimize the bias towards acoustic response. But you see that the responses to linguistic speech features are really, really, really small. So that's also very, very challenging.
Marlies: But using this framework. We applied it on all the existing linguistic features, or or features that were already discussed in literature. And then we we we observed that not all of them survived this approach. So
Marlies: we we observe this phoneme
Marlies: and linguistic features, so phoning surprise, line, port entropy, and these linguistic features at word, level. So word, surpriseal and word frequency. Those were significant. So for the, for the rest of my talk or we, we focus, or the rest of the studies that we that we did. We focus on those 4 linguistic features.
Marlies: But for this talk I will only focus on board surprise all because it's a very intuitive feature. If you want to know more details about the other features, that's content for the workshop.
Marlies: So basically, word, surprising is
Marlies: mit ctl, and how predictable or surprising a word is given the previous words. So, for example, here this sentence, even in the darkest of times.
Marlies: If one only remembers to turn on the
Marlies: it's not surprising you're going to expect light. So the surprisal value of light is rather low.
Marlies: If it would be. To turn up the music. That would be surprising, because it changes the concept of, or the context of of the sentence that you had in mind.
Marlies: So these previous results we obtained by looking at prediction accuracies. So by looking at whether the linguistic
Marlies: features improve the prediction, accuracy, so the measure of neural tracking. But we also had this other important outcome of the model. So the the temporal response function which tells you how to brain response
Marlies: and for word, surpriseal. It looked like this. So
Marlies: you have a clear negativity around 400 ms, which is really prominent for this central channels here.
Marlies: and it's strikingly so. This is very similar to the
Marlies: Even in this case it's using continuous speech and not carefully created controlled sentences.
Marlies: So in this first study we identified which speech features we should use. The second question was, well.
Marlies: These speech features they characterize so linguistic information that? Do they really represent speech understanding?
Marlies: So to answer that we manipulated the the amount of speech understanding by
Marlies: speeding up or manipulating the speech rate. And we did this in 18 young normal hearing participants
Marlies: in total, we had around 6 speech speech rates, and each speech fragment was around
Marlies: 10 min. So, to give you an example, I hope, rates, I am quickly going to check whether
Marlies: my sound is shared as well. and. Kate.
Marlies: hope you can still see my screen, and hope you can hear this example, Mary, out a to name. So
Marlies: Dutch speaking. So this is really slow speech for some of you it might be intelligible, but for our participants it was understandable.
Marlies: So this is was a slow condition. This was a fast condition.
Marlies: It's very difficult to understand. No participants actually reported that it was well intelligible. So what happens with linguistic tracking? What happens with this added value
Marlies: of these linguistic speech features.
Marlies: so the the red and dots here indicates
Marlies: the significant channels or the channels which drive the topography to be significantly different from 0, and in the slow and normal condition. You see in these conditions that it's. You see these rich channels
Marlies: meaning that the linguistic features do have an added value, but as it becomes faster and faster, less red circles. So this added value of linguistic features.
Marlies: it disappears.
Marlies: Another. Then again, you can also look at the temporal response function and how to bring response.
Marlies: And then,
Marlies: you see.
Marlies: and this so
Marlies: mit Ctl, and although it's super noisy because this time we only have 10 min of data. Well, previously we had 45 min of data.
Marlies: but you see, in the yellowish stones we see a significant and negativity
Marlies: which is only present for the slower as slower speech rates which were intelligible by the listeners
Marlies: also this characteristic topography. It fades away, and the faster the speech becomes.
Marlies: So this gives us a first insight like, okay, maybe maybe linguistic speech tracking really gives an indication
Marlies: of speech understanding.
Marlies: But there is a big butch
Marlies: because
Marlies: this paradigm. It's used. It used artificial speech. So we artificially sped up speech. So what actually happens with natural speech?
Marlies: So in this case.
Marlies: we did a new experiment with 19 young normal hearing participants which I listened to different fragments, Dutch, Frisian, and a scrambled word list, and then we assess the effect on linguist
Marlies: to give you again an example. So for Dutch listen listeners, this will be intelligible from the Harvard happen of ejector Ripper
Marlies: and the Frisian story several Maddie did benefit of tall for my tattoos and traea.
Marlies: So the interesting about Dutch and Frisian, so Frisian, is, and spoken in the upper north of the Netherlands, and it follows a very similar grammatical structure as as Dutch
Marlies: it's near me on a one on map, one map mapping. So that's why we chose Frisian
Marlies: but the participants reported that they had around 50% of intelligibility. But this is, of course, subjectively rated and then a last example is the work list, play for May in is Whom is is a Nadim.
Marlies: So
Marlies: all these speech fragments are spoken by the same speaker, and they are also spoken in a very natural way, aiming to preserve the normal prosody.
Marlies: Then I. So we can actually again plot the same results. So can we
Marlies: see added value of these linguistic speech features. So now we had only 7 min, and we only observe, these significant channels for the Dutch story, but not for the Frisian, nor the word is audition.
Marlies: Then in looking at the Trfs, we also see only this prominent negativity around 400 ms for the Dutch story, but not but not for the Frisian or the wordless condition. And again, this characteristic topography, it disappears whenever it becomes not intelligible.
Marlies: Yeah.
Marlies: So this again gives us some insight that maybe linguistic tracking might be a good indicator on whether or not speech was understood.
Marlies: But there are some clear pro and cons. So because it's indicated like, or my results show that it might be an indicator for speech understanding. So it can really pinpoint understanding and not just hearing
Marlies: the disadvantage is that you require lots of data. You you need quite some participants, because currently we can only and look at its own population level
Marlies: and also for participants. There's a minimum of of 10 min. But I would
Marlies: give you like, ideally the more data the better because it's such a small effect which makes it really, really challenging. But we're currently working on an approach to make this more robust. But it's challenging.
Marlies: So in my, my workshop, my goal is to really
Marlies: make you able to craft your own speech features to make these phoning segmentations
Marlies: do this language model so you can create your own word, surprise all stuff, and you can apply this for your experiments so that we can all accelerate science. Together.
Marlies: So yeah, this is, that's it. And I'm happy to take any questions.
CNSP: Cool, cool talk, thanks, Marlise.
CNSP: We have a question in the chat, but
CNSP: the the asker is going to ask for themselves.
danna pinto: Hi! Can you hear me?
CNSP: Yep, yes.
danna pinto: my question was when you were looking at the the different
danna pinto: the difference between the fast and the slow
danna pinto: speech. Did you also look at the differences in just the acoustic model
danna pinto: to see if, if
danna pinto: changing this change, the
danna pinto: the tracking of the envelope, or the other acoustic features that you used.
Marlies: Yes, good point. Yes, we did look at at the acoustics as well.
Marlies: Maybe I can can an update I'm trying to.
Marlies: So I'm going to. I hopefully. You can see my draw. Can you see?
Marlies: Okay?
Marlies: yeah, it's beautiful.
Marlies: And
Marlies: so what? We actually well. and anyway.
Marlies: what? What? My point what what else? I cannot proceed with this drawing, and not the technical enough, I guess, to manage zoom. But anyway, what we saw this that's with speech rates. So this is increasing speech rates. And this is a neural tracking. So in a beginning we see an increase with higher speech rates, and if I'm not mistaking this effect was also seen yesterday. So
Marlies: with with higher speed rates. In the beginning there's higher acoustic tracking, which I think makes sense, and because there are just more acoustic boundaries to to track and to respond to.
Marlies: So your neural checking improves. But then, at the at the end, where it's super super fast, we see again a decrease.
Marlies: and this post puzzle those a bit. So we went digging in literature.
Marlies: And then we could form form some hypothesis. But this is purely speculations that this might be due to motor cortex involvement. So when.
Marlies: whenever you listen to speech, your motor cortex is active, and it sings a bit to how your lips would move.
Marlies: But when the speech is so so fast
Marlies: you don't have this lip image, and you cannot mimic on how your muscles would be able to pronounce this. Hence your acoustic tracking would be
Marlies: slightly lower when it's super super fast, when it's unnatural, unnatural to be
Marlies: pronounced.
Marlies: So I hope that's clear. And it answered your question. Yeah, yeah, that was great. Thank you so much. And it was a great talk. Thank you. Thanks
CNSP: any other questions.
CNSP: and if you want to ask that, you're more than welcome to. I can unmute you, or I can ask it.
CNSP: If you raise your hand you can ask, but if you don't raise your hand I'll just read it.
CNSP: There you go.
EDMUND LALOR: Hi, Marlise! That was a really nice talk. I don't know. I think I've asked you this question before just about kind of linking across the different levels of
EDMUND LALOR: processing where you know some of the time if you're calculating things like phonemes, surpriseal and so on.
EDMUND LALOR: It's it's kind of word specific phonemesal. Whereas obviously the phoneme.
EDMUND LALOR: the surprise of a given phone name will be affected by how likely that particular word was to come up, and all that kind of stuff. So I don't know if you thought about that any further or
EDMUND LALOR: have anything to say on that
Marlies: yeah,
Marlies: so
Marlies: indeed and there are like lots of ways, that linguistic features can be improved.
Marlies: We currently calculate for phoning supervisor for a word, independent of the previous words. So that's indeed a drawback of the of the current approach.
Marlies: But there are papers. A fairly recent one.
Marlies: I broke back
Marlies: where he really and try to. Yeah, see at which level. This phone.
Marlies: phoning, surpriseal stripy models.
Marlies: whether? How? How much? Yeah.
Marlies: Context windows and where it's more top-down or a bottom up effect.
Marlies: I
Marlies: I feel using linguistic features that it's bit like
Marlies: Our brain is not exactly the same
Marlies: as a language model.
Marlies: So there there are. There will be ways to improve this
Marlies: but I think it's very challenging to do it really in the right way, because even if you take further context into account. How will it affect? Like.
Marlies: yeah, I
Marlies: I don't know. I'm puzzled by this myself to to craft a optimal linguistic feature. I think this is a good one.
Marlies: Starting points. But yeah, way, more model
Marlies: using lots of participants
Marlies: and 45 min of data. I cannot find a significant difference between those 3 language language models, although the complexity is really really different.
Marlies: So my takeaways like
Marlies: this is a group, and like linguistic features are good proxy.
Marlies: That's like, I don't think the brain is really a language model.
EDMUND LALOR: I hope that's not your question. Yeah, that's lots of interesting stuff there. In that answer. Thank you
CNSP: have a question if no one else has one.
CNSP: I know Tom is big on big on creation and patents, and I was wondering if if your involvement, or if the lab involvement in in this sort of project sort of ends at like the basic science, or if if you all are working on
CNSP: implementing this
CNSP: for audiologists or clinical practice
CNSP: like to any degree, any level of degree.
Marlies: Well.
Marlies: I'm not involved in any patterns. So and I'm also not really aware of all the
Marlies: all those things. So
CNSP: if you want to use this stuff, you can use this.
CNSP: Okay welfare no more.
CNSP: Someone has a hand raised. Ola has a hand raised.
CNSP: Feel free to ask a question, or.
CNSP: Ola.
okay.
CNSP: did you raise your hand? All the
CNSP: we can't hear you. Can you hear me now?
Ole: Okay.
Ole: So my question was in the, I think, last experiment you showed like the contrast between word lists and the contentious story that the word list doesn't show like an effect of word surprising.
Ole: Like, have you tried mo like actively modulating the supprisal of words in the word list, like in the sense that you know. If I have like a list where I say, like
Ole: house, cathedral, bridge, elephant, or something like that, like the word elephant, would be, would be surprising in that scenario. And maybe that would be like a way we could disentangle sort of narrative surprising from this sort of simplistic word model. That just word by word probability. Is that something you've looked into?
Marlies: It's not something I've looked into. But it's very interesting.
Marlies: because I think the and 400 ish response that you will get using
Marlies: your like word lists
Marlies: which are actually semantically re related, and as certainly unrelated. Those will yield a way bigger response and narratives.
Marlies: I
Marlies: that's my my gut feeling because
Marlies: those like
Marlies: you are really manipulating it in order for the participant to be surprised.
Marlies: while
Marlies: it is not really natural like, no one is going to say to you that he? Just 8% which would send which with elephant. So
Ole: yeah, I mean, II gave an extreme example now, of course. But I mean, you could use language models to like carefully sort of titrate the amount of surprise in the word list. So it kind of matches, that of a of a natural story, and then.
Marlies: compared with you. And then what you think the 2 responses would be similar, or there would still be some difference between the 2. Yeah. Yeah. Okay. I I made a wrong interpretation. No, I know. I think then probably the effects
Marlies: hmm.
Marlies: might be similar. But it will be very, very tricky, because you always need to be primed.
Marlies: And
Marlies: so, if you're in word lists like house cathedral
Marlies: those words are will probably have.
Marlies: like similar words, probabilities, or or maybe, like semantically, they're in a in a very same category, and those artificial settings will be also very difficult to recreate in a naturalistic
Marlies: story condition, because in the naturalistic story condition, we really have these grammatical structures, pronouns and verbs.
Marlies: it's it's interesting to think about. And maybe if you're really clever, you can come up with a cool design. That's
Marlies: II think it's
Marlies: where to wrap your head about it. But I'm not 100% sure what the outcome will be, but it would be cool to check it out.
Ole: Thank you.
CNSP: Yeah. I agree. I think that's pretty cool idea. I think there's a little bit of stuff to like think through that. But I think that's a great idea. Thanks, Marley, for that. Or with that, I think we'll we'll thank Morley virtually at least, and move on
CNSP: for the sake of time. I'm gonna steal the screen back from you, Morley.
CNSP: So up next we have our
CNSP: keynote from
CNSP: Micah Heilbroth.
CNSP: So Micah is a brand new professor
CNSP: assistant professor. Let me get on to my
CNSP: slide show there brand new assistant professor of Cognitive AI at the University of Amsterdam.
CNSP: and his is interested in how brains and artificial systems understand language and make sense to the world
CNSP: who strained at the Dawners Institute and the Max Blanc Institute for Psycho Linguistics, under the supervision of 2 highly regarded scientists, Flora Stalong and Peter Haggart, and wrote a provocatively titled thesis. At least, I think maybe not so much provocative. But
CNSP: interesting
CNSP: we have.
CNSP: which is here. II like the the idea of the the bi-directional window into things. So I really, I wanted to put that up on the screen.
CNSP: Mica also published one of my labs favorite, or at least Ads Labs, favorite papers last year, and not just because they rail an analyze some of our labs data which happens to be available on the Cnsb website. But in this paper he analyzed word level predictions from a large language model and showed multiple hierarchical predictions. So maybe getting at some of those questions that we're asking. Well, yeah, then there,
CNSP: that were dissociated in neural activity
CNSP: which we just thought was really exciting that he could pull that out of our data and some other data. So I'm really excited to introduce our early career. Keynote to be given by Micah Heibron. So join us
CNSP: and welcoming him
Micha Heilbron: alright. So if everything should be.
CNSP: where can you be able to see my slides. Now is that correct? See your slides and hear you. Yes.
Micha Heilbron: perfect. Thank you so much. Well, thanks for the very kind introduction, and for the invitation. Of course it's it's an honor, and it's a lot of fun to be here.
Micha Heilbron: It's also very nice and cool speaking just after marlies, because our work is very similar in in some ways, at least, this is some of what I will be presenting. But I'll be
Micha Heilbron: coming from a pretty different theoretical background, but the methods are in many aspects quite similar. So I guess that's also very nice and pedagogical for all the attendees. They can see very similar ways of working and drawing quite different conclusions, or learning different aspects of the same types of analyses.
Micha Heilbron: But rights.
Micha Heilbron: So today, what I want to talk about is a sort of more high level framework for studying the brain. And we can use both the types of naturalistic data analyses that we've been that we are discussing in these 2 days, and modern AI developments.
Micha Heilbron: and how we can combine them. So now I hope that my clicker will work
Micha Heilbron: all right cool. So the theoretical backdrop here is the idea of the predictive brain
Micha Heilbron: and the predictive brain is this sort of very high level term
Micha Heilbron: for a family of models and theories really? That all revolve around the idea that the brain neuroprocessing in the brain relies on continuous prediction.
Micha Heilbron: and mechanistically, this is formalized in most detail in vision, in particular, around mechanistic ideas, like predictive coding.
Micha Heilbron: And here it is typically formalized as this bi directional processing cascade, where the feed forward component works something akin to a con ofnnet in the sense that it extracts increasingly abstract features, and this is then complemented by top down component which conveys predictions from a generative model.
Micha Heilbron: And so these predictions are constantly compared to what is coming in. And by making this comparison the brain computes a prediction error which is used both for inference at short time skills to try to make sense what is out there, and like longer time skills to learn to perform self-supervised learning.
Micha Heilbron: And importantly, this is possible to be a canonical cortical computation, which is, I guess, a fancy way of saying that it's something really quite general and something fundamental and something that is found all over the brain.
Micha Heilbron: So this should be a fundamental principle that applies not just to to vision, but also to hearing and to language. And what have you
Micha Heilbron: now? There is quite some
Micha Heilbron: evidence
Micha Heilbron: that is in line with this idea.
Micha Heilbron: and that has made the framework pretty influential over the past years.
Micha Heilbron: So, for instance, in the domain of visual perception, very influential set of paradigms is, for instance, statistical learning. So participants would then be exposed over and over to one stimulus, predicting another stimulus, such as here a jukebox predicting a clock.
Micha Heilbron: and what is observed is that as participants are exposed to these regularities, a jukebox clock, jukebox, clock, jukebox clock over and over, as one stimulus becomes predictable to hit the clock.
Micha Heilbron: You see that the brain responds differently. much more weekly than if it's unpredictable.
Micha Heilbron: And you see this in many domains. So, for instance, in language, Marlise already referred to the N. 400. So when a sentence ends expectedly, so I drink my coffee with cream and sugar.
Micha Heilbron: the brain responds very differently, and much more weakly than if the sentence ends unexpectedly like I drink my coffee with cream and dog.
Micha Heilbron: and so time and again, in all these domains. But all these types of evidence seem to point to is this predictive brain idea where the brain is seemingly constantly comparing what is going in
Micha Heilbron: to its own expectations.
Micha Heilbron: Now.
Micha Heilbron: all this foundational work using, let's say, classical cognural methods is, of course, very important, and it has taught us a lot.
Micha Heilbron: And it's the reason that I got interested in the predictive brain in the first place.
Micha Heilbron: But there are some limitations of this sort of classical approach. and the first is generalization so. But we now know very well that the brain indeed, starts predicting in these specific paradigms, like jukebox clock jukebox clock.
Micha Heilbron: What is not so clear is whether the findings in those jukebox clock scenarios would also generalize the perception more broadly. And this is actually very important for the predictive brain, because for the predictive brain. The idea is not that the brain can predict.
Micha Heilbron: but that it always does
Micha Heilbron: right, so does it also apply to a more naturalistic scenario like when you're listening to music, or you're watching movies. Or perhaps you're just seeing complex images.
Micha Heilbron: And the second question is, what is meant by a prediction, because this predictive brain theory is a very general theory. So according to the predictive brain, a prediction can be anything. So if you're predicting a clock, for instance, you might predict some high level idea, or some visual prototype, or perhaps even the very specific, low, level features of that exact clock.
Micha Heilbron: Now, of course, nearly these different options, these different scenarios would imply a very different known mechanism. But it's often not so clear in these sort of classical methods. What types of predictions are manipulated, and therefore what is going on.
Micha Heilbron: Now, there are 2 recent sort of developments in neuroscience and AI machine learning
Micha Heilbron: that allow addressing both these issues.
Micha Heilbron: and the first is the rise of generative AI. So for a long time this notion of a generative model that predicted brain theories talked about was quite elusive, and people could write down the equations, and it was formulated in a Bayesian brain framework. But no one could actually implement the types of predictions that we thought the brain would be making.
Micha Heilbron: And this is not changing, since generative models are actually working so. But, for instance, in language, one sort of very striking example in GPT. 4, and all other sort of more sophisticated generative models. So these are models that can predict stimuli, and you can. And then they can generate text. For instance, if the same is true for for videos. So they predict video frames. And they could generate video.
Micha Heilbron: Now, the second development is sort of the rise of a neural data science and open science more broadly. So we now have these large data sets of very high quality, naturalistic images of naturalistic stimuli that is both sort of in the mouse or rodents, like invasive electrophysiology, and also the types of data sets we've been talking about today.
Micha Heilbron: So what I've been doing is sort of combining these 2 trends.
Micha Heilbron: And then I call it the generative AI framework
Micha Heilbron: for studying the predictive brain. So how does that work?
Micha Heilbron: So instead of so, yeah, the set, the setup in general is quite simple.
Micha Heilbron: We start with a stimulus, often a naturalistic stimulus and not a jukebox clock stimulus, but, for instance, in language could be an audiobook or natural music, or simply natural images.
Micha Heilbron: and these are simply shown to the brain and the the brain activity is recorded.
Micha Heilbron: At the same time the same naturalistic images are analyzed with a neural network, and then specifically a generative model. So a model that can predict, for instance, the next word, or it can predict
Micha Heilbron: the next note in music, and under these predictions can be compared to the actual stimuli to compute the predictability.
Micha Heilbron: and then we perform a regression to test. If, indeed, the brain response is different. When this, the stimulus is predictable versus unpredictable.
Micha Heilbron: and this regression can be depending on the the situation. So if it's a time, continuous naturalistic stimulus, you would need deconvolution approaches like implemented in the Mtrf toolbox. It can also be a non deconvolution if there's no temporal overlap. But the the important aspect is that it's multiple regression approach. So we can include all the baseline variables like Marlis already talked about, such as acoustics or invasion can be local contrast.
Micha Heilbron: And on top of that we add the expectedness estimates to test if the brand responds differently
Micha Heilbron: to unexpectedness while controlling for low-level conference.
Micha Heilbron: Now this has 2 sort of main advantages. And the first is generalization, right? So we can move beyond the jukebox, gloves, clock scenario and test. If even when you're simply watching a movie, if the brain responds differently. When a stimulus.
Micha Heilbron: it's it's predictable.
Micha Heilbron: And the second, perhaps more interesting aspect, is that because we model the predictions instead of manipulating them. You can make the concept of prediction as precise as you want.
Micha Heilbron: so you can, for instance, compute the overall predictability of a word. But you can also separately model the predictability at different levels of abstraction, like the predictability of the meaning
Micha Heilbron: or the predictability of the exact sound.
Micha Heilbron: and so on. So here. You're only limited by your own modeling skills and imagination, and you can make this concept of prediction arbitrarily precise. So you can have as I like to go. As I like to God, you can have an unlimited conceptual resolution.
Micha Heilbron: So if we're using really powerful and beautiful open data sets, we have very nice temporal resolution. Very nice spatial resolution, and
Micha Heilbron: only limited by your own modeling. You can have very strong conceptual resolution.
Micha Heilbron: Alright. So I so that's still very high level now, the first sort of study where I was approaching this framework and I've worked out in detail was a study for my Ph. D. Which was on language.
Micha Heilbron: And here we analyzed and
Micha Heilbron: people listening to audiobooks. And one of the data sets was actually the data set recorded by, I think, Jeff, funny. And it was yeah graciously shared by the Lalore lab
Micha Heilbron: and another data set. They say, EG. The MEG. Data set that we be granted ourselves at the dumbbells.
Micha Heilbron: and to model the predictions we use. GPT. 2, which is this large generative model which can predict the upcoming word.
Micha Heilbron: Now from Gp. 2. What you get are the high level predictabilities of words. So, for instance, this is one in the sentence from one of the audiobooks. It made the boy sad, too.
Micha Heilbron: And then what you get from the model is a probability distribution over every potential next word.
Micha Heilbron: and what we came up with in this, in this paper was a method to then disentangle this overall prediction into different levels of prediction.
Micha Heilbron: So, for instance, how does that work.
Micha Heilbron: You can see here that it made the bystand to thing to see, to note, to hear, to say, to B. If you analyze all these predictions. You see, even though GPT. 2 is not exactly sure about the identity of the next word.
Micha Heilbron: All these are verbs, so it's pretty damn sure that the next word will be a verb. You can do the same for us, for semantics and for speech sounds
Micha Heilbron: so you can compute not just how predictable a word is overall, but also at what level? So syntactically, lexically, or phonologically.
Micha Heilbron: So. The striking thing of this paper was that we found that the brain responds not just to word unexpectedness, but it responds differently to each type of unexpectedness.
Micha Heilbron: So here you can see the Trfs.
Micha Heilbron: and you see that, for instance, the syntactic predictability leads to a very early frontal effect, whereas semantic predictability
Micha Heilbron: results in a much later and especially much more distributed effect, whereas phonological or phoning predictability into this early effect.
Micha Heilbron: And here I'm showing the Crfs. But we also did a analysis like Marlise described where you look at. If the model. If each of these variables explained unique cross-validated variants over and above the the baseline variables. And so we found
Micha Heilbron: independent, dissociable, and sort of unique contributions of predictions at each, like each level of abstraction.
Micha Heilbron: So given that it seems that when you simply listen to speech, the brain predicts at multiple levels of abstraction. The next question we asked was.
Micha Heilbron: How do different levels interact? And we already touched on this a bit during Q&A with malice. One way to look at that
Micha Heilbron: is, to look at different ways of doing phoneme prediction
Micha Heilbron: so normally how this works phonium predictions are sort of formalized by the idea of
Micha Heilbron: incremental word recognition. So as phonemes come in
Micha Heilbron: the cohort of potential electrical candidates that are consistent with the funding so far shrinks
Micha Heilbron: until the word is incrementally recognized. And you can cast this as a process of probabilistic phony and prediction by grouping each of the candidate's next words by their identifying next funny.
Micha Heilbron: So facts. Family fat would be at false 4. Fortunate would be all figure fin.
Micha Heilbron: finish fishing over here, etc.
Micha Heilbron: If you then attach a weight to each of these candidates you can do the math and derive a probability distribution over phonemes for each phone. Now, there are 2 ways of assigning the weights.
Micha Heilbron: and one way is to use this simple dilexical frequency or the unogram probability.
Micha Heilbron: So here you would take the probability of of an overall word overall in English. So, for instance, the word 4, it's very frequent. So we would have very high a priori likelihood.
Micha Heilbron: and the overall probability for the next phoneme is they're not highly biased by these high frequency words.
Micha Heilbron: Now, alternative possibility is that it's the the weight is not determined by the overall word frequency or word frequency, but by the contextual probability that we take from GPT. 2.
Micha Heilbron: And here the distribution would be quite different. Because if we look at this context. They say his father was a
Micha Heilbron: the word, for even though its very high frequency is not very likely because it wouldn't be even chromatical, they say its father was a. For whereas fishermen, even though over all, it's a very unlikely word
Micha Heilbron: in this context, it's very likely, because this is from the Altman and the sea. I don't know if you've read it, but it's all about fishing and and and fishermen. So Gbt, too, understood that, and knew that it was indeed very likely.
Micha Heilbron: so the overall probability distribution looks very different.
Micha Heilbron: Importantly.
Micha Heilbron: While the this frequency model is a single level model in the sense that it uses a short sequences of context so short within word, sequences of phonemes as a context plus a fixed prior, namely, the the overall probability of each word in the English language.
Micha Heilbron: The contextual model is hierarchical
Micha Heilbron: in the sense that you use not just short intra word frequencies of prior phonemes, but the prediction is also based on the longer sequences of prior words. So it's hierarchical in this way that the concept of
Micha Heilbron: hierarchical graphical models in Bayesian inference, for instance, you use 2 levels of context.
Micha Heilbron: And so what we did is we compared the sort of cross, validated prediction performance of both models and compared it to the baseline model, and what we found both in the Eeg and in the Meg participants.
Micha Heilbron: that the the models that include word phoneme unexpectedness were better than the baseline, but the models that include hierarchical unexpectedness work better still.
Micha Heilbron: and this is indeed very much in line with the sort of concurrent paper from Christian broadback.
Micha Heilbron: and where they actually found they asked a different question, and they found that sync the single level and hierarchical model seem to work in parallel. So we did. A very rough is one better than the other analysis.
Micha Heilbron: and it seems that the brain, indeed performs hierarchical computation. But there may, it may also perform single level computations.
Micha Heilbron: So the story may be a bit more subtle there.
Micha Heilbron: But overall, what we found in language is that when people listen to natural language. The brain constantly predicts, it predicts at multiple levels of abstraction. And we find this very suggestive evidence that at least in the domain of phonemes, higher level context can inform lower level context, which is exactly what you would expect from this predictive brain idea.
Micha Heilbron: And with a master student we performed a similar study in music. So the setup was quite similar. We again did one Mvg. Study at the Donders, and graciously, thankfully used the data set provided by Giovanni, which is EG.
Micha Heilbron: And the setup was again similar. So people listened to Midi, which is converted to to audio.
Micha Heilbron: and then, at the same time, we have a transformer very similar to Gpd. 2 which analyzes the media, and then we ask if the brain responses to the musical notes or not words but hear. Musical modes are different as a function of the unexpectedness of the musical notes.
Micha Heilbron: And again, we found, as well as in the Meg Eeg as in the Meti data, we find these characteristic Trfs and we find that the brain response is indeed modulated by the unexpectedness of the time.
Micha Heilbron: also similar to what Giovanni and others found. The twist of this study was
Micha Heilbron: not the Trf. Itself, but was what type of information is used in the prediction.
Micha Heilbron: So here we use not just one neural network, but we use actually many neural networks which each use a different amount of context. And what we found first is that the neural network that used the most context
Micha Heilbron: could predict the music best.
Micha Heilbron: And this is not surprising, but it was an important sanity check. So the more information we gave to the transformer the better it became at predicting the next node.
Micha Heilbron: But when we did the same analysis
Micha Heilbron: for the the data, for the for, the, not the ability to predict the music, but the ability to predict the brain. Using the unexpectedness of the music, you find a very different curve.
Micha Heilbron: So here we found that the the models that best predict the musical surprisal response was actually not the model that used the most context, but that used one of the fewest context. So context window of maybe only 5 to 10 words. So this is a very short.
Micha Heilbron: just a few seconds, just one or 2 musical motifs
Micha Heilbron: so contrary to language. What we found is that the musical surprise response actually seems to reflect very low level in the sense of short-range acoustic predictions.
Micha Heilbron: And yeah, this study was, by the way, very much
Micha Heilbron: spearheaded by Piers, who is now starting a Ph. D. And I wanted to mention him specifically
Micha Heilbron: alright. So in the last study, I wanted to highlight.
Micha Heilbron: I wanna take this idea to the visual domain.
Micha Heilbron: And here it's just the the visual perception of naturalistic images. And here the situation is quite different.
Micha Heilbron: because when you simply see a natural scene, the complex image like this, there's no temporal prediction.
Micha Heilbron: however, one form of prediction that might be there, and that is very important in sort of classical, predictive coding theories is spatial prediction or pattern completion.
Micha Heilbron: Now pattern completion is pretty easy to think about in very extreme cases, like a checkerboard, where, if you perceive only a small part of the stimulus, you can perfectly pat and complete and predict the rest.
Micha Heilbron: But how can you probe this in more naturalistic images? Now here we took an idea inspired by a paper from last year from neuron, which is very nice paper.
Micha Heilbron: And the key idea is that it relies on receptive fields. So just to make sure that everyone understand what I mean by that. The the visual cortex is sensitive to only these small apertures, small parts of the visual world.
Micha Heilbron: and if you have a dataset where you know the receptive field for each voxel or for each neuron, like in the Allen Institute's visual coding. Dataset of the Mouse visual cortex.
Micha Heilbron: What you can do is that you can take out. You can mask out the receptive field from the perceived image.
Micha Heilbron: Take a generative model that is trained for filling in, for, for impending, for reconstructing, to reconstruct and to predict the content of the receptive field. and then compare the predicted receptive field to the actual receptive field to compute the visual predictability.
Micha Heilbron: And then you regress that again on the brain response by taking into account lower level visual information like
Micha Heilbron: the acoustic kind of the visual contrast. importantly like before, you can compute not just the overall predictability. But you can compute this predictability at multiple levels of abstraction.
Micha Heilbron: So if you take your actual receptive field patch and your predicted receptive field patch, what you can do is extract the hierarchy of features.
Micha Heilbron: Here. We did that with a convolutional neural network you could theoretically also use something else.
Micha Heilbron: And in CNN. The early layers extract simple patterns like edges and lines, whereas later layers extract things like textures and object parts some more high level features. If you do the comparison at each level, you can compute not just how unexpected or how unpredictable the visual information is overall, but also at what level of abstraction, like at the level of simple edges or textures or shapes.
Micha Heilbron: etc.
Micha Heilbron: And so, just to make sure they understand what's going on. The data set here is a data of mice viewing the large full field images, all for 250 ms at the same time.
Micha Heilbron: and each image is repeated 50 times in total. That's like 6,000 trials. And you have thousands of neurons.
Micha Heilbron: All right. So if we look at V one, if we, the average response looks like this, it's the blue line.
Micha Heilbron: We see that when a natural image comes online, the the the fry roommate starts to peak.
Micha Heilbron: And so the red line here is sort of analogous to the Trf. Although we didn't use a Trf and method, but just linear regression. But it's the effect of the the average response to a natural image plus one Sd of the unpredictability.
Micha Heilbron: and just for comparison, the the gray line here is the effect of the average response, plus that of local contrast in the preferred spatial orientation and spatial frequency.
Micha Heilbron: So it's quite interesting, as you can see, that unpredictability modulates the response, but it modulates the response only later in line with the more recurrent process, whereas a low-level contrast modulates the immediate feed-forward sweep, and it has also stronger effects.
Micha Heilbron: If you do that for each convolutional layer. So for each level of abstraction. You see that
Micha Heilbron: overall each layer seems sort of the unexpectedness of each convolutional layer modulates the brain response.
Micha Heilbron: but especially the highest convolutional layers. So if we now summarize over time and look at the effects, the level of abstraction in terms of their effect. Size.
Micha Heilbron: But we find is that indeed the later layers of Alec Smith have the largest effects. and if you're a vision scientist, you may not be a bit puzzled, because this is indeed visually early. This is v. One.
Micha Heilbron: So we see that the the most most abstract feature on predictability has the largest effect.
Micha Heilbron: We know from v. One that v. One itself is sensitive to low, level features.
Micha Heilbron: So indeed, if you do, the classical sort of decarlo encoding approach where you so we're not thinking about unpredictability now, but just the sensitivity to the visual features we see, we see exactly the opposite curve. So v. One neuron neuro responses are best predicted by low, level features.
Micha Heilbron: But high level unpredictability.
Micha Heilbron: So you see exactly the opposite June. which is kind of interesting.
Micha Heilbron: Oh, yeah. And then, sort of as a last tidbit. All this work, by the way, is still very much in progress, so it may still change a bit
Micha Heilbron: as we do some additional analyses. But one thing that I didn't want to ignored today is the laminar analysis.
Micha Heilbron: So in this data set, the myas have these neural pixels probes in their visual cortex, and they measure not just
Micha Heilbron: at one visual and one cortical location, but they measure all throughout the cortex. So what you can do is look up for every unit that you record where they are in cortex, and then sort of
Micha Heilbron: very roughly subdivided them into superficial or deep cortical layers.
Micha Heilbron: And if you do that, you find that the effect is much stronger in the neurons that are classified as superficial than those in deep. But this is only for the feature, unpredictability.
Micha Heilbron: We don't find this for contrast.
Micha Heilbron: So it seems that this unpredictability effect is not just tuned inversely to what you might expect from v. One. But it's also strongest for the superficial layers compared to the deep, which is exactly in line with many of these theories about the predictive brain.
Micha Heilbron: Alright. So if you come back to that question, so generalization. do we find that indeed the brain always predicts
Micha Heilbron: so what did these studies so far show, including others that in language, yes, in music, yes, in vision. Yes, we find predictability effects and and continuous prediction in all these domains.
Micha Heilbron: If we don't ask the second question.
Micha Heilbron: what is actually in this prediction we see that in language we find this hierarchy of predictions where the brain is constantly predicting at multiple levels of abstraction and higher levels seem to be able to inform lower level predictions.
Micha Heilbron: whereas in music we find strikingly low level in the sense of short range predictions.
Micha Heilbron: And then finally, envision, we find these higher level predictions that are then fed back to V one. Although I have to say that yeah, this visual, this vision project is still work in progress that the exact conclusions may still change
Micha Heilbron: alright. So, summing up
Micha Heilbron: the rise of generative Iai AI enables stronger and more precise tests of the predictive brain hypothesis.
Micha Heilbron: They so far reveal ubiquitous prediction in many domains and across levels of representation. And to me, at least, this really showcases the power of AI to change the way we do cognitive neuroscience.
Micha Heilbron: And yeah, but the last point is just,
Micha Heilbron: it's my own personal view. But I think this is really the future for the study of the predicted break.
Micha Heilbron: So with that, I just wanted to thank you, of course, for your attention and all my co-authors, and especially the people who provided all the open source, software and data with special mention here, Giovanni and the Lol lower lab more generally.
Micha Heilbron: because without you it would really would it definitely possible. But I've been doing over the past years.
Micha Heilbron: So with that, thank you very much.
CNSP: Well, cool talk. Really like it. I'll anyone have any questions we have.
Micha Heilbron: Should I stop sharing?
CNSP: Oh, you can. You can keep it up in in case you have slides that could possibly answer questions.
CNSP: So we have one for one of the attendees. I'll we'll let attendees ask questions first, and then, you know, I have a question. Giovanni has a question, so if there's no questions, we'll
CNSP: we'll ask some questions. But there's an attendee who has a question feel free to unmute if you'd like.
danna pinto: So I have a question about the language model.
danna pinto: Yeah.
danna pinto: So you talked about how? Ii mean. I know you use it as an example. But in the old man and the C example it sort of use the context I I'm assuming I don't know so much about the Gpt models. But I'm assuming it uses context that it finds online to sort of make the predictions. Is that
danna pinto: correct assumption?
Micha Heilbron: Yeah, definitely.
danna pinto: So would then understanding the human brain like, wouldn't that be a big jump? Because
danna pinto: the human brain doesn't have all of the information that.
danna pinto: like the the model, would have so like context, clues, would be more in terms of like the meaning of the sentence. But then, when you look at explosives that include all of the literature you know ever written.
Micha Heilbron: Wouldn't that be like a not equal comparison? Yeah.
Micha Heilbron: yeah, no, I'm really happy. You'll ask this question.
Micha Heilbron: because, yeah, I first had a slide on sort of clarifications and common objections, but I removed it for the sake of time. But one of these things it was actually really happy. You ask, because it's a question that comes up a lot.
Micha Heilbron: So in this
Micha Heilbron: approached where you use a generative model to quantify the statistics that you hope that you assume that the brain is tracking.
Micha Heilbron: It doesn't require assuming that the exact statistics computed by Gbd are exactly what is tracked by the brain. and what I often
Micha Heilbron: do is make a comparison to something like how we compute word frequency, for instance.
Micha Heilbron: and so for word frequency. We often have these algorithms that that we count the word occurrences in a huge corporate. and they are often much larger than what you read
Micha Heilbron: in your entire lifetime.
Micha Heilbron: But so this is not meant to be an exact statistic of what you exactly have in mind, but what people tend to find is that the better you objectively approximate this statistic on average, the better that you would track the specific predictions in the brain. And that's also what you find in language.
Micha Heilbron: Some are least mentioned before that she couldn't rarely find significant effects
Micha Heilbron: differences between different models like an Rnn. And an ngram. But in reading time, in the reading time literature, people have done this a lot, and what people always find is that the better the language model becomes so going from an Ngram via an Rnn.
Micha Heilbron: Via S. It's simple transformers up to like GPT. There's this monotonic, almost linear relationship between the quality of the language model and the quality of the fit. And that's probably not because the human brain is exactly making that prediction. Probably the human brain is doing something less sophisticated. It has also read read less and learned less, but the more it matches the objective statistics.
Micha Heilbron: the more, on average, it will be closer to the idiosyncratic statistics on individual participants. So that was a long-winded explanation.
Micha Heilbron: But I hope it makes it clear.
danna pinto: Yeah, it makes more sense. Thank you. Thank you.
CNSP: Hi, Mike.
EDMUND LALOR: this is Ed Laura here I had a question on them.
EDMUND LALOR: If I just trying to understand the you had a kind of an X shape for these predictions versus low, level contrast that the was sensitive to is that is that right?
EDMUND LALOR: So I'm I'm maybe you could just talk us through that again, because I was trying to understand how
EDMUND LALOR: what V one might care about in terms of low, level features. So if if you know, if if I think about predictive coding theory, surely it should be getting predictions
EDMUND LALOR: that are of predicted low, level features, so maybe you could talk me through this curve again. Oh, no, no, you you didn't misunderstand it at all, and it is also surprising, and it's also what I expected, even though this particular neuron paper already pointed a bit in this direction.
Micha Heilbron: It was also what I expected. So I was. I was expecting to find that early level, early visual cortex is sensitive to predictability of lower level features, whereas later visual cortex would be sensitive to the predictability of high high, level features, and that would indeed be in line with
Micha Heilbron: very strict or sort of cartoonist view of predictive coding. But what we find is the opposite.
Micha Heilbron: So I think you understood it perfectly. We find this inverted
Micha Heilbron: tuning functions, where, in terms of the load level features. V. One is indeed sensitive to the lowest levels of here you quantify its via the encoding performance of Alex Net.
Micha Heilbron: But in terms of the unpredictability, it's exactly the other way around. And
Micha Heilbron: and yeah.
Micha Heilbron: I can speculate why, that may be
Micha Heilbron: But in in terms of, if you've understood it correctly, I think you understood it perfectly.
Umhm. Okay, yeah, thanks.
Micha Heilbron: not yet. If there's not another question, I can also speculate a bit. Why, this may be the case.
Micha Heilbron: One thing that I find myself especially interesting is that there are now results in
Micha Heilbron: in, in, in machine learning.
Micha Heilbron: where people are impressed by sort of the performance of GPT. For instance. And they, they think, okay, if we can learn everything there is about language, just from predicting words. Can we also learn everything there is to learn about a vision just from predicting pixels. So they've trained these large models to predict pixels based on their current.
Micha Heilbron: And what they find is actually that models that have become really good at predicting pixels are really not so useful for high level visual tasks, and the only methods that work very well in terms of this self, supervised pre training our
Micha Heilbron: methods that explicitly, in some way or another, discourage the prediction of low level features and anchorage, the prediction of high level features. So what you see from machine learning is that the thing that's really useful in self, supervised pre-training is prediction of high, level features. And then what we see in the brain is that the thing that really drives early visual cortex is the predictability of high, level, visual features.
Micha Heilbron: So that's to me at least a striking parallel. And to me, what that may suggest is that this may so this may reflect a learning signal which we know from machine learning to work really well.
Micha Heilbron: but that's, of course, speculation. You could also come up with other explanations. But it's my best theory for why we may find this opposite tuning.
EDMUND LALOR: That's very interesting. Thank you.
CNSP: Joe. Did you want to ask a question, or should I?
Giovanni Di Liberto: Yeah, have a have a quick one. I don't
CNSP: go for it.
Giovanni Di Liberto: Music, hierarchy
Giovanni Di Liberto: have you tried to look into? You know your second point in the conclusions was that yeah, there is the hierarchy in speech.
Giovanni Di Liberto: and but in music you find this slow. small, you know short context to be the optimal. But have you tried to combine short, long context?
Micha Heilbron: No, no. So the you don't know. And so the yeah
Micha Heilbron: the answer is, no, I also really wouldn't know how you can do that actually, that this is why I like working with language so much, because we have this large inventory, and we know that sort of words exist. At least, if you believe words exist in the mind, we have syllables and phones, and we can really dissect stimuli at these multiple levels, and then explicitly. Look how one level informs the other. I wouldn't really know. So I know what a note is.
Micha Heilbron: but I wouldn't. I don't have the conceptual apparatus to know what a word would be in in in music. So if we could, II wasn't smart enough. Let Le let's move like that to to perform the same types of analysis. But what do you think? How can you test this music? What's the right level of currency?
Giovanni Di Liberto: Yeah, I mean, I'm just thinking, I mean, from a purely methodological point of view, you could have multiple features fit with different context lines. That that's the boring thing. The other thing is, is inspired from what Ed said to me years ago on classical music, that the first time you listen to it, maybe you just focus on the on on patterns that are short term. And then you start getting the the, you know, different levels of
Giovanni Di Liberto: of of context. And so maybe maybe modulating that on the participant
Giovanni Di Liberto: getting multiple repetitions of complex patterns, and see whether that curve that you have there changes, you know, having the same identical method you have there, but with different conditions or different. Yeah.
Micha Heilbron: yeah, no, that's true. It would require yeah. And and we wouldn't be able to use your download. Get your data and use such an open ended data set. But indeed, yeah, I agree with sort of very clever experimental design. I think you you can go there.
Micha Heilbron: And what did you mean with the first boring one you meant like a decoding analysis
Micha Heilbron: and then sort of comparing the decoding performance on a time result basis and relating that to expectedness or something. Or, Yeah, sorry, Adam, this is not as quick as I thought, and a bit like what you're doing language that would include multiple features.
Giovanni Di Liberto: multiple surprise vectors that were fit using different context lengths. You know what I mean.
Micha Heilbron: Reds you you could. You could either simply use that model and have different context lengths if they're independent of each other.
Giovanni Di Liberto: Sorry. And I mean they could be both. It just may. Maybe that's not. Maybe not with that model. I don't know. Maybe the other models.
Micha Heilbron: Yeah, we're not.
Micha Heilbron: I don't know. II do like the this effect, because I, initially pious, came up with the idea. And I was like, Are you sure, Piers, because I think the better the model is, the better it will fit the brain. Do you think we can have an interesting effect, and then piece like well, we don't know blah blah blah! And then we did it. And we found this really striking dissociation. And I never thought we we would have
Micha Heilbron: so yeah.
Giovanni Di Liberto: good. Thanks.
Micha Heilbron: thanks a lot.
CNSP: Thanks, Mike. I think I will forego my question might ask you offline for the sake of time
CNSP: suggestions in the chat. So thank you very much for that. Yeah. anyway. So I'll let we should move on Ola. I'll I'll let you take over the screen. Share for the sake of time.
CNSP: Alright! Can can you hear me?
CNSP: Great Ola? Bialas is a postdoc with Ed Lawlor at the University of Rochester?
CNSP: He's a scientist after my own heart, I'd say, as a grad student, he helped write a toolbox, conducting audits for conducting auditory, spatial experiments and analyzing auditory spatial data.
CNSP: and since joining the lab one of his endeavors, along with one of our grad students, Jendo has been converting the Mtrf. Toolbox into a Python library and leading an unfortunate charge in converting the lab into a python lab.
CNSP: His tutorial today will be full of simulations and interesting minutia, so II hope hope you'll enjoy it as much as I will. I'm looking forward to this talk from Ola, so I'll turn it over to him.
CNSP: No.
yeah, thanks, Aaron, for the nice introduction. And Hello, everyone
CNSP: you're not sharing yet. By the way.
Ole: yeah, yeah, about you.
CNSP: hey? I'm on the screen twice.
Ole: Can you see my screen now.
CNSP: Yep, all good.
Ole: Alright. So I'm I'm very happy that we already got you here to really nice talks about all kinds of different aspects
Ole: of speech and linguistics. Because I don't want to talk about any of this in this talk. Instead, I just want to focus on the basic signal processing aspects of of Trf models. And but before I dive into that.
Ole: the talk is called Fundamentals of Trf analyses in Python. And so I want to take a moment to argue, for why one should use python.
Ole: So for once, and most importantly, it's open source, meaning everyone can access it. It is one off, or even the most popular programming language. And to show that I've included this graph here from a project called Popularity of Programming Language, which quantifies the share of different programming languages over time based on some Google statistics.
Ole: And it's the data is on a log scale. So linear growth is actually exponential growth, and you can see Python is just on a ever steady rise, especially since 2,015, 2,016 triggered by the machine learning and AI boom. It's also easy to learn which is in big parts because it is so popular meaning. There's just tons of materials online.
Ole: And then, finally, it's a general purpose programming language, which means that all things equal, it might be a better idea to learn python than something domain-specific, such as Matlab, because there's just a wider set of use cases for the language
Ole: and for those reasons. And Gin and I wrote an implementation of the Mtrf. Toolbox
Ole: in python called Mtrf. Pi. We just recently published it in the Journal of Open Source software. And so our objectives in making the software were that the Api should be as similar as possible to the Ntrf. Toolbox.
Ole: so that people who already knew that would have a easy time using Mpr. Pi.
Ole: However, we also wanted to use an object oriented design because we felt it. The code will just be easier to use if all the important methods and variables you would be interacting with are packed into one class rather than distributed across many functions.
Ole: We use continuous integration, which means that we have a set of test scripts that runs automatically at any time. We edit the code to to check that. Everything is producing the results it should. And this is just generally a good practice in in software design.
Ole: But it's especially important in our case, because, of course, we want to make sure that running the functions in Mtrf. Toolbox
Ole: and in the end. The corresponding functions in Mtrf. Pi. Produces the same results if you run them with the same parameters, and we ensure that by using this continuous integration process.
Ole: We wanted to keep it lightweight with minimal dependencies. So it right now only uses core python functions and numpy, which is the basic
Python library for doing matrix computation.
Ole: and this also helps us to make the code compatible with all kinds of existing pipelines. And we, for example, did not introduce things, such as custom data types. Because we want to avoid that. People who want to use the code have to rewrite a bunch of their existing pipelines just to fit it in.
Ole: And then, finally, the
Ole: software should be easy to use and learn, which is why we provide a quite extensive online documentation and tutorials.
Ole: So let me give you a quick overview of the code.
Ole: and the package consists of 3 modules. For once the Trf. Class, which is the main class that the user is interacting with. There's also a module called stats, which features functions for cross validation and permutation.
Ole: And then there is a module called matrices, which contains functions to, for example, compute covariances and generate lag matrices, and so on. Those functions are usually not accessed by the user directly, but rather called on by the other 2 modules.
Ole: And then the Trf. Class takes a bunch of arguments, and those are kind of the Meta parameters that define what kind of model you're using. So the direction, whether it's a forward or backward model, the method of regularization or the metric that you're using to evaluate and and compare different models.
Ole: And then the important variables that you're usually interested in are contained in the attributes of the class, such as the weights of the Trf. The time lags and the regularization coefficient. That's 1 1 point where we deviate from the Matlab Api in Python. We can't call it lambda, because Lambda has already defined function in Python which definitely bothers me. But it is what it is, so we just have to call it regularization.
Ole: And here's a little example of how to use it. I use this little function called load sample data, which downloads and returns about 2 min of eg. Recording, along with a 16 band spectrogram of the speech the participant listen to.
Ole: And I create an instance of the Trf class. So here, using empty parentheses, meaning it's just going to use the default parameters which will create a forward model which, with the Ridge regularization
Ole: and then use Tearf train, which is the equivalent of Mtrf train
Ole: fitting the Trf. To the stimulus and response. And then here I use the plot method to get just a quick visualization of this nice spectro temporal response function. Here.
Ole: We also use the opportunity to implement some functions that are not part of the original Mtrf toolbox, for example. Here I use this function called permutation distribution.
Ole: which randomly
Ole: scrambles the stimulus and response trials, and then trains trf models on the on this permuted data
Ole: and returns a correlation coefficient, and repeats this over and over again here a thousand times set by this parameter and permute. And so you're getting a thousand correlation values
Ole: and the distribution of these of these values, the permutation distribution
Ole: reflects the effect size that you're expecting in the absence of any actual causal relationship between stimulus and response. And then you can use this distribution to compare your actually observed correlation to and assess the significance of your result.
Ole: So now I want to move to some demonstrations in the tutorial. I'll be using actual brain recordings. Also, actually, the old man in the C data. So that's just kind of the canonical data set for doing these kinds of analyses.
Ole: apparently. But here I want to just use a simulations.
Ole: because the simulations are a really useful way of testing our intuitions and our understanding of these kinds of models and all the code I'll be using from now on to generate the simulations and the plots I've shared with with the Cns team, so I don't know if you already got it, but if you wanted, you should be able to get it and play around with all of the parameters yourself
Ole: erez agmoni. So I'm using a wavelet here to simulate neural responses, and a wavelet is generated by simply
Ole: multiplying a sinusoid with the Gaussian
Ole: erez agmoni in in time, frequency analysis. We also use wavelets rather complex wavelets, but here I only use a real valued 100
Ole: and the neural response, then, is simulated by con taking a stimulus which is here just some square pulses that are randomly spaced and vary in their length.
Ole: and then we convolve that stimulus with the wavelet, meaning, we slide the wavelet over the stimulus and multiply it with the stimulus at every point, and then for each.
Ole: for each multiplication, we sum all the points and get 1 point of the response. So this basically means we're filtering the stimulus with the wavelet, and the result is our stimulated brain response, which looks like this. So whenever there's a pulse, you get like this one
Ole: mit ctl, and thing here which kind of looks like the wavelet, or like a stretched version of the wavelet. And then, if you have multiple pulses close to each other, the single responses kind of start to blend into each other. So if the distance between the pulses is less than the duration of the wavelet.
Ole: and then we can use the Trf on the simulated response to see if we can recover this wavelet.
Ole: So we're going basically going to act as if the wavelet is not some approximation, but literally is how the how the brain responds to the stimulus. And then we're
Ole: we're checking whether we can use the Trf to recover this ground truth from the data. And in this scenario I'm showing here where there's no noise. There's just the stimulus in the wavelet.
Ole: We can use ordinary least square regression, meaning, we can set the regularization value to 0 and just use Trf train
Ole: to fit the model and then use T. Rf, to predict to get a prediction of the simulated response. And on the left side I'm showing the wavelet and the simulated response. And on the right side I'm showing the Trf reconstruction and the Trf. Prediction. And you can see they're identical. So we're able to perfectly recover the the wavelet in the absence of noise.
Ole: But so now what happens if we start to introduce noise into the picture?
Ole: So if we start adding normally distributed noise of increasing amplitude, then at some point this Trf is starting to break down. So here you can see at minus 30 dB, SNR, it's really hard to still pick out the shape of the of the trf from among the noise
Ole: and minus 30 dB, isn't that noisy? So here in the bottom you can see the the noisy response, and even with the minus 30 dB, just by eye, you're still easily able to to pick out the stimuli from among the noise. But the the ols. Regression is not
Ole: so. We can circumvent that by starting to use regularization. So you can call the Trf dot train function.
Ole: and instead of using a single value for regularization. You can just pass a list of values here. I use a log space array. and then what will happen internally in the Trf function is that it will train a Trf. For each value in this list
Ole: and and test them, using cross-validation, and then select the value that gives the best model.
Ole: the best model, according to the metric you're using, which per default is Pearson's correlation between the predicted and the actual response.
Ole: And then what you're seeing here on the right plot is the optimal value of the regularization coefficient lambda plotted against the signal to noise ratio, and you can see that the the lower the SNR is meaning, the more noise there is, the ha, the higher the value of optimal, the optimal value of lambda. So the more noise, the more
Ole: more regularisation. And with that regularisation we're able to recover the Trf.
Ole: Mit Ctl, and even at much higher noise levels like at minus 40 dB, and here in the bottom you can see the noise response to give you an idea what minus 40 dB. Looks like. So at minus 42
Ole: erez agmoni the response, here you can see the clean response in yellow is really buried in noise, but still we can get back our trf 20.
Ole: Of course it. It looks pretty. Mikey.
Ole: and and that is because we have a lot of high, frequent noise in that signal. Usually in in Eeg we filter out the the high, frequent signal. But here, since it's just plain white noise, there's a lot of high, frequent power
Ole: mit Ctl and one way to to help with. That is to use what we call Tekonov regularization. So Tekonov regularization penalizes the square difference between 2 neighboring weights in the Trf. Meaning it kind of forces the neighboring points to take on similar values.
Ole: and this usually leads to a smoother solution. So here, even at the minus 40 dB, we get a really good approximation of our original wavelet from the Trf, the other other than that, it's the same thing in that. The regular amount of regularization just kind of
Ole: linearly increases on the log scale with with lower SNR,
Ole: but there's also a downside to the Tikonov regularization, which is that if you use it in multivariate models, it can cause leakage between the features, which is something I'm gonna come back to in a few slides.
Ole: So so far we've only used a single predictor meaning. We've used the univariate model. But of course, one really cool thing about these models is that they're able to incorporate multiple predictors and disentangle their relative contributions to the signal.
Ole: So what I did here is I simulated the second stimulus, which is just a sinusoid. and and added a second differently shaped wavelet.
Ole: and then I convolve the the green stimulus with the green wavelet and the yellow stimulus with the yellow wavelet, and II just sum the result of those 2 convolutions to get this stimulus here, which just looks like another sine wave with with with some spikes added on top.
Ole: And then again, in this noise, free scenario, we can use the Trf. With both stimulus features, and we can perfectly recover both of our wavelets, meaning the Trf. Is able to disentangle the 2 contributions, and the weights actually tell you how big the effect of each feature over time is on the signal.
Ole: But so now, what if the stimulus features are confounded? So here I created what in causal analysis is known as a fork?
Ole: So 4 happens when one variable z causes both x and y.
Ole: and that causes, and that makes it so that x and y are correlated erez agmoni. But this correlation does not reflect any causal relationship, and in fact, the correlation disappears once you account for the common cause. Z. One.
Ole: So here z is this sinusoid. and then the another. The other stimulus X
Ole: generated by just setting it to one wherever Z exceeds a certain threshold.
Ole: and it's 0. Otherwise, meaning the value of x completely depends on Z,
Ole: and then the neural response Y here is created by convolving z with a wavelet, meaning X does not affect the neural response. The neural response only depends on z one.
Ole: And so now, what happens if we train 2 Trf models to predict the neural response, one using stimulus feature Z. And one using stimulus feature. X.
Ole: Well, what if if we use Z, we'll just get again our perfect reconstruction of the wavelet. This is just exactly the same scenario I showed in the beginning. If we just ignore X, however, if we only consider stimulus X, we'll still get a somewhat weirdly look in trf.
Ole: but that's definitely not random. And this is because X is actually correlated with the response. It does not, reflect any causal relationship. But you're definitely able to predict
Ole: the response based on X. So if you're just interested in prediction, it's totally fine to use X.
Ole: Now, the interesting thing is that if we then use both Z and X
Ole: in one common feature space, so here I create one stimulus by just stacking those 2 stimulus features, Z and X into one matrix and then use that to train the Trf.
Ole: now we're able to disentangle the contributions. And this is what I'm so now all the weights for X are 0, and this is what I mentioned in the beginning, that the the correlation that's caused by the fork disappears once you account for the common cause. Z.
Ole: But so earlier I mentioned that Tekonov regularization can result in leakage between the features. And now I want to show you what that means. So here, if we use Tikanov regularization, and we start using increasing amounts of regularization. So we increase the value of the regularization parameter from point 0 0 one. Up to one.
Ole: We can see that the more and more we're regularizing with Tikanov
Ole: erez agmoni, the more the Trf. Or X starts to approximate the Trf. Or z whereas here, with lambda equals one, we might be tempted to think that they're both about equally important
Ole: erez agmoni for generating the response, which, of course, is not true. We we know this in this case because we simulated it that way.
Ole: And this is the reason the reason why
Ole: it's usually not a good idea to use Teaconoff for multivariate models, although this effect is not completely absent. If you use Ridge regularization, so you need to. You need to crank up the regularization much more with Ridge to get nonzero values in in X, but it's but it is just technically possible.
Ole: So I would argue that it's always a better idea to just not throw in every feature possible, and let and let the the model decide which features are important and which not, but rather a priori, based on theoretical coni. Considerations decide which features are important. So here the best possible scenario is, if you can already a priori decide that X should not affect the neural response. And so you're just gonna leave it out as a predictor
Ole: and with that I want to thank you for your attention. Again, I would
Ole: mit Ctl and encourage you to check out the online documentation of the project which contains tutorials on all the different aspects of Trf analyses. You can find it at the website called Read the Docs, or just via the projects Github.
Ole: And also, if you're interested in contributing to this pretty new project. Please get in touch both Jin, who you can see here in this picture, and I are very happy to receive any contributions.
Ole: and that's it. Thank you again. And I'm looking forward to seeing some of you in the tutorial
CNSP: cool. That was a lot of great minutia. Thanks, Ola. So if anyone has any questions please feel free to ask.
CNSP: I'll hold my question until others have a chance to ask.
CNSP: But I didn't realize that you had a cat, Ola. It looks like it is not my cat. It's just some random cat in Greece on a holiday that walked like next to me.
CNSP: bummer. I thought it. It looked like Cosmo.
CNSP: Alright. Well, I'm gonna go ahead and ask my question. This is the first time I've actually ever seen the taking off regularization, demonstrating like that that leakage
CNSP: which, thanks for that. I I've always been like aware of it, like superficially and always had like kind of thoughts. And I wonder if I could get your thoughts on it, too? Is it possible, do you think.
CNSP: to implement a way in which Tikanov doesn't leak across channels
CNSP: for
CNSP: other features. Is it in the way that the X design matrix is constructed? Or is it just because there's something next door to it. It's gonna
CNSP: like influence it.
Ole: III really don't have a good explanation of that. Ii think my understanding of linear algebra is insufficient to to do that. But that's by the way, one thing which I think is a great reason to use simulations. If you can't kind of decide it from first principles.
Ole: A good idea is just to try it out in the scenario where you know the brand truth and see what happens. So no, this is actually also just something II played around with to to find out. But I don't have like mathematically grounded answer for you.
Ole: I'm sorry.
CNSP: I just
CNSP: I was gonna see if, like Nick had any, input I just realized that Nick has been a participant this whole time, and not a panelist. Sorry about that, Nick. But if if you have any like
CNSP: thoughts on that like I think Mick's is going to be in
CNSP: in your tutorial. So maybe you could say some words about that there, and we can figure that out because I think that'd be an interesting thing to figure out
CNSP: any other questions.
CNSP: Well.
CNSP: there's a link.
CNSP: Oh, Giovanni, put in the chat. That. I don't know if you knew this, Ola, but we've integrated your Mtrf. Pi into our web browser.
Ole: Oh, that's cool. Yeah.
CNSP: Think it's probably running completely on that right, Joe.
CNSP: Yeah, it's it's all. It's all python, unfortunately.
CNSP: anyway. So for the for the sake of time, we'll we'll cut it there. I will hang around in this in this zoom room, and then some people will be in the tutorial zoom room
CNSP: to answer any questions about setting up
CNSP: your environment for the 2 tutorials. So from here we'll go to the tutorials. Everyone think, Ola, for us people are doing that in the chat. There'll be 2 links that were sent to you. And so you'll just choose one of those links to go to.
CNSP: and we'll meet there in about 15 min. Okay.
CNSP: thank you. To all of the speakers and and to all the participants for interesting questions. We'll see you in a few.