You can perfectly see my screen.
Yes.
Okay, great. So, in the next couple of 20 min, I want to give you a brief introduction.
About linguistic speech tracking by the brain. And I will start with a very, very general introduction.
To even just neural speech tracking. I know there are a lot of experts in the audience, but just to make sure we're all on the same page.
So whenever you have an issue with hearing, the ideologist which will test how much speech you can understand can rely on an extensive test battery which holds for example, cure to an audiometry or sentence recall tests.
This says they have some clear disadvantages. Of course this pewter aromatries they work with tones and tones you cannot understand and this sentence recall tests They actually work with really standardized and controlled sentences which is nothing like real listening environments.
Additionally, if you fail on these tests or cannot perform these tests, And then the outcome does not allow you to pinpoint the origin of the deficit.
Is it situated more peripherally and can the patient be helped to the hearing aid or is it situated more cortically and where maybe language treatment is a better option?
Additionally, there are also or another drawback of this test is that they are not inclusive.
Not everyone can perform these task. You need to have attention and you need to be able to give feedback.
But sometimes this is not possible. Think about working with very young children. Or patients which come off for example.
So that's why our research group and many others and tried to go towards a more objective way of testing speech understanding.
Meaning that we look at brain signals to see or to try and see whether or not someone understands speech.
And to do so, we rely on the sentiment of neural checking. So very quickly, I know you've heard this yesterday.
But when you whenever you listen to speech and you would meanwhile record brain signals, then you can actually trace certain characteristics of the speech back in the brain signals.
And previous studies showed that using specific features of the speech such as acoustics, they can really give you insight in how much speech you can understand.
So if neural tracking to the acoustic feature of the speech is higher, the speech is better understood.
But then comes issue does neural checking to acoustics? Means speech understanding. Well, that's not necessarily the case because we also observe.
Like significant acoustic neural checking whenever you listen to music. Or, whenever you listen to the ignored speaker in a cocktail paradigm.
So you cannot say that if you have acoustic checking that there is a successful speech understanding it might be that acoustic speech tracking is necessary in order to understand speech but it's not a cause allergy.
So before we delve into the linguistic tracking stuff that I'm really eager to talk about, I want to first take a step back and go several conceptually conceptually about methods that we use.
And how this method actually works. So basically we have a participant coming to our lab and we make them listen to stories or podcasts.
And in the first step, we, estimate a brain model. So we accept some speech and representations.
Think about the envelope and acoustic speech representation here shown in the pinkish line. This can also be duplexical features.
So it's a classical features, denote the onset of a word or phoning and those are indicated by the the pulses so whenever for example here whenever you have a phoneme there's a pulse and disposals are always of the same height.
And then you have the, linguistic features. So basically these are also features with a lot of posts, but this time the amplitude changes.
Depending on how much information it's, but more details on those literacy features later. So whenever you have extracted the speech representations, you have of course also the, recorded brain responses and then in the next step you basically link those speech features which the recorded brain signals.
And that's to obtain this temporal response function.
In a second step, you actually have this estimated brain model. You will use this, TRF.
Of response function. To in order to extract a measure of neural tricking. So first again, you extract the speech features and then you do a convolution with the temporal response function.
. sorry, a convolution is basically a multiplication over time Just as a quick recap.
So we have a feature here. We have a TRF. Is multiplication over time or convolution works as follows.
So Whenever there's an impulse, you see the waveform of the temporal response potential.
No, whenever the signals become more complex, you can predict more complex signals. For example, here you have 2 impulses next to each other.
So the 2 TRFs are very close to each other. So the outcome of the multiplication over time or this convolution will be a summation of the 2 signals.
On the other hand, these waveforms can also be out of phase, so there's like a cancellation of this wave here.
Now, if you also very impulses heights or amplitudes, then you can also, yeah, this also has an effect on your predicted signal.
So using this multiplication over time, we have actually our speech feature that we extracted. We have the temporal response function which tells you how the brain responds to the speech feature using this multiplication over time.
We're basically giving or predicting a brain response. And evoked by the speech. We, we not only have the predicted brain signal, but also the actual brain signal, and that allows us to calculate a prediction accuracy which is a measure of neural checking.
It's basically a correlation between each channel and of the predicted signal with the actual reported signal.
I hope, this very brief introduction was brief enough for the experts, but introductionary enough for all the beginners or for people.
We're unfamiliar with this. The takeaway is you have 2 important outcomes. You have this simple response section which tells you how to bring response to the speech features and you have this prediction accuracy which tells you how well the brain checks the speech.
It's the measure of neural checking. And this you have for every or both measures you have for every sensors that you use.
So using this approach. I will, I try to answer a couple of questions and I will cover these 3 questions during the talk.
So the first question is which speech feature should we actually use whenever you want to investigate linguistic processing by the brain.
And then the second and also the church is whether the speech features really reflects speech understanding because we tested this via 2 different paradigms.
And I would like to go into them. So let's start with this first question. For this we, tested, 29.
You know, normal healing participants, we listen to a really long story presented in silence.
At the beginning, of my PhD, the first, studies were published using linguistic speech features.
So as a starting point we We made like a scheme of all the speech features that we were aware of.
That used linguistic speech features or other features. And we wanted to test. Whether these intrusive speech features at different levels so you have at voting level so which characterizes the information at the level of phoning at the word or even at the level of the context.
Whether this linguistic speech features actually. Explains more variance in the EGG signal in the neural signal.
On top of, acoustical and seplexical features. Now why is this important?
If you look at speech features, they are all derived from exactly the same signal. And this causes that these speech features are highly correlated.
So whenever you have, a burst of, acoustic energy. You know that's the envelope is going to be like a high waveform which Similarizes this burst, but at the beginning of this burst, There there is phoneme on such.
And also an impulse of how much information this morning. There is. So if you would only test a linguistic feature.
It is still highly correlated with the response to, for example, the envelope or purely the acoustics of signal.
So to make sure that. We estimated linguistic tracking. In a way that it's not biased by acoustic responses.
We used, a specific framework. I, will call it the subtraction approach.
So basically we, calculated 2 types of models. We complete model, this complete model. Contains acoustic speech features.
Seplexical and speech features, sort of volume and word on sets, as well as these linguistic speech features.
And then we subtracted the baseline model which Actually included all control and variables. So these acoustic features and these are flexible features and by taking the difference you can and we could investigate the added value of these linguistic features.
So how much does a model improve? In predicting the neural response whenever we add these linguistic features on top of the acoustics and seplexical features.
This way, although it's not the perfect approach of course. This way we try to minimize the bias towards acoustic response.
But you see that the responses to linguistic speech features are really, really, really small. So that's also very, very challenging.
But using this framework, we applied it on all the existing linguistic features or features that were already discussed in literature.
And then we, we observed that not all of them survived this approach. So, We, we observe this, Linguistic features.
So for in surprise, and port entropy and these linguistic features at work levels, so word surprises and Which frequency those were significant.
So for the for the rest of my talk or we would be focused on the rest of the studies that we that we did we focus on those 4 linguistic features.
But for this stock, I will only, focus on what's surprising because it's a very, intuitive feature.
If you want to know more details about the other features, that's, content for the workshop.
So basically, word surprising is how predictable or surprising a word is given the previous words.
So for example here this sentence even in the darkest of times If one only remembers to turn on the It's not surprising you're going to expect light sort of surprise or value of night is rather low.
If it would be, to turn up the music, that would be surprising because it's changes the concept of or the context of the sentence that you had in mind.
So this previous results we obtained by looking at prediction curses. So by looking at whether the linguistic features improve the prediction accuracy, so the measure of neural checking.
If we also had this other important outcome of the model. So the temporal response function which tells you how to bring a response.
And for, word surprises, it looked like this. So, You have a clear negativity around, 400 ms, which is really prominent for these central, channels here.
And it's, strikingly, so this is very similar to the N. 400 paradigms.
Even in this case, it's using continuous speech and not. Carefully created control sentences.
So in his first city we identified which speech features we should use. The second question was, well, These speech features, they characterize so many linguistic information.
The today really represent speech understanding. So to answer that, we, manipulated the, the amount of speech understanding by and speeding up or manipulating the speech rate.
And we did this in, 18 young, normal hearing participants. And in total we had, around 6 speech, speech rates and each speech fragment was, around 10 min.
So to give you an example, I hope that I am quickly going to check whether My. So it is shared as well.
Okay. Hope you can still see my screen and hope you can hear. This example.
Mary, Igil, you're right, may, out a name.
So I'm that speaking, so this is really slow speech. For some of you it might be intelligible but for our participants it was understandable.
So this was a slow condition. This was a fast condition. It's very difficult to understand.
No.
No participants actually reported that it was well intelligible. So what happens with linguistic tracking? What happens with this added value?
Of these linguistic speech features. So the red, dot here indicates the significant channels. Or the channels which drive the topography to be significantly different from 0.
And in the slow and normal condition, You see in these conditions that it's you see these rich channels.
Meaning that's the linguistic features do have an added value, but as it becomes faster and faster, less red circles, so just added value of linguistic features.
It disappears. I know, then again, you can also look at the support of response function and how to bring response.
And then. You see, this soon. Although it's super noisy because this time we only have, 10 min of data, well, previously we had 45 min of data.
But you see in the yellowish stones we see a significant and negativity. Which is only present for the slower as slower speech rates, which were intelligible by the listeners.
Also, this characteristic topography, it fades away and the faster the speech becomes. So this gives us the first insight like, okay, maybe linguistic speech tracking really gives an indication.
Of speech understanding. But there's a big butch. Because it's.
Paradigm it used it's used artificial speech so we artificially sped up the speech so what actually happens with natural speech.
So in this case. We did a new experiments which 19 young normal hearing participants which listened to different fragments or Dutch fusion and a scrambled word list and then we assess the effect on linguistic.
To give you again an example. So for that, listen, this will be, intelligible.
From Bahavard, heaven object the ripper.
Then the Fijian story.
S of all Maddie did Benavid up tall for my atatu's tray.
So they're interesting about Dutch and Fusion. So it's, in the upper north of the Netherlands and it follows a very similar grammatical structure in the upper north of the Netherlands and it follows a very similar grammatical structure as Dutch.
It's nearly a one on map, one man, nothing. So that's why we chose, but the participants reported that they had around 50% of intelligibility.
But this is of course subjectively rated. And then a last example is the work list. Okay.
Blade for May in Ishoon is is a Nadin.
So all these speech fragments are spoken by the same speaker and they are also spoken in a very natural way, aiming to preserve the normal process.
Okay, so we can actually again, plot the same results. So can we. See add value of these linguistic speech features.
So now we had only 7 min and we only observe the significant channels for the Dutch story but not for the Fesian nor the word is condition.
Then in, looking at the TRFs, we also see only this prominent negativity around 400 ms for a Dutch story, they're not, but not for the Freedom or the wordless condition.
And again, this characteristic topography, it disappears whenever it becomes not intelligible. Yeah.
So this again, it gives us some insight that maybe linguistic tracking might be a good indicator on whether or not speech was understood.
But there are so clear, pro and cons. So because it's, like, or my results show that it might be an indicator for speech understanding.
So it can really pinpoint understanding and not just hearing. The disadvantage is that you require lots of data.
You You need. Quite some participants because currently we can only, look at it on a population level.
And also for, participants. There's a minimum of 10 min, but I would. Give you, like ideally the more Data, the better.
Because it's such a small effect, which makes it really, really challenging. But we're currently working on an approach to make this more robust.
But it's challenging. So in, my, my workshop. My goal is to really.
Make you able to craft your own speech features to make these phoning segmentations. Do this language model so you can create your own word surprising stuff and you can apply this for your experiments so that we can all accelerate signs, together.
So yeah, that's it. That's it. I'm happy to take any questions.
Cool, cool talk. Thanks, Marise. We have a question in the chat, but.
The the asker is going to ask themselves.
Hi, can you hear me?
Yep.
Yes.
My question was when you were looking at the, the different the difference between the fast and the slow.
Speech. Did you also look at the differences in just the acoustic model? To see if if changing this change the the tracking of the envelope or the other acoustic features that you used.
Yes, good points. Yes, we did look at, at the acoustics as well.
Maybe I can. I'm trying to So, I'm going to, hopefully you can see my draw, case it.
Yep. It's a great drawing.
Okay. Yes, beautiful.
So, anyway. What my point? I cannot proceed with this drawing. And that's technical enough, I guess, to manage zoom.
But anyway, what we saw us that's with speech rate, so this is increasing speech rates and this is a neural checking so in the beginning we see an increase with higher speed rates and if I'm not mistaken this effect was also seen yesterday so with, with higher speech rates in the beginning, there's higher acoustic tracking, which I think makes sense.
In the beginning, there's higher acoustic tracking, which I think makes sense. And because there are just more acoustic tracking, which I think makes sense and because there are just more acoustic boundaries to to track and to respond to.
So your neural checking improves, but then at the end where it's super, super fast we see again a decrease.
And this puzzle does a bit, so we went digging in literature. And then we could form, form some, hypothesis, but this is purely, speculations.
That this might be due to motor cortex involving that. And so when Whenever you listen to speech, your motor cortex is active.
And it sings a bit through how your lips would move. But when the speech is so, so fast.
You don't have this nip image and you cannot mimic on how your muscles would be able to pronounce this hence your acoustic tracking would be slightly lower when it's super super fast when it's unnatural, a natural to be.
Pronounced. So I hope that's clear and it answered your question.
Thanks.
Yeah, yeah, that was great. Thank you so much. And it was a great talk. Thank you.
Any other questions?
And if you wanna ask that you're more than welcome to, I can unmute you or I can ask it.
If you raise your hand, you can ask, but if you don't raise your hand, I'll just read it.
There you go.
Hi, Marlise. That was a really nice talk. I don't know, I think I've asked you this question before just to beg kind of linking across the different levels of.
Processing where you know some of the time if you're calculating things like phoneme, surprising and so on.
It's kind of word specific, phoneme, surprising, whereas obviously the phoning. This surprise of a given phoneme will be affected by how likely that particular word was to come up and all that kind of stuff.
So I don't know if you'd tolerate that any further or. Have them to say on that.
Yeah. So. Indeed, and there are like lots of ways that linguistic features can be improved.
We currently, calculate for phoning, for a words independent of the previous words so that's indeed a drawback of the of the current approach.
But there are papers, a fairly recent one. I brought back. Where he really, tried to.
Thank you.
Yes, see and which level, this morning, phoneing, whether how much, yeah.
Contacts Windows and, whether it's more top-down or whatnot effects.
I
I feel, using linguistic features that it's a bit like,
Our brain is not exactly the same. As a language model?
So there are. There will be ways to improve this. But I think it's very challenging to do it really in the right way because even if you take further context into accounts, how will it affect?
Thank you. Yeah, I. I don't know, I'm puzzled by this myself to turn to craft optimal.
Linguistic feature. I think this is a good. Starting points, but yeah, because for example, for, comparing different language models and rams which are the most simple ones which I usually use.
RNNs, LSDMs, which are able to have like way more context than this simple engram model.
Using, lots of, participants. And, 45 min of data, I cannot find a significant difference between those 3 language, the language models, although the complexity is really, really different.
So my takeaways like. This is the group, like legacy features are good proxy. Good.
Like, I don't think the brain is really a language. I hope that answer your question.
Yeah, that's lots of interesting stuff there in that answer. Thank you.
I have a question if no one else has one. I know Tom is, Big on, on creation and patents.
And I was wondering if, If your involvement or if the lab involvement in this sort of project sort of ends at like the basic science or if you all are working on implementing this.
Or audiologist or clinical practice. Like to any degree, any level of degree.
Well, I'm not in full in any patterns, so and I'm also not really aware of all the all those things.
So. If you want to use this stuff, you can use this.
Okay, fair enough.
Okay, Someone has a hand raise. Ola has a hand raised.
Feel free to ask a question or.
Ola
Can you hear me now?
Did you raise your hand, Ola? We can't hear you. Yo.
Okay. So my question was in the I think last experiment you showed like the contrast between word lists and the continuous story that the wordless doesn't show like an effect of words, like have you tried?
Like actively modulating the surprises of words in the word list. Like in the sense that, you know, if I have like a list where I say like, House, Cathedral, Bridge, Elephant or something like that, like the word elephant would be, would be surprising in that scenario and maybe that would be like a way we could disentangle sort of narrative, surprising from this sort of simplistic word
model, just word by word probability. Is that something you've looked into?
Right.
It's not something I've looked into, but it's very interesting.
Because I think the and 400 each response that you will get using your like, which are actually semantically related and are certainly unrelated.
Those will yield a way bigger response than narratives. That's my good feeling. Because Those like you are really manipulating it in order for the participant to be surprised.
While it is not really. Natural, like no one is going to say to you that he just 8% which we sandwich with elephant.
So. Yeah.
Hmm.
Yeah, I mean, I gave an extreme example now, of course, but, I mean, you could use language models to like carefully sort of titrate the amount of surprise in the word list so it kind of matches that of a of a natural story and then compared with you and then would you think the 2 responses would be similar or there would still be some difference between the 2?
Yeah, yeah. Yeah, okay. I, I made a wrong interpretation. No, I know.
I think then probably the effects. Hmm. Might be similar, but it will be very, very cheeky because you always need to be And so if you're in word list, like how's Cathedral?
Those words are, will probably have. Like similar words, probabilities or maybe like semantically they're in a very same category and those artificial settings will be also very difficult to recreate in a naturalistic story condition because in the naturalistic story condition we really have this grammatical structure with nouns and pronouns and verbs.
It's interesting to think about and maybe if you're really clever you can come up with a cool design that's I think it's where to wrap your head about it, but 100% sure what the outcome will be but it would be cool to check it out.
Okay, thank you.
Yeah, I agree. I think that's pretty full idea. I think there's a little bit of stuff to like.
Stink through that, but I think that's a great idea. Thanks, Marley. For that or with that, I think I will, will think morally.
Virtually at least. And move on. For the sake of time, I'm gonna steal the screen back from you, Morley.
So up next. We have our. He knows from. Like a higher breath.
So Michael is a brand new professor.
Assistant professor, let me get on to my.
Slide show there. It's brand new assistant professor of cognitive AI at the University of Amsterdam and is interested in how brains and artificial systems understand language and make sense in the world.
You're strained at the Donders Institute and the Max Planck Institute for Psycho Linguistics under the supervision of 2 highly regarded scientists forced along.
And Peter Haggart and wrote a provocatively titled thesis. At least I think maybe not so much provocative but Interesting.
She's here. I like the idea of the bi-directional window. Into things.
So I really wanted to put that up on the screen. Michael also published one of my lab's favorite Early Stads Labs favorite papers last year.
Not just because they really analyze some of our labs data. Which happens to be available on the CNSD website.
But in this paper you analyze word level predictions from a large language model. And showed, multiple hierarchical predictions.
So maybe getting at some of those questions that we're asking. Then there. That were dissociated in neural activity.
Which we just thought was really exciting that he could pull that out of our data and some other data.
So I'm really excited. To introduce our early career keynote. To be given by Mike. So join us.
We're welcoming him.
Alright, so. If everything should be. Working you should be able to see my slides now. Is that correct?
See you slides in here.
Yes.
Perfect. Thank you, so much. But thanks for the very kind introduction and for the invitation, of course.
It's an honor and it's a lot of fun to be here. It's also really nice and cool speaking just after the, because our work is very similar in some ways, at least this is a sum of what I will be presenting.
But I'll be coming from a pretty different theoretical background, but the methods are in many aspects quite similar.
So I guess that's also very nice and pedagogical for all the attendees. They can see very similar ways of working.
And drawing quite different conclusions or learning different aspects of the same types of analyses. Right.
So today what I want to talk about is a sort of more high level framework for studying the brain and how we can use both the types of naturalistic data analyses that we've been that we are discussing in these 2 days and modern AI developments.
And I can combine them. So now I hope that my clicker will work. Alright, cool. So the theoretical backdrop here is the idea of the predictive brain.
And the predictive brain is this sort of very high level term. For a family of models and theories really.
That all revolve around the idea that the brand neural processing in the brain relies on continuous prediction.
And mechanistically, this is formalized in most detail in vision. In particular around. Mechanistic ideas like predictive coding.
And here it's typically formalized as bidirectional processing cascade. Where the feed-forward component works something akin to a con of that in the sense that it extracts increasingly abstract features and this is then complement by top-down component, which conveys predictions from a generative model.
And so these predictions are constantly compared to what is coming in. And by making this comparison the brain computes a prediction error which is used both for inference at short time skills to try to make sense what is out there and longer timescales to learn to perform self-supervised learning.
And importantly, this is postulated to be economical cortical computation, which is I guess a fancy way of saying that it's something really quite general.
And something fundamental and something that is found all over the brand. So this should be a fundamental principle that applies not just to vision but also the hearing and the language and what have you.
Now, there is quite some evidence. That is in line with this idea. And that has made the framework pretty influential over the past years.
It's a way this is in the domain of visual perception. Very influential set of paradigms.
It's for instance statistical learning. So participants would then be exposed over and over to one stimulus predicting another stimulus such as here a jukebox predicting a clock.
And what is observed is that as participants are exposed to these regularities, a jukebox clock, hookbox clock, 2 box clock over and over.
As once stimulus becomes predictable, say the clock. You see that the brain responds differently. Much more weekly.
Than if it's unpredictable. And you see this in many domains. So for instance in language, in Malaysia, if you' to the N 400.
So when a sentence and expectedly, so I drink my coffee with cream and sugar. And the brain responds very differently and much more weekly than if the sentence ends unexpectedly.
Like I drink my coffee with cream and dog. Right, it's a time and again in all these domains, but all these types of evidence seem to point to is this predictive brain idea but the brain is seemingly constantly comparing what is going in.
To its own expectations.
Now, oh, this foundational work using, let's say, classical Gog Neural Methods.
Is of course very important and it has taught us a lot and it's the reason that I got interested in the predictive brain in the first place.
But there are some limitations of this. Sort of classical approach. And the first is generalization. So, but we now know very well that the brain indeed starts predicting in these specific paradigms, like 2 box clock to a box clock.
But it's not so clear is whether the findings in those twobox clock scenarios would also generalize the perception more broadly.
And this is actually very important for the predictive brain because for the predictive brain, the idea is not that the brain can predict.
But that it always does.
Right, so does it also apply to a more naturalistic scenario like when you're listening to music or you're watching movies or perhaps you just see complex images.
The second question is what is meant by a prediction? Because this predictive brain theory is a very general theory.
So according to the predictive brain, a prediction can be anything. So if you're predicting a clock for instance, you might predict some high-level idea or some visual prototype or perhaps even the very specific low level features of that exact clock.
No, of course, nearly these difference options, these different scenarios would imply a very different mechanism.
But it's often not so clear in these sort of classical methods what types of predictions are manipulated and that for what is going on.
No, there are 2 reasons for the developments in neuroscience and AI machine learning. That allow addressing both these issues.
And the first is the rise of generative AI. So for a long time, this notion of a generative model that predicted brain theory is talked about was quite elusive and people could write down the equations and it was formulated in a Bayesian brain framework, but no one could actually implement the types of predictions that we thought the brand would be making.
And this is not changing since generative models are actually working. So what for instance in language one sort of very striking example is in GPT 4.
And other sort of more sophisticated generative models. So these are models that can predict stimuli and and then they can generate text for instance.
It's the same is true for videos so they predict video frames and they could generate a video.
Now the second development is sort of the rise of neural data science and open science more broadly. So we now have these large data sets of very high quality, naturalistic images of naturalistic stimuli that is both sort of in the mouse or rodents like invasive electrophysiology and also the types of data set that we've been talking about today.
So what I've been doing is sort of combining these 2 trends. And then I call it the generative AI framework.
For studying the predictive. So how does that work? So instead of, so the setup in general is quite simple.
You start with stimulus, often a naturalistic stimulus and not a dupex block stimulus, but for instance in language could be an audio book or natural music.
Or simply, natural images. And these are simply shown to the brain and the brain activity is recorded.
At the same time, the same naturalistic image is analysed with a neural network and then specifically a generative model.
So a model that can predict for instance an expert or it can predict the next note in music.
And then these predictions can be compared to the actual stimuli to compute the predictability. And then we, inform the regression to test if indeed the brain response is different.
When this the stimulus is protectable for this unpredictable. And this regression can be depending on the the situation.
So if it's a time continuous naturalistic stimulus you would need decomposition approaches like implemented in the MTRF toolbox.
It can also be a NON deconvolution if there's no temper overlap. But the important aspect is that it's a multiple regression approach.
So we can include all the baseline variables like Marty's already talked about such as acoustics or in vision can be local contrast.
And on top of that, we add the expectedness estimates to test if the brand responds differently. To unexpectedness.
For low level conference. Not this has 2 sort of main advantages and the first is generalization, right?
So we can move beyond the due box clock scenario and test if even when you're simply watching a movie if the brand responds differently when stimulus.
It's, it's predictable. And the second, perhaps more interesting aspect, is that because we model the prediction instead of manipulating them, You can make the concept of prediction as precise as you want.
So you can friends this, compute the. Overall predictability of a word, but you can also separately model the predictability at different levels of abstraction, like the predictability of the meaning.
Or the predictability of the exact sound. And so on. So here you're only limited by your own modeling skills and imagination and you can make this concept of production arbitrarily precise.
So you can have as I like to go. As I like to go, you can have an unlimited conceptual resolution.
Sorry, for using really powerful and beautiful, open data sets, we have very nice temporal resolution, very nice spatial resolution and Only limited by your own modeling you can have very strong conceptual resolution.
Alright, so I, so that's still very high level. Now the first sort of study where I was approaching this framework and I worked out in the deal.
Was a study from a PhD, which was on language. And here we analyze. People listening to audio books and one of the datasets was actually The data set recorded by, I think, to Jeff funny and it was, yeah, graciously shared by the Lor Lab.
On another day at the set to say EEG, the MEG data set that we regarded ourselves at the D.
And to model the predictions. We use GPT 2 which is this large generative model which can predict upcoming word.
Not from GP to do what you get on the high level of predictability of work. So for instance, this is one of the sentence for one of the audio books.
It made the boy sad too. And then we can get from the model is a probability distribution over every potential next word.
And what we, so what we came up with in this, in this paper was a method to then disentangle this overall prediction into different levels of prediction.
So for instance, how does that work? You can see here that it made the by start to think the seat and note the hair to say to be to analyze all these predictions.
You see, even though GPT 2 is not exactly sure about the identity of the next word, All these are verbs.
So it's pretty damn sure that the next word will be a verb. You can do the same for semantics and for speed sound.
So you can compute not just how predictable or is overall. But also at what level, syntactically, lexically or phonologically.
So the striking thing of this paper was that we found that the brain responds not just to where the unexpectedness but it responds differently to each type of unexpectedness.
So here you can see the TRF. And you see that for instance, the syntactic predictability.
Leads to a very early Frontal effect whereas semantic predictability. Results in much later and especially much more distributed effect.
Whereas, phenomenological, phoning predictability, it's this early effect. And here I'm showing the TRF, but we also did.
A analysis like Malise described where you look at if the model if each of these variables explains unique cross-falated variants over and above the the baseline variables.
And so we found Independent, dissociable, and sort of unique contributions of predictions that eat like each level of abstraction.
So given that it seems that when you simply listen to speech, the brain predicts that multiple levels of instruction.
The next question we asked was How do different levels interact? And we already touched on this a bit during Q&A with Malise.
One way to look at that. Is to look at different ways of doing phone in prediction. So normally how this works, phoning predictions are sort of formalized by the idea of incremental word recognition.
So as phonemes come in, The cohort of potential lexical candidates that are consistent with the funding so far shrinks.
Until the word is incrementally recognized. You can cost this as a process of probabilistic phony in prediction.
By grouping each of the candidates next words. By their identifying next phony. So of fact, family fat will be at false for fortunate would be all figure if Finish fishermen would be, etc.
And if you then attach a weight to each of these candidates. You can do the math. And derive a probability distribution over phone names for each farm.
Now there are 2 ways of assigning the weights. And one way is to use the simple, the lexical frequency or the uni.
So here you would take the probability of an overall word overall in English. So for instance, the word 4 is very frequent, so we would have very high a priority likelihood.
And the overall, probability for the next phoneme is they're not highly biased by these high frequency words.
Now alternative possibility is that it's the way it's not determined by the overall word, word probability or at frequency, but by the contextual probability that we take from GPT 2.
And here the distribution would be quite different because if we look at this context, they say his father was it.
The word for even though it's very high frequency. It's not very likely because it wouldn't be even chromatical.
They say his father wasn't a 4. But as fisherman, even though overall it's a very unlikely word.
In this context, it's very likely because this is from the old man in the sea. I don't know if you've read it, but it's not about fishing and fishermen.
So GPT 2 understood that and knew that it was indeed very likely. So the overall probability distribution looks very different.
Importantly, while the this frequency model is a single level model in the sense that It uses short sequences of context, so short within words, it uses a phone, as a context plus fixed prior, namely the overall probability of each word in the English language.
The contextual model is hierarchical. In the sense that you use not just shorts intra word frequencies of prior phonemes But the prediction is also based on the longer sequences of prior words.
So it's hierarchical in this way. The concept of I brought graphical models in Bayesian inference.
For instance, you use 2 levels of context. And so what we did is we compared the, sort of cross validated prediction performance of both models and compared it to the baseline model.
I hope we found both in the EEG and in the imaging participants. That the models that include word phonium unexpectedness were better than the baseline but the models that include hierarchical unexpectedness work better still.
And this is indeed very much in line with the sort of concurrent paper from Kirsten brought back.
Where they actually found they asked a different question and they found that Sing the single level and the hierarchical model.
Seems to work in parallel. So we did a very. Rough is one better than the other analysis.
And it seems that the brand indeed performs a Rococo computation but they may it may also perform a single level of computation.
So the story may be more subtle there. But overall what we found in language is that when people listen to natural language, the brain constantly predicts, it protects multiple levels of abstraction and we find this very suggestive evidence that at least in the domain of phonemes, higher level context can inform lower-level context, which is exactly what you would expect from this predicted brain idea.
And with a master student, we performed a similar study in music. So the setup was quite similar.
We again did one MVG study at the Dunders and graciously, very thankfully used the data set it provided by Giovanni just EG.
And the setup was again similar so people listened to MIDI which is converted to audio. And then at the same time, we have a transformer very similar to GPD 2.
It analyzes the media and then we ask if the brand responses to the music or notes or not words but hear musical modes are different as a function of the unexpectedness of the music or not.
And again, we found as well as in the MEG, ED as in the ME, that we find these.
Characteristic TRFs and we find that the brain response is indeed modulated by the unexpectedness of time.
Also similar to Matiovanni and others found. The twist of this study was not the TRF itself but was what type of information is used in the prediction.
So here we use not just one neural network, but we use it actually many neural networks, which each use a different amount of context.
And what we found first is that the neural network that is the most context. Could predict the music best.
And this is not surprising, but it was an important sanity check. So the more information we gave to the transformer, the better it became at predicting the next night.
But when we did the same analysis for the the delta. It's for the for the not the ability to predict the music, but the ability to predict the brain.
Using the unexpectedness of the music, we find a very different curve. So here we found that the models that best predict the musical surprising response was actually not the model that used the most context but that used one of the fewest contexts.
So context window maybe only 5 to 10 words. So this is a very short Just a few seconds, just one or 2 musical motives.
So contrary to language, what we found is that the musical surprise response actually seems to be reflect very low level in the sense of short-range acoustic predictions.
And yeah, this study was by the way very much. Spearheaded by peers who is now starting a PhD and I wanted to mention him specifically.
Alright, so in the last study I wanted to highlight. I wanna take this idea to the visual domain.
And here it's just the visual perception of naturalistic images. And here's the situation is quite different.
Because when you simply see a natural scene, a complex image like this, there's no temporal prediction.
I have a one form of prediction that might be there and that is very important in sort of classical predictive coding theories is spatial prediction or pattern completion.
No, pattern completion is pretty easy to think about in very extreme cases like a checkerboard where if you perceive only a small part of the stimulus you can perfectly pat and complete and predict the best.
But how can you probe this in more naturalistic images? Now here we took an idea. Inspired by a paper from last year from Neuron, which is very nice paper.
And the key idea is that it relies on receptive fields. So just to make sure that everyone understands what I mean by that, the visual cortex is sensitive to only these small apertures, more parts of the visual world.
And if you have a data set where you know the receptive field for each voxel over each neuron, like in the Allen Institute visual coding.
Data set of the mouse visual cortex. What you can do is that you can take out, you can mask out the receptive field from the perceived image.
Take a generative model that is trained for filling in for in-panting, for reconstructing, to reconstruct and to predict the content of the receptive field.
And then compared to predicted receptive field to the actual receptive field to compute the visual predictability.
And then you regress that again on the brand response. We're taking into account another level of visual information like.
The acoustic, the visual contrast. And importantly, like before, you can compute not just the overall predictability, but you can compute this predictability at multiple levels of abstraction.
So if you take your actual receptive field batch and your predicted receptive field batch, what you can do is extract the hierarchy of features.
Here we did that with a convolutional neural network. You could theoretically also use something else.
And in CNN, the early layers extract simple patterns like edges and lines, whereas later layers extract things like textures and object parts, some more high level features.
If you do the comparison at each level, you can compute not just how unexpected or how unpredictable the visual information is overall, but also at what level of abstraction.
Like at the level of simple edges or textures or shapes. etc. And so just to make sure they understand what's going on, the data set here is the data of mice viewing the large full field images all for 250 ms at the same time.
And each image is repeated 50 times. In total, it's like 6,000 trials and you have thousands of neurons.
Alright, so if you look at V one, the average response look like this, it's the blue line.
We see that when a natural image comes online, the The fibering rate starts to peek.
And so the red line here is sort of analogous to the TRF, although we didn't use a TRF method, but just linear regression.
But it's the effect of the average response to a natural image plus one as D of the unpredictability.
And just for comparison, the gray line here is the effect of the average response plus that of local contrast in the preferred a spatial orientation and special frequency.
Sounds quite interesting as you can see that unpredictability modulates the response. But it modulates the response only later in line with the more recurrent process, whereas a low level contrast modulates the immediate feed-forward sweep and it has also stronger effect.
Actually do that for each convolutional layer, so for each level of abstraction, you see that Overall, into layer seems sort of the unexpectedness of each convolutional layer.
Modulates the brand response. But especially the highest convolutional layers. So if we now summarize over time and look at the effects, the level of abstraction.
In terms of their effect size. What we find is that indeed. And the later layers of Alex N have the largest effect.
If you're a vision scientist, you may not be a bit puzzled because this is in the visual early, this is V one.
So we see that the most abstract feature on predictability has the largest effect. We know from V one that V one itself is sensitive to low level features.
So indeed if you do the classical sort of decarlo encoding approach, where you so we're not thinking about unpredictability now, but just the sensitivity to the visual features.
We see, we see exactly the opposite curve. So V one neuron neural responses are best predicted by low level features.
But high level unpredictability. So you see exactly the opposite. Which is kind of interesting.
Oh yeah, and then sort of as a last. Tidbit all these work by the way is still very much in progress.
So it may still change a bit. As we do some additional and analyses. But one thing that I didn't wanna Ignored today is the laminar analysis.
So in this data set, the mice have these neural pixels probes in their visual cortex and they measure not just at one, at one cortical location, but they measure all throughout the cortex.
So what you can do is look up for every unit that you record where they are in cortex and then sort of.
Very roughly, you subdivided them into superficial or deep cortical layers. If you do that, you find that the effect is much stronger in the neurons that are classified as superficial.
Then does in deep. This is only for the future unpredictability. We don't find this for contrast.
So it seems that this unpredictability effect is not just tuned inversely to what you might expect from V one, but it's also strongest.
For the superficial layers compared to the deep, which is exactly in line with many of these. Theories about the predictive brain.
Alright, so if you come back to that question, so generalization. Do we find that indeed the brain always predicts?
So what did these studies so far show, including others, that in language, yes, in music, yes, and vision, yes, we find predictability effects and continuous prediction in all these.
If you then ask the second question. What is actually in this prediction? We see that in language we find this hierarchy of predictions.
But the brain is constantly predicting at multiple levels of abstraction and higher levels seem to be able to inform lower level predictions.
Where's in music we find strikingly low level in the sense of short range predictions. And I find it in vision, we find these higher level predictions that are then fed back to V one.
But I have to say that, yeah, this visual, this vision project is still work in progress. The exact conclusion is may still change.
Alright, so summing up. The rise of generative I, AI, enables stronger and more precise deaths of the predicted brain hypothesis.
And then so far, we've ubiquitous prediction in many domains and across levels of representation.
And to me at least this really showcases the power of AI to change the way we do cognitive neuroscience.
And yeah, the last point is just. It's my own personal view, but I think this is really the future for the study of the predicted brick.
So with that I just wanted to thank you of course for your attention and all my co authors and especially the people who provided all the open source software and data.
With the special mention here, Giovanni and the lower lap more generally. Because we like to give it would have been possible what I've been doing over the past years.
So with that, thank you very much.
Well, we'll talk. Really liked it. I'll anyone have any questions?
Should I stop sharing?
We have I have a question. Oh, you, you can keep it up in case you have slides that could possibly answer questions.
That's true. Yeah.
So we have one for one of the attendees. I'll, we'll let attendees ask questions first and then, you know, I have a question. Giovanni has a question.
So if there's no questions, we'll. We'll ask some questions, but there's an attendee who has a question.
Feel free to unmute if you'd like.
Yeah.
So, I have a question about the language model. So you talked about how, I mean, I know you used it as an example, but in the old man in the sea.
Example, it sort of used the context. I'm assuming I don't know so much about the GPT models, but I'm assuming it uses context that it finds online to sort of make the predictions.
Is that a correct assumption?
Yeah, definitely.
So would then understanding the human brain, like wouldn't that be a big jump because the human brain doesn't have all of the information that like the model would have.
So like context clues would be more in terms of like the meaning of the sentence. But then when you look at that, that include all of the literature, you know, ever written.
Yeah. Yeah.
Wouldn't that be like a not equal comparison?
Yeah, yeah. No, I'm very happy you ask this question. Because it's yeah, I first had a slide on sort of clarifications and common objections but I removed it for the sake of time but one of these things but it was actually really happy to ask because it's a Question it comes up a lot.
So in this approach where you use a generative model to quantify the statistics that you hope that you assume that the brand is tracking.
It doesn't require assuming that the exact statistics computed by GPD are exactly what is dragged by the brain.
And what I often, Do is make a comparison to something like, how we compute word frequency, for instance.
And so, for, frequency, we often have these algorithms that, that we count the word occurrences in a huge corporate.
And they are often much larger than what you read. In your entire lifetime. But so this is not meant to be an exact statistic of what you exactly have in mind.
But what people tend to find is that you exactly have in mind. But what people tend to find is that the better you objectively approximate this statistic, On average, the better it that you track.
The specific. Predictions in the brain. And that's also what you find in language. So, Marise mentioned before that she couldn't really find significant effects.
Differences between different models like an RNN and an NG, but in reading time in the reading time literature people have done this a lot and what people always find is that the better the language model becomes.
So going from an NG, via an RNN, files, simple transformers up to like GPT.
There's this monotonic, almost linear relationship between the quality of the language model and the quantity of the fit.
And that's probably not because the human brain is exactly making that prediction. Probably the human brain is doing something less sophisticated.
It has also read read less and learned less. But the more it matches the objective statistics. The more on average it will be closer to the idiosyncratic statistics on individual participants.
So that was a long winded explanation, but I hope it makes it clear.
Thank you.
Yeah, it makes more sense. Thank you.
Hey, go ahead.
Hi Mike, this is, Ed Laura here. I had a question on, If I just try to understand the you had a kind of an X shape for these and predictions versus low-level contrast that the V one was sensitive to.
No.
Is that right, So I'm maybe you could just talk us through that again because I was trying to understand how what V one might care about in terms of low level features.
Yeah.
No, you know, if I think about predictive coding theory, surely it should be getting predictions.
Yeah.
That are of predicted low level features. So My misunderstanding. I don't maybe you could talk me through this curve again or just
No, no, no, you didn't misunderstand it at all and it's also surprising and it's also what I expect that even though this particular neuron paper already pointed a bit in this direction.
It was also what I expected. So I was I was expecting to find at early, early visual cortex is sensitive to predictability of lower level features whereas later visual cortex would be sensitive to predictability, high level features.
And that would indeed be in line with. And very strict or sort of cartoonist view of predictive coding, but we find is the opposite.
So I think you understood it perfectly. We find this inverted. Tuning functions where in terms of the log level features v one is indeed sensitive to the lowest levels.
Of, here you quantify its file, the encoding forms of Alex. In terms of unpredictability, it's exactly the other way around.
And, yeah. I can. Speculate why that may be. But in terms of if you've understood it correctly, I think you understood it perfectly.
Okay, yeah, thanks.
No, yeah. If there's not another question, I can also speculate a bit why this may be the case.
One thing that I find myself especially interesting is that there are now results in in machine learning. Where people are impressed by sort of the performance of GPT for instance and they think okay if we can learn everything there is about language from predicting words can we also learn everything there is to learn about.
A vision just from predicting pixels. So they train these large models to predict pixels based on their occurrence.
And what they find is actually that models that become really good at predicting pixels are really not so useful for high level visual tasks.
And the only methods that work really well in terms of this self-supervised free training, our methods that explicitly in some way or another discourage the production of low level features and encourage the prediction of eye-level features.
So what you see from machine learning is that the thing that's really useful in self-supervised breed training is prediction of high-level features.
And then what we see in the brain is that the thing that really drives early visual cortex is the predictability of high level visual features.
So that's to me at least is striking parallel. And to me, what that may suggest is that this may, this may reflect a learning signal, which we know from machine learning to work really well.
But that's of course speculation. You could also come up with other explanations, but it's my Beth theory for why we may find this opposite tune.
That's very interesting. Thank you.
Jodie, do you want to ask a question or should I?
Yeah, have a have a quick one.
Go for it.
Yeah.
Music. Hierarchy. Have you tried to look into, you know, your second point in the conclusions was that, yeah, there is the hierarchy in speech.
But in music, you find this slow. Small, you know, short context to be the optimal, but have you tried to combine short long context?
No, no, I said the, you don't know. So yeah. The answer is no.
I also really wouldn't, know how you can do that actually. This is why I like working with language so much because we have this large inventory and we know that sort of words exist, at least if you believe words exist in the mind, we have syllables and phonemes and we can Really?
Dissect stimuli at these multiple levels and then explicitly look at how one level informs the other.
I wouldn't really know, so I know what a note is, but I wouldn't, I don't have the conceptual apparatus to know what a word would be in in music.
So we could, I wasn't smart enough. Let's do it like that, to perform the same types of analysis.
But what do you think? How can you test this in music? What is the rights level of.
Yeah, I mean, I'm just thinking, I mean, from a purely methodological point of view, you could have multiple features fit with different context lines.
That's the boring thing. The other thing is inspired from what I said to me years ago on classical music that the first time you listen to it, maybe you would just focus on the on on patterns that are short-term and then you start getting the you know different levels of of context and so maybe maybe modulating that on the participants.
Got in multiple repetitions of complex patterns and see whether that curve that you have there changes, you know, having the same identical method you have there, but with different conditions.
Or different. Yeah.
Okay.
Right. Yeah, no, that's true. It would require, yeah, and we wouldn't be able to use your download, your data and use such an open end today.
It is about indeed. Yeah, I agree with sort of very clever experimental design. I think you can go there.
And what did you mean with the first boring one? You meant like a decoding analysis. And then sort of comparing the decoding performance on a time result basis and relating that to expectedness or something or
Sorry, I don't this is not as quick as I thought. And, and it's like what you do in language, where you include multiple features.
Multiple surprise vectors that were fits using different context lengths. You know what I mean?
Right.
If they're independent of each other.
You could you could either simply use that model and have different context lengths. Okay. Sorry, and I mean, they could be both.
Right.
And it just, maybe that's not, maybe not with that model, I don't know, maybe the other models.
Yeah. I don't know. I do like this effect because initially, Piers came up with the idea and I was like, are you sure of?
Because I think the better the model is, the better it will fit the brain. Do you think we can have an interesting effect?
And then peace lies like, well, we don't know, blah, blah, blah.
And then we did it and we find it's really strike and dissociation and I never thought we would have.
So, yeah.
Good. Thanks.
Thanks, Lance.
Thanks, Margaret. I think I will. Forgo my question.
Might ask you offline for the sake of time. I'll just say it was a pretty
I see also many interesting suggestions in the jet. So thank you very much for that.
Yeah, I'll say that it was an interesting question, so. Anyway, so I'll let, we should move on.
Ola, I'll let you take over the screen share. Let me take a time.
Thanks again.
Alright, can you hear me? Great.
Yep. All of, Beale is a postdoc, with Ed Lawler.
At the University of Rochester. He is a scientist after my own heart, I'd say.
As a grad student, he helped write a toolbox conducting auditory spatial experiments. Spatial data and since joining the lab one of his endeavors along with one of our grad students Jindo has been converting the M.
Terf toolbox into a Python library. And, leading an unfortunate charge and converting the lab into a Python lab.
His tutorial today will be full of, simulations and interesting minutia. So I hope, you'll enjoy it as much as I will.
I'm looking forward to this talk, from Ola. So I'll turn it over to him.
You're not sharing yet by the way.
Yeah, thanks, Aaron, for the nice introduction and hello, everyone. Let me just. Yeah, yeah, about 2.
Hey, I'm on the screen twice.
And you see my screen now?
Yep, all good.
Alright, and so I'm very happy that we already and got to hear 2 really nice talks about all kinds of different aspects.
Of speech and linguistics because I don't want to talk about any of this in this talk.
Instead, I just want to focus on the Basic signal processing aspects of TRF models. And but before I dive into that, The talk is called Fundamentals of TRF Analysis in Python.
And so I wanna take a moment to argue for why one should use Python. So for once and most importantly, it's open source, meaning everyone can access it.
It is one off or even the most popular programming language. And to show that I've included this graph here from a project called.
Popularity of programming language which quantifies the share of different programming languages over time based on some Google statistics.
And it's the data is on the log scale. So linear growth is actually exponential growth. And you can see Python is just on a ever steady rise.
Especially since 2,015, 2,016 triggered by the machine learning and AI boom.
It's also easy to learn which is in big parts because it is so popular, meaning there's just tons of materials online.
And then finally, it's a general purpose programming language, which means that all things equal, it might be a better idea to learn Python than something domain specific such as MATLAB because there's just a wider set of use cases for the language.
And for those reasons, and Jenna and I wrote an implementation of the MTRF toolbox.
In Python called MTRF Pi. We just recently published it in the Journal of Open Source software.
And so our objectives in making the software were that the API should be as similar as possible to the NTRF toolbox.
So that people who already knew that would have a easy time using Mtrf Pi. However, we also wanted to use an object oriented design because we felt it the code will just be easier to use if all the important methods and variables you would be interacting with are packed into one class rather than distributed across many functions.
Any time we edit the code to check that everything is producing the results it should. And this is just generally a good practice in.
Software design, but it's especially important in our case because of course we want to make sure that running the functions in MTRF toolbox and in the and the corresponding functions and MTRF pi produces the same results if you run them with the same parameters and we ensure that by using this continuous integration.
We wanted to keep it lightweight with minimal dependencies. So right now only uses core Python functions and Numpy, which is the basic.
Python library for doing matrix computation. And this also helps us to make the code compatible with all kinds of existing pipelines.
And we, for example, did not introduce things such as custom data types because we want to avoid that people who want to use the code have to rewrite a bunch of their existing pipelines just to fit it in.
And then finally, the. Software should be easy to use and learn, which is why we provide a quite extensive online documentation and tutorials.
So let me give you a quick overview of the code. And the package consists of 3 modules for once the TRF class.
Which is the main class that the user is interacting with. There's also a module called stats, which features functions for cross validation and permutation.
And then there is a module called matrices which contains functions to, for example, compute covariances and generate lag matrices and so on.
Those functions are usually not accessed by the user directly, but rather called called on by the other 2 modules.
And then the TRF class takes a bunch of arguments and those are kind of the meta parameters that define what kind of model you're using.
So the direction, whether it's a forward or backward model, the method of regularization. Or the metric that you're using to evaluate and compare different models.
And then the important variables that you're usually interested in are contained in the attributes of the class such as the weights of the TRF, the time lags and the regularization coefficient.
That's 1 1 point where we deviate from the MATLAB API. In Python, we can't call it lambda because lambda has already defined function in Python, which definitely bothers me, but it is what it is.
So we just have to call it regularization. And here's a little example of how to use it.
I use this little function called load sample data. Which downloads and returns about 2 min of EG recording along with a 16 band spectrogram of the speech the participant listened to.
And I create an instance of the TRF class. So here using empty parentheses, meaning it's just going to use the default parameters.
Which will create a forward model which with a ridge regularization. And then use TRF the train which is the equivalent of MTRF train.
Fitting the TRF to the stimulus and response. And then here I use the plot method to get just a quick visualization of this nice spectral temporal response function here.
We also use the opportunity to implement some functions that are not part of the original MTRF toolbox.
For example, and here I use this function called permutation distribution. Which, randomly scrambles the stimulus and response trials and then trains TRF models on the on this permuted data.
And returns a correlation coefficient and repeats this over and over again here a thousand times set by this parameter and permute.
And so you're getting a thousand correlation values? And the distribution of these of these values, the permutation distribution.
Reflects the effect size that you're expecting in the absence of any actual cause of relationship between stimulus and response.
And then you can use this distribution. To compare your actually observed correlation to and assess the significance of your result.
So now I want to move to some demonstrations in the tutorial. I'll be using actual brain recordings.
Also actually the old man in the C data. So that's just kind of the canonical data set for doing these kinds of analyses.
Apparently, and but here I wanna just use the simulations. Because the simulations are a really useful way.
Of testing our intuitions and our understanding of these kinds of models. And all the code I'll be using from now on to generate the simulations and the plots.
I have shared with with the CNSP team, so I don't know if you already got it but if you wanted you should be able to get it and play around with all of the parameters yourself.
So I'm using a wavelet here to simulate neural responses and a wavelet is generated by simply multiplying assignasoid with the Gaussian.
In time frequency analysis, we also use wavelets rather complex wavelets, but here I only use a real value.
And the neural response then is simulated by taking a stimulus which is here just some square pulses that are randomly spaced and vary in their length.
And then we convolve that stimulus with the wavelet, meaning we slide the wavelet over the stimulus and multiply it with the stimulus at every point.
And then for each. For each multiplication, we sum all the points and get 1 point of the response.
So this basically means we're filtering the stimulus with the wavelet. And the result is our stimulated brain response, which looks like this.
So whenever there is a pulse, you get like this. Thing here, which kind of looks like the wavelet or like a stretched version of the wavelet.
And then if you have multiple pulses close to each other. The single responses kind of start to blend into each other.
So if the distance between the pulses is less than the duration of the wavelet. And then we can use the TRF on the simulated response to see if we can recover this wavelet.
So we're going basically going to act as if the wavelet is not some approximation, but literally is how the how the brain responds to the stimulus and then we're We're checking whether we can use the TRF to recover this.
A ground truth from the data. And in this scenario I'm showing here where there is no noise, there's just a stimulus in the wavelet.
We can use ordinary least square regression, meaning we can set the regularization value to 0 and just use TR.
To fit the model and then use TRF that predict to get a prediction of the simulated response.
And on the left side, I'm showing the wavelet and the simulated response. And on the right side, I'm showing the TRF reconstruction.
And the TRF prediction and you can see they're identical so we're able to perfectly recover the the wavelet in the absence of noise.
But so now what happens if we start to introduce noise into the picture? So if we start adding normally distributed noise of increasing amplitude, then at some point this TRF is starting to break down.
So here you can see at minus 30 dB SNR. It's really hard to still pick out the shape of the of the TRF from among the noise.
And minus 30 dB isn't that noisy. So here in the bottom you can see then the noisy response and even with the minus 30 dB.
Just by eye, you're still easily able to pick out the stimuli from among the noise.
But the OLS regression is not. So we can circumvent that by starting to use regularization.
So you can call the TRF. Dot train function. And instead of using a single value for regularization, you can just pass a list of values.
Here I use a log space array. And then what will happen internally in the TRF function is that it will train a TRF for each value in this list.
And test them using cross validation and then select the value that gives the best model. The best model according to the metric.
You're using, which per default is Pearson's correlation between the predicted and the actual response.
And then what you're seeing here on the right plot is the optimal value of the regularization coefficient lambda.
Plotted against the signal to noise ratio. And you can see that the lower the SNR is, meaning the more noise there is.
The higher the value of optimal the optimal value of lambda. So the more noise is the more more regularization and with that regularization we're able to recover the TRF.
Even at much higher noise levels like at minus 40 B and here in the bottom you can see the noise response to give you an idea what might 40 dB.
Looks like. So at minus 40. The response here you can see the clean response in yellow is really buried in noise but still we can get back our TRF.
Of course, it looks pretty. Spikey and that is because we have a lot of high frequent noise in that signal.
Usually in EEG, we filter out the high frequent signal, but here since it's just plain white noise there's a lot of high frequent power.
And one way to to help with that is to use what we call Tikonov regularization. So Tikonov regularization penalizes the square difference between 2 neighboring weights in the TRF.
Meaning it kind of forces the neighboring points to take on similar values. And this usually leads to a smoother solution.
So here even at the minus 40 dB, we get a really good approximation of our original wavelet from the TRF.
The other other than that it's the same thing in that the regular amount of regularization just kind of linearly increases on the log scale with, with lower SNR.
But there is also a downside to the TCON of regularization, which is that if you use it in multivariate models it can cause leakage between the features.
Which is something I'm gonna come back to in a few slides. So, so far we've only used the single predictor, meaning we've used the univariate model.
But of course, one really cool thing about these models is that they're able to incorporate multiple predictors and disentangle their relative contributions to the signal.
So what I did here is I simulated the second stimulus which is just a sinusoid. And and added a second differently shaped wavelet.
And then I convolve the the green stimulus with the green wavelet and the yellow stimulus with the yellow wavelet.
And I just sum the result of those 2 convolutions to get this stimulus here, which just looks like another sine wave with some spikes added on top.
And then again, in this noise free scenario, we can use the TRF with both stimulus features and we can perfectly recover both of our wavelets, meaning the TRF is able to disentangle the 2 contributions and the weights.
Actually tell you how big the effect of each feature over time is on the signal.
But so now what if the stimulus features are confounded? So here I created what in causal analysis is known as a fork.
So fork happens when one variable z causes both x and y. And that cause and that makes it so that x and y are correlated.
But this correlation does not reflect any causal relationship. And in fact, the correlation disappears.
Once you account for the common cause Z. So here Z is this sinusoid. And then the another the other stimulus X.
Is generated by just setting it to one where ever z exceeds a certain threshold. And it's 0 otherwise, meaning the value of x completely depends on z.
And then the neural response why here is created by convolving z with a wavelet. Meaning X does not affect the neural response.
The neural response only depends on Z. And so now what happens if we train 2 TRF models to predict the neural response?
One using stimulus feature Z and one using stimulus feature X. Well, what if we use z we'll just get again our perfect reconstruction of the wavelet.
This is just exactly the same scenario I showed in the beginning if we just ignore x. However, if we only consider stimulus x, we'll still get a somewhat weirdly look in TRF, but that's definitely not random.
And this is because X is actually correlated with the response. It does not reflect any causal relationship.
But you're definitely able to predict. The response based on X. So if you're just interested in prediction, it's totally fine to use X.
Now, the interesting thing is that if we then use both z and x, in one common feature space.
So here I create. One stimulus by just stacking those 2 stimulus features Z and X into one matrix and then use that to train the TRF.
Now we're able to disentangle the contributions. And this is what I'm, so now all the ways for X are 0.
And this is what I mentioned in the beginning. That the the correlation that's caused by the fork disappears once you account for the common cause Z.
But so earlier I mentioned that Tikonov regularization. Can result in leakage between the features.
And now I want to show you what that means. So here if we use tikon off regularization and we start using increasing amounts of regularization so we increase the value of the regularization parameter.
From point 0 one up to one. We can see that the more and more we're regularizing with Taken off, the more the TRF for X starts to approximate the TRF for Z, whereas here with lambda equals one, we might be tempted to think that they're both about equally important.
For generating the response, which of course is not true. We know this in this case because we simulated it that way.
And this is the reason why. It's usually not a good idea to use TikTok for multivariate models.
Although this effect is not completely absent if you use bridge regularization. So you need to you need to crank up the regularization much more with ridge to get non-zero values in in X, but it's but it is just technically possible.
So I would argue that it's always a better idea to just not throw in every feature possible and let and let the model decide which features are important and which not but rather a priori based on yourtical considerations.
Decide which features are important. So here, the best possible scenario is if you can already a priori decide that X should not affect the neural response and so you just gonna leave it out as a predictor.
And with that, I wanna. Thank you for your retention. Again, I would.
Encourage you to check out the online documentation of the project, which contains tutorials and all the different aspects of TRF analyses.
You can find it at the website called Read the Docs or just via the projects GitHub.
And also if you're interested in contributing to this pretty new project, please get in touch both Jin who you can see here in this picture and I are very happy to receive any contributions.
And that's it. Thank you again and I'm looking forward to seeing some of you in the tutorial.
Cool. That was a lot of great minutia. Thanks, Ola. So if anyone has any questions, please feel free to ask.
I'll hold my question. Until others have a chance to ask.
But I didn't realize that you had a It looks like.
It is not my cat, it's just some random cat in Greece on a holiday that walked like next to me.
On there. I thought it looked like Cosmo.
Alright, well I'm gonna go ahead and ask my question. This is the first time I've actually ever seen the taken off regularization demonstrating.
Like that leakage. Wish thanks for that. I've always been like aware of it like superficially and always had like kind of thoughts and I wonder if I could get your thoughts on it too.
Hmm.
Is it possible? Do you think to implement a way in which Taken Off doesn't. Leak across channels for other features is it in the way that the X design matrix is constructed or is it just because there's something next door to it, it's gonna.
Like influence it.
I really don't have a good explanation of that. I think my understanding of linear algebra is insufficient to to do that but that's by the way one thing which I think is a great reason to use simulations if you can't kind of decide it from first principles.
Yeah.
A good idea is just to try it out in a scenario where you know the brown truth and see what happens.
So, no, this is actually also just something I played around with to to find out but I don't have like a a mathematically grounded answer for you.
I'm sorry.
Hmm. I just, I was gonna see if like Mick had any input I just realized that Nick has been a participant this whole time and not a panelists.
Sorry about that, Nick. But if you have any like thoughts on that. Like I think Mix is gonna be in.
In your tutorial. So maybe you could say some words about that there. And we can. Figure that out, cause I think that'd be an interesting thing to figure out.
Any other questions?
Well, There's a link. Oh, Giovanni, put in the chat, that I don't know if you knew this, Ola, but we've integrated your MTRF pie into our web browser.
Oh, that's cool. Thanks.
Yeah. I think it's probably running completely on that, right, Joe? Yeah, it's all it's all Python, unfortunately.
Oh, neat.
Yes. It is, yeah.
Anyway, so for the For the sake of time, we'll, we'll cut it there.
I will hang around in this. In this Zoom room and then some people will be in the tutorial Zoom room.
To answer any questions about setting up your environment for the 2 tutorials. So from here we'll go to the tutorials.
Everyone think Ola for us people are doing that in the chat. There'll be 2 links that were sent to you and so you'll just choose one of those links to go to.
And we'll meet there in about 15 min. Okay.
Thank you to all of the speakers and. And to all of the participants for interesting questions. We'll see you in a few.