You can perfectly see my screen.
Yes.
Okay, great. So, in the next couple of 20 min, I want to give you a brief introduction.
About linguistic speech tracking by the brain. And I will start with a very, very general introduction.
To even just neural speech tracking. I know there are a lot of experts in the audience, but just to make sure we're all on the same page.
So whenever you have an issue with hearing, the ideologist which will test how much speech you can understand can rely on an extensive test battery which holds for example, cure to an audiometry or sentence recall tests.
This says they have some clear disadvantages. Of course this pewter aromatries they work with tones and tones you cannot understand and this sentence recall tests They actually work with really standardized and controlled sentences which is nothing like real listening environments.
Additionally, if you fail on these tests or cannot perform these tests, And then the outcome does not allow you to pinpoint the origin of the deficit.
Is it situated more peripherally and can the patient be helped to the hearing aid or is it situated more cortically and where maybe language treatment is a better option?
Additionally, there are also or another drawback of this test is that they are not inclusive.
Not everyone can perform these task. You need to have attention and you need to be able to give feedback.
But sometimes this is not possible. Think about working with very young children. Or patients which come off for example.
So that's why our research group and many others and tried to go towards a more objective way of testing speech understanding.
Meaning that we look at brain signals to see or to try and see whether or not someone understands speech.
And to do so, we rely on the sentiment of neural checking. So very quickly, I know you've heard this yesterday.
But when you whenever you listen to speech and you would meanwhile record brain signals, then you can actually trace certain characteristics of the speech back in the brain signals.
And previous studies showed that using specific features of the speech such as acoustics, they can really give you insight in how much speech you can understand.
So if neural tracking to the acoustic feature of the speech is higher, the speech is better understood.
But then comes issue does neural checking to acoustics? Means speech understanding. Well, that's not necessarily the case because we also observe.
Like significant acoustic neural checking whenever you listen to music. Or, whenever you listen to the ignored speaker in a cocktail paradigm.
So you cannot say that if you have acoustic checking that there is a successful speech understanding it might be that acoustic speech tracking is necessary in order to understand speech but it's not a cause allergy.
So before we delve into the linguistic tracking stuff that I'm really eager to talk about, I want to first take a step back and go several conceptually conceptually about methods that we use.
And how this method actually works. So basically we have a participant coming to our lab and we make them listen to stories or podcasts.
And in the first step, we, estimate a brain model. So we accept some speech and representations.
Think about the envelope and acoustic speech representation here shown in the pinkish line. This can also be duplexical features.
So it's a classical features, denote the onset of a word or phoning and those are indicated by the the pulses so whenever for example here whenever you have a phoneme there's a pulse and disposals are always of the same height.
And then you have the, linguistic features. So basically these are also features with a lot of posts, but this time the amplitude changes.
Depending on how much information it's, but more details on those literacy features later. So whenever you have extracted the speech representations, you have of course also the, recorded brain responses and then in the next step you basically link those speech features which the recorded brain signals.
And that's to obtain this temporal response function.
In a second step, you actually have this estimated brain model. You will use this, TRF.
Of response function. To in order to extract a measure of neural tricking. So first again, you extract the speech features and then you do a convolution with the temporal response function.
. sorry, a convolution is basically a multiplication over time Just as a quick recap.
So we have a feature here. We have a TRF. Is multiplication over time or convolution works as follows.
So Whenever there's an impulse, you see the waveform of the temporal response potential.
No, whenever the signals become more complex, you can predict more complex signals. For example, here you have 2 impulses next to each other.
So the 2 TRFs are very close to each other. So the outcome of the multiplication over time or this convolution will be a summation of the 2 signals.
On the other hand, these waveforms can also be out of phase, so there's like a cancellation of this wave here.
Now, if you also very impulses heights or amplitudes, then you can also, yeah, this also has an effect on your predicted signal.
So using this multiplication over time, we have actually our speech feature that we extracted. We have the temporal response function which tells you how the brain responds to the speech feature using this multiplication over time.
We're basically giving or predicting a brain response. And evoked by the speech. We, we not only have the predicted brain signal, but also the actual brain signal, and that allows us to calculate a prediction accuracy which is a measure of neural checking.
It's basically a correlation between each channel and of the predicted signal with the actual reported signal.
I hope, this very brief introduction was brief enough for the experts, but introductionary enough for all the beginners or for people.
We're unfamiliar with this. The takeaway is you have 2 important outcomes. You have this simple response section which tells you how to bring response to the speech features and you have this prediction accuracy which tells you how well the brain checks the speech.
It's the measure of neural checking. And this you have for every or both measures you have for every sensors that you use.
So using this approach. I will, I try to answer a couple of questions and I will cover these 3 questions during the talk.
So the first question is which speech feature should we actually use whenever you want to investigate linguistic processing by the brain.
And then the second and also the church is whether the speech features really reflects speech understanding because we tested this via 2 different paradigms.
And I would like to go into them. So let's start with this first question. For this we, tested, 29.
You know, normal healing participants, we listen to a really long story presented in silence.
At the beginning, of my PhD, the first, studies were published using linguistic speech features.
So as a starting point we We made like a scheme of all the speech features that we were aware of.
That used linguistic speech features or other features. And we wanted to test. Whether these intrusive speech features at different levels so you have at voting level so which characterizes the information at the level of phoning at the word or even at the level of the context.
Whether this linguistic speech features actually. Explains more variance in the EGG signal in the neural signal.
On top of, acoustical and seplexical features. Now why is this important?
If you look at speech features, they are all derived from exactly the same signal. And this causes that these speech features are highly correlated.
So whenever you have, a burst of, acoustic energy. You know that's the envelope is going to be like a high waveform which Similarizes this burst, but at the beginning of this burst, There there is phoneme on such.
And also an impulse of how much information this morning. There is. So if you would only test a linguistic feature.
It is still highly correlated with the response to, for example, the envelope or purely the acoustics of signal.
So to make sure that. We estimated linguistic tracking. In a way that it's not biased by acoustic responses.
We used, a specific framework. I, will call it the subtraction approach.
So basically we, calculated 2 types of models. We complete model, this complete model. Contains acoustic speech features.
Seplexical and speech features, sort of volume and word on sets, as well as these linguistic speech features.
And then we subtracted the baseline model which Actually included all control and variables. So these acoustic features and these are flexible features and by taking the difference you can and we could investigate the added value of these linguistic features.
So how much does a model improve? In predicting the neural response whenever we add these linguistic features on top of the acoustics and seplexical features.
This way, although it's not the perfect approach of course. This way we try to minimize the bias towards acoustic response.
But you see that the responses to linguistic speech features are really, really, really small. So that's also very, very challenging.
But using this framework, we applied it on all the existing linguistic features or features that were already discussed in literature.
And then we, we observed that not all of them survived this approach. So, We, we observe this, Linguistic features.
So for in surprise, and port entropy and these linguistic features at work levels, so word surprises and Which frequency those were significant.
So for the for the rest of my talk or we would be focused on the rest of the studies that we that we did we focus on those 4 linguistic features.
But for this stock, I will only, focus on what's surprising because it's a very, intuitive feature.
If you want to know more details about the other features, that's, content for the workshop.
So basically, word surprising is how predictable or surprising a word is given the previous words.
So for example here this sentence even in the darkest of times If one only remembers to turn on the It's not surprising you're going to expect light sort of surprise or value of night is rather low.
If it would be, to turn up the music, that would be surprising because it's changes the concept of or the context of the sentence that you had in mind.
So this previous results we obtained by looking at prediction curses. So by looking at whether the linguistic features improve the prediction accuracy, so the measure of neural checking.
If we also had this other important outcome of the model. So the temporal response function which tells you how to bring a response.
And for, word surprises, it looked like this. So, You have a clear negativity around, 400 ms, which is really prominent for these central, channels here.
And it's, strikingly, so this is very similar to the N. 400 paradigms.
Even in this case, it's using continuous speech and not. Carefully created control sentences.
So in his first city we identified which speech features we should use. The second question was, well, These speech features, they characterize so many linguistic information.
The today really represent speech understanding. So to answer that, we, manipulated the, the amount of speech understanding by and speeding up or manipulating the speech rate.
And we did this in, 18 young, normal hearing participants. And in total we had, around 6 speech, speech rates and each speech fragment was, around 10 min.
So to give you an example, I hope that I am quickly going to check whether My. So it is shared as well.
Okay. Hope you can still see my screen and hope you can hear. This example.
Mary, Igil, you're right, may, out a name.
So I'm that speaking, so this is really slow speech. For some of you it might be intelligible but for our participants it was understandable.
So this was a slow condition. This was a fast condition. It's very difficult to understand.
No.
No participants actually reported that it was well intelligible. So what happens with linguistic tracking? What happens with this added value?
Of these linguistic speech features. So the red, dot here indicates the significant channels. Or the channels which drive the topography to be significantly different from 0.
And in the slow and normal condition, You see in these conditions that it's you see these rich channels.
Meaning that's the linguistic features do have an added value, but as it becomes faster and faster, less red circles, so just added value of linguistic features.
It disappears. I know, then again, you can also look at the support of response function and how to bring response.
And then. You see, this soon. Although it's super noisy because this time we only have, 10 min of data, well, previously we had 45 min of data.
But you see in the yellowish stones we see a significant and negativity. Which is only present for the slower as slower speech rates, which were intelligible by the listeners.
Also, this characteristic topography, it fades away and the faster the speech becomes. So this gives us the first insight like, okay, maybe linguistic speech tracking really gives an indication.
Of speech understanding. But there's a big butch. Because it's.
Paradigm it used it's used artificial speech so we artificially sped up the speech so what actually happens with natural speech.
So in this case. We did a new experiments which 19 young normal hearing participants which listened to different fragments or Dutch fusion and a scrambled word list and then we assess the effect on linguistic.
To give you again an example. So for that, listen, this will be, intelligible.
From Bahavard, heaven object the ripper.
Then the Fijian story.
S of all Maddie did Benavid up tall for my atatu's tray.
So they're interesting about Dutch and Fusion. So it's, in the upper north of the Netherlands and it follows a very similar grammatical structure in the upper north of the Netherlands and it follows a very similar grammatical structure as Dutch.
It's nearly a one on map, one man, nothing. So that's why we chose, but the participants reported that they had around 50% of intelligibility.
But this is of course subjectively rated. And then a last example is the work list. Okay.
Blade for May in Ishoon is is a Nadin.
So all these speech fragments are spoken by the same speaker and they are also spoken in a very natural way, aiming to preserve the normal process.
Okay, so we can actually again, plot the same results. So can we. See add value of these linguistic speech features.
So now we had only 7 min and we only observe the significant channels for the Dutch story but not for the Fesian nor the word is condition.
Then in, looking at the TRFs, we also see only this prominent negativity around 400 ms for a Dutch story, they're not, but not for the Freedom or the wordless condition.
And again, this characteristic topography, it disappears whenever it becomes not intelligible. Yeah.
So this again, it gives us some insight that maybe linguistic tracking might be a good indicator on whether or not speech was understood.
But there are so clear, pro and cons. So because it's, like, or my results show that it might be an indicator for speech understanding.
So it can really pinpoint understanding and not just hearing. The disadvantage is that you require lots of data.
You You need. Quite some participants because currently we can only, look at it on a population level.
And also for, participants. There's a minimum of 10 min, but I would. Give you, like ideally the more Data, the better.
Because it's such a small effect, which makes it really, really challenging. But we're currently working on an approach to make this more robust.
But it's challenging. So in, my, my workshop. My goal is to really.
Make you able to craft your own speech features to make these phoning segmentations. Do this language model so you can create your own word surprising stuff and you can apply this for your experiments so that we can all accelerate signs, together.
So yeah, that's it. That's it. I'm happy to take any questions.
Cool, cool talk. Thanks, Marise. We have a question in the chat, but.
The the asker is going to ask themselves.
Hi, can you hear me?
Yep.
Yes.
My question was when you were looking at the, the different the difference between the fast and the slow.
Speech. Did you also look at the differences in just the acoustic model? To see if if changing this change the the tracking of the envelope or the other acoustic features that you used.
Yes, good points. Yes, we did look at, at the acoustics as well.
Maybe I can. I'm trying to So, I'm going to, hopefully you can see my draw, case it.
Yep. It's a great drawing.
Okay. Yes, beautiful.
So, anyway. What my point? I cannot proceed with this drawing. And that's technical enough, I guess, to manage zoom.
But anyway, what we saw us that's with speech rate, so this is increasing speech rates and this is a neural checking so in the beginning we see an increase with higher speed rates and if I'm not mistaken this effect was also seen yesterday so with, with higher speech rates in the beginning, there's higher acoustic tracking, which I think makes sense.
In the beginning, there's higher acoustic tracking, which I think makes sense. And because there are just more acoustic tracking, which I think makes sense and because there are just more acoustic boundaries to to track and to respond to.
So your neural checking improves, but then at the end where it's super, super fast we see again a decrease.
And this puzzle does a bit, so we went digging in literature. And then we could form, form some, hypothesis, but this is purely, speculations.
That this might be due to motor cortex involving that. And so when Whenever you listen to speech, your motor cortex is active.
And it sings a bit through how your lips would move. But when the speech is so, so fast.
You don't have this nip image and you cannot mimic on how your muscles would be able to pronounce this hence your acoustic tracking would be slightly lower when it's super super fast when it's unnatural, a natural to be.
Pronounced. So I hope that's clear and it answered your question.
Thanks.
Yeah, yeah, that was great. Thank you so much. And it was a great talk. Thank you.
Any other questions?
And if you wanna ask that you're more than welcome to, I can unmute you or I can ask it.
If you raise your hand, you can ask, but if you don't raise your hand, I'll just read it.
There you go.
Hi, Marlise. That was a really nice talk. I don't know, I think I've asked you this question before just to beg kind of linking across the different levels of.
Processing where you know some of the time if you're calculating things like phoneme, surprising and so on.
It's kind of word specific, phoneme, surprising, whereas obviously the phoning. This surprise of a given phoneme will be affected by how likely that particular word was to come up and all that kind of stuff.
So I don't know if you'd tolerate that any further or. Have them to say on that.
Yeah. So. Indeed, and there are like lots of ways that linguistic features can be improved.
We currently, calculate for phoning, for a words independent of the previous words so that's indeed a drawback of the of the current approach.
But there are papers, a fairly recent one. I brought back. Where he really, tried to.
Thank you.
Yes, see and which level, this morning, phoneing, whether how much, yeah.
Contacts Windows and, whether it's more top-down or whatnot effects.
I
I feel, using linguistic features that it's a bit like,
Our brain is not exactly the same. As a language model?
So there are. There will be ways to improve this. But I think it's very challenging to do it really in the right way because even if you take further context into accounts, how will it affect?
Thank you. Yeah, I. I don't know, I'm puzzled by this myself to turn to craft optimal.
Linguistic feature. I think this is a good. Starting points, but yeah, because for example, for, comparing different language models and rams which are the most simple ones which I usually use.
RNNs, LSDMs, which are able to have like way more context than this simple engram model.
Using, lots of, participants. And, 45 min of data, I cannot find a significant difference between those 3 language, the language models, although the complexity is really, really different.
So my takeaways like. This is the group, like legacy features are good proxy. Good.
Like, I don't think the brain is really a language. I hope that answer your question.
Yeah, that's lots of interesting stuff there in that answer. Thank you.
I have a question if no one else has one. I know Tom is, Big on, on creation and patents.
And I was wondering if, If your involvement or if the lab involvement in this sort of project sort of ends at like the basic science or if you all are working on implementing this.
Or audiologist or clinical practice. Like to any degree, any level of degree.
Well, I'm not in full in any patterns, so and I'm also not really aware of all the all those things.
So. If you want to use this stuff, you can use this.
Okay, fair enough.
Okay, Someone has a hand raise. Ola has a hand raised.
Feel free to ask a question or.
Ola
Can you hear me now?
Did you raise your hand, Ola? We can't hear you. Yo.
Okay. So my question was in the I think last experiment you showed like the contrast between word lists and the continuous story that the wordless doesn't show like an effect of words, like have you tried?
Like actively modulating the surprises of words in the word list. Like in the sense that, you know, if I have like a list where I say like, House, Cathedral, Bridge, Elephant or something like that, like the word elephant would be, would be surprising in that scenario and maybe that would be like a way we could disentangle sort of narrative, surprising from this sort of simplistic word
model, just word by word probability. Is that something you've looked into?
Right.
It's not something I've looked into, but it's very interesting.
Because I think the and 400 each response that you will get using your like, which are actually semantically related and are certainly unrelated.
Those will yield a way bigger response than narratives. That's my good feeling. Because Those like you are really manipulating it in order for the participant to be surprised.
While it is not really. Natural, like no one is going to say to you that he just 8% which we sandwich with elephant.
So. Yeah.
Hmm.
Yeah, I mean, I gave an extreme example now, of course, but, I mean, you could use language models to like carefully sort of titrate the amount of surprise in the word list so it kind of matches that of a of a natural story and then compared with you and then would you think the 2 responses would be similar or there would still be some difference between the 2?
Yeah, yeah. Yeah, okay. I, I made a wrong interpretation. No, I know.
I think then probably the effects. Hmm. Might be similar, but it will be very, very cheeky because you always need to be And so if you're in word list, like how's Cathedral?
Those words are, will probably have. Like similar words, probabilities or maybe like semantically they're in a very same category and those artificial settings will be also very difficult to recreate in a naturalistic story condition because in the naturalistic story condition we really have this grammatical structure with nouns and pronouns and verbs.
It's interesting to think about and maybe if you're really clever you can come up with a cool design that's I think it's where to wrap your head about it, but 100% sure what the outcome will be but it would be cool to check it out.
Okay, thank you.
Yeah, I agree. I think that's pretty full idea. I think there's a little bit of stuff to like.
Stink through that, but I think that's a great idea. Thanks, Marley. For that or with that, I think I will, will think morally.
Virtually at least. And move on. For the sake of time, I'm gonna steal the screen back from you, Morley.