See my screen.
Stephanie Haro: Yes.
Marlies: okay, great.
Marlies: So in the next couple of 20 min I want to give you a brief introduction. About linguistic speech tracking by the brain. And I will start with a very, very general introduction
Marlies: to even just neural speech tracking. I know there are a lot of experts in the audience, and but just to make sure we're all on the same page.
Marlies: So whenever you have an issue with hearing
Marlies: the the audiologist, which will test how much speech you can understand can rely on an expensive test battery which
Marlies: Holds, for example, pure to an audiometry or sentence recall tests.
Marlies: and this test they have some clear disadvantages. Of course, these pure tone audiometries. They work with tones and tones you cannot understand. And these sentence recall tests.
Marlies: They actually work with really standardized and control sentences, which is nothing like real listening environments.
Marlies: Additionally, if you fail on these tests or cannot perform these tests?
Marlies: Then the outcome does not allow you to pinpoint the origin of the deficit. Is it situated more peripherally, and can the patient be help to the hearing aid? Or is it situated more cortically? Where maybe language treatment is a better option
Marlies: mit ctl, and additionally, there are also, or the another drawback of this test is that they are not inclusive. Not everyone can perform this task. You need to have attention, and you need to be able to give feedback.
Marlies: But sometimes this is not possible. Think about working with very young children or patients with with coma, for example.
Marlies: So that's why our research group and many others, and try to go towards a more objective way of testing speech, understanding, meaning that we look at brain signals to see or to try and see whether or not someone understand speech, and to do so we rely on the phenomenon of neural checking
Marlies: so very quickly. II know you've heard this yesterday, but when you, whenever you listen to speech, and you would meanwhile record brain signals, then you can actually trace certain characteristics of the speech back in the brain signals
Marlies: and previous studies showed that, using specific features of the speech, such as acoustics, it can really give you insight in how much speech you can understand. So if neural tracking to the acoustic feature of the speech is higher, the speech is better understood.
Marlies: but then becomes the issue. Does neural checking to acoustics means
Marlies: speech understanding?
Marlies: Well, that's not necessarily the case, because we also observe like significant acoustic neural checking. Whenever you listen to music or whenever you listen to the ignored speaker in a cocktail paradigm.
Marlies: so you cannot say that if you have acoustic tracking, that there is a successful speech understanding. It might be that acoustic speech tracking is necessary in order to understand speech, but it's not a causality.
Marlies: So before we delve into. The linguistic tracking stuff that I'm really eager and to talk about. I want to first take, take a step back and go fairly, conceptually, conceptually, about the methods that we use
Marlies: and how this method actually works.
Marlies: So basically we have a participant coming to our lab and we need them listen to stories or podcasts.
Marlies: And in a first step, we estimate a brain model.
Marlies: So we accept some speech and representations think about the envelope and acoustic speech representation here shown in the pinkish line.
Marlies: This can also be suplexical features, so as the lexical features denote the onset of a word or a phoneme, and those are indicated by the the pulses. So whenever, for example, here, whenever you have a phoneme, there's a pulse, and disposers are always of the same height.
Marlies: and then you have the linguistic features. So basically, these are also features with a lot of puzzles. But this time the amplitude changes, depending on how much information the following carries, but more details on those linguistic features later.
Marlies: So whenever you have extracted these speech representations, you have, of course, also the recorded brain responses. And then in in the next step, you basically link those speech features with the recorded brain signals 250,
Marlies: and that's to obtain this temporal response function
Marlies: in a second step. You actually have this estimated brain model. You will use this trf, is sample of response function
Marlies: to in order to extract a measure of neural tracking. So first again you extract speech, features, and then you do a convolution with the temporal response function.
Marlies: a temporal sorry. A convolution is basically a multiplication over time. Just as a quick recap. So we have a feature here. We have a Trf. This multiplication over time or convolution works as follows, so
Marlies: whenever there's an increase, you see the wave form of the temporal response function.
Marlies: Now, whenever the signals become more complex. You can predict more complex signals. For example, here you have 2 impulses next to each other, so the 2 Trfs are very close to each other, so the outcome of the multiplication over time, or this convolution will be a summation of the 2 signals.
Marlies: On the other hand, these waveforms can also be out of phase. So there's like a cancellation of this way here.
Marlies: Now, if you also vary the impulses, heights, or amplitudes, then you can also. Yeah, this also has an effect on your predicted signal.
Marlies: So
Marlies: using this multiplication over time, we have actually our our speech feature that we extracted, we have the temporal response function which tells you how the brain responds to the speech feed feature. Using this multiplication over time. We're basically giving or predicting a brain response
Marlies: and evoked by the speech.
Marlies: we, we not only have the predicted brain signal, but also the actual brain signal, and that allows us to calculate a prediction accuracy which is a measure of neural tracking.
Marlies: It's basically a correlation between each channel and of the predicted signal with the actual recorded signal.
Marlies: I hope.
Marlies: this very brief introduction. Was brief enough for the experts, but introductionary re enough for all the beginners or for people who are unfamiliar with this method. The takeaway is, you have 2 important outcomes. You have this simple response version which tells you how the brain response to the speech features. And you have this prediction, accuracy which tells you how well the brain
Marlies: tracks the speech. It's the measure of neural tracking, and this you have for every
Marlies: or both measures you have for every sensors that you use.
Marlies: So, using this approach.
Marlies: I will. I tried to answer a couple of questions. And I will cover these 3 questions during the talk. So the first question is, which
Marlies: speech feature should we actually use whenever you want to investigate linguistic processing by the brain.
Marlies: and then the the second, and also the third, is whether these speech features really reflect speech understanding, because we tested this via 2 different paradigms, and I would like to go into them both.
Marlies: So let's start with this first question.
Marlies: For this we tested, and 29 young normal healing participants. We listened to a really long story presented in silence
Marlies: at the beginning of my Ph. D. The first and studies were published using linguistic speech, features.
Marlies: So, as a starting point, we we made like a scheme of all the speech features that we were aware of. That used linguistic speech features for other features.
Marlies: and we wanted to test
Marlies: whether these linguistics speech features at different levels. So you have at the phoning level. So which characterizes the information at the level of a phoning at the word, or even at the level of the context.
Marlies: Whether these linguistic speech features actually
Marlies: explains more variance in the Eeg signal, in the neural signal on top of acoustical and suplexical features.
Marlies: Now, why is this important?
Marlies: If you look at speech features. They're all derived from exactly the same signal, and this causes that these speech features are highly correlated.
Marlies: So whenever you have a burst of
Marlies: Acoustic energy.
Marlies: You know that the envelope is going to be like a high waveform
Marlies: symbolizes this first. but at the beginning of this burst
Marlies: there there is phoneme on such, and also an impulse of how much information this phoneme
Marlies: Gary's.
Marlies: So if you would only test a linguistic feature. it is still highly correlated with the response to, for example, the envelope, or purely the acoustics of signal.
Marlies: So to make sure that we estimated linguistic tracking
Marlies: in a way that's it's not biased by acoustic responses. We used. A specific framework.
Marlies: II will call it the subtraction approach.
Marlies: So basically, we calculated 2 types of models we complete model. This complete model
Marlies: contains acoustic speech, features, suplexical speech, features, sort of phonemic word on sets as well as these linguistic speech features, and then we subtracted the baseline model which actually included all control variables. So these acoustic features and the suplexable features, and by taking the difference
Marlies: we could investigate the added value of these linguistic features. So
Marlies: how much does a model improve in predicting the neural response? Whenever we add these linguistic features on top of the acoustics and suplexical features
Marlies: this way, and although it's not the perfect of of approach, of course.
Marlies: this way, we try to minimize the bias towards acoustic response. But you see that the responses to linguistic speech features are really, really, really small. So that's also very, very challenging.
Marlies: But using this framework. We applied it on all the existing linguistic features, or or features that were already discussed in literature. And then we we we observed that not all of them survived this approach. So
Marlies: we we observe this phoneme
Marlies: and linguistic features, so phoning surprise, line, port entropy, and these linguistic features at word, level. So word, surpriseal and word frequency. Those were significant. So for the, for the rest of my talk or we, we focus, or the rest of the studies that we that we did. We focus on those 4 linguistic features.
Marlies: But for this talk I will only focus on board surprise all because it's a very intuitive feature. If you want to know more details about the other features, that's content for the workshop.
Marlies: So basically, word, surprising is
Marlies: mit ctl, and how predictable or surprising a word is given the previous words. So, for example, here this sentence, even in the darkest of times.
Marlies: If one only remembers to turn on the
Marlies: it's not surprising you're going to expect light. So the surprisal value of light is rather low.
Marlies: If it would be. To turn up the music. That would be surprising, because it changes the concept of, or the context of of the sentence that you had in mind.
Marlies: So these previous results we obtained by looking at prediction accuracies. So by looking at whether the linguistic
Marlies: features improve the prediction, accuracy, so the measure of neural tracking. But we also had this other important outcome of the model. So the the temporal response function which tells you how to brain response
Marlies: and for word, surpriseal. It looked like this. So
Marlies: you have a clear negativity around 400 ms, which is really prominent for this central channels here.
Marlies: and it's strikingly so. This is very similar to the
Marlies: Even in this case it's using continuous speech and not carefully created controlled sentences.
Marlies: So in this first study we identified which speech features we should use. The second question was, well.
Marlies: These speech features they characterize so linguistic information that? Do they really represent speech understanding?
Marlies: So to answer that we manipulated the the amount of speech understanding by
Marlies: speeding up or manipulating the speech rate. And we did this in 18 young normal hearing participants
Marlies: in total, we had around 6 speech speech rates, and each speech fragment was around
Marlies: 10 min. So, to give you an example, I hope, rates, I am quickly going to check whether
Marlies: my sound is shared as well. and. Kate.
Marlies: hope you can still see my screen, and hope you can hear this example, Mary, out a to name. So
Marlies: Dutch speaking. So this is really slow speech for some of you it might be intelligible, but for our participants it was understandable.
Marlies: So this is was a slow condition. This was a fast condition.
Marlies: It's very difficult to understand. No participants actually reported that it was well intelligible. So what happens with linguistic tracking? What happens with this added value
Marlies: of these linguistic speech features.
Marlies: so the the red and dots here indicates
Marlies: the significant channels or the channels which drive the topography to be significantly different from 0, and in the slow and normal condition. You see in these conditions that it's. You see these rich channels
Marlies: meaning that the linguistic features do have an added value, but as it becomes faster and faster, less red circles. So this added value of linguistic features.
Marlies: it disappears.
Marlies: Another. Then again, you can also look at the temporal response function and how to bring response.
Marlies: And then,
Marlies: you see.
Marlies: and this so
Marlies: mit Ctl, and although it's super noisy because this time we only have 10 min of data. Well, previously we had 45 min of data.
Marlies: but you see, in the yellowish stones we see a significant and negativity
Marlies: which is only present for the slower as slower speech rates which were intelligible by the listeners
Marlies: also this characteristic topography. It fades away, and the faster the speech becomes.
Marlies: So this gives us a first insight like, okay, maybe maybe linguistic speech tracking really gives an indication
Marlies: of speech understanding.
Marlies: But there is a big butch
Marlies: because
Marlies: this paradigm. It's used. It used artificial speech. So we artificially sped up speech. So what actually happens with natural speech?
Marlies: So in this case.
Marlies: we did a new experiment with 19 young normal hearing participants which I listened to different fragments, Dutch, Frisian, and a scrambled word list, and then we assess the effect on linguist
Marlies: to give you again an example. So for Dutch listen listeners, this will be intelligible from the Harvard happen of ejector Ripper
Marlies: and the Frisian story several Maddie did benefit of tall for my tattoos and traea.
Marlies: So the interesting about Dutch and Frisian, so Frisian, is, and spoken in the upper north of the Netherlands, and it follows a very similar grammatical structure as as Dutch
Marlies: it's near me on a one on map, one map mapping. So that's why we chose Frisian
Marlies: but the participants reported that they had around 50% of intelligibility. But this is, of course, subjectively rated and then a last example is the work list, play for May in is Whom is is a Nadim.
Marlies: So
Marlies: all these speech fragments are spoken by the same speaker, and they are also spoken in a very natural way, aiming to preserve the normal prosody.
Marlies: Then I. So we can actually again plot the same results. So can we
Marlies: see added value of these linguistic speech features. So now we had only 7 min, and we only observe, these significant channels for the Dutch story, but not for the Frisian, nor the word is audition.
Marlies: Then in looking at the Trfs, we also see only this prominent negativity around 400 ms for the Dutch story, but not but not for the Frisian or the wordless condition. And again, this characteristic topography, it disappears whenever it becomes not intelligible.
Marlies: Yeah.
Marlies: So this again gives us some insight that maybe linguistic tracking might be a good indicator on whether or not speech was understood.
Marlies: But there are some clear pro and cons. So because it's indicated like, or my results show that it might be an indicator for speech understanding. So it can really pinpoint understanding and not just hearing
Marlies: the disadvantage is that you require lots of data. You you need quite some participants, because currently we can only and look at its own population level
Marlies: and also for participants. There's a minimum of of 10 min. But I would
Marlies: give you like, ideally the more data the better because it's such a small effect which makes it really, really challenging. But we're currently working on an approach to make this more robust. But it's challenging.
Marlies: So in my, my workshop, my goal is to really
Marlies: make you able to craft your own speech features to make these phoning segmentations
Marlies: do this language model so you can create your own word, surprise all stuff, and you can apply this for your experiments so that we can all accelerate science. Together.
Marlies: So yeah, this is, that's it. And I'm happy to take any questions.
CNSP: Cool, cool talk, thanks, Marlise.
CNSP: We have a question in the chat, but
CNSP: the the asker is going to ask for themselves.
danna pinto: Hi! Can you hear me?
CNSP: Yep, yes.
danna pinto: my question was when you were looking at the the different
danna pinto: the difference between the fast and the slow
danna pinto: speech. Did you also look at the differences in just the acoustic model
danna pinto: to see if, if
danna pinto: changing this change, the
danna pinto: the tracking of the envelope, or the other acoustic features that you used.
Marlies: Yes, good point. Yes, we did look at at the acoustics as well.
Marlies: Maybe I can can an update I'm trying to.
Marlies: So I'm going to. I hopefully. You can see my draw. Can you see?
Marlies: Okay?
Marlies: yeah, it's beautiful.
Marlies: And
Marlies: so what? We actually well. and anyway.
Marlies: what? What? My point what what else? I cannot proceed with this drawing, and not the technical enough, I guess, to manage zoom. But anyway, what we saw this that's with speech rates. So this is increasing speech rates. And this is a neural tracking. So in a beginning we see an increase with higher speech rates, and if I'm not mistaking this effect was also seen yesterday. So
Marlies: with with higher speed rates. In the beginning there's higher acoustic tracking, which I think makes sense, and because there are just more acoustic boundaries to to track and to respond to.
Marlies: So your neural checking improves. But then, at the at the end, where it's super super fast, we see again a decrease.
Marlies: and this post puzzle those a bit. So we went digging in literature.
Marlies: And then we could form form some hypothesis. But this is purely speculations that this might be due to motor cortex involvement. So when.
Marlies: whenever you listen to speech, your motor cortex is active, and it sings a bit to how your lips would move.
Marlies: But when the speech is so so fast
Marlies: you don't have this lip image, and you cannot mimic on how your muscles would be able to pronounce this. Hence your acoustic tracking would be
Marlies: slightly lower when it's super super fast, when it's unnatural, unnatural to be
Marlies: pronounced.
Marlies: So I hope that's clear. And it answered your question. Yeah, yeah, that was great. Thank you so much. And it was a great talk. Thank you. Thanks
CNSP: any other questions.
CNSP: and if you want to ask that, you're more than welcome to. I can unmute you, or I can ask it.
CNSP: If you raise your hand you can ask, but if you don't raise your hand I'll just read it.
CNSP: There you go.
EDMUND LALOR: Hi, Marlise! That was a really nice talk. I don't know. I think I've asked you this question before just about kind of linking across the different levels of
EDMUND LALOR: processing where you know some of the time if you're calculating things like phonemes, surpriseal and so on.
EDMUND LALOR: It's it's kind of word specific phonemesal. Whereas obviously the phoneme.
EDMUND LALOR: the surprise of a given phone name will be affected by how likely that particular word was to come up, and all that kind of stuff. So I don't know if you thought about that any further or
EDMUND LALOR: have anything to say on that
Marlies: yeah,
Marlies: so
Marlies: indeed and there are like lots of ways, that linguistic features can be improved.
Marlies: We currently calculate for phoning supervisor for a word, independent of the previous words. So that's indeed a drawback of the of the current approach.
Marlies: But there are papers. A fairly recent one.
Marlies: I broke back
Marlies: where he really and try to. Yeah, see at which level. This phone.
Marlies: phoning, surpriseal stripy models.
Marlies: whether? How? How much? Yeah.
Marlies: Context windows and where it's more top-down or a bottom up effect.
Marlies: I
Marlies: I feel using linguistic features that it's bit like
Marlies: Our brain is not exactly the same
Marlies: as a language model.
Marlies: So there there are. There will be ways to improve this
Marlies: but I think it's very challenging to do it really in the right way, because even if you take further context into account. How will it affect? Like.
Marlies: yeah, I
Marlies: I don't know. I'm puzzled by this myself to to craft a optimal linguistic feature. I think this is a good one.
Marlies: Starting points. But yeah, way, more model
Marlies: using lots of participants
Marlies: and 45 min of data. I cannot find a significant difference between those 3 language language models, although the complexity is really really different.
Marlies: So my takeaways like
Marlies: this is a group, and like linguistic features are good proxy.
Marlies: That's like, I don't think the brain is really a language model.
EDMUND LALOR: I hope that's not your question. Yeah, that's lots of interesting stuff there. In that answer. Thank you
CNSP: have a question if no one else has one.
CNSP: I know Tom is big on big on creation and patents, and I was wondering if if your involvement, or if the lab involvement in in this sort of project sort of ends at like the basic science, or if if you all are working on
CNSP: implementing this
CNSP: for audiologists or clinical practice
CNSP: like to any degree, any level of degree.
Marlies: Well.
Marlies: I'm not involved in any patterns. So and I'm also not really aware of all the
Marlies: all those things. So
CNSP: if you want to use this stuff, you can use this.
CNSP: Okay welfare no more.
CNSP: Someone has a hand raised. Ola has a hand raised.
CNSP: Feel free to ask a question, or.
CNSP: Ola.
okay.
CNSP: did you raise your hand? All the
CNSP: we can't hear you. Can you hear me now?
Ole: Okay.
Ole: So my question was in the, I think, last experiment you showed like the contrast between word lists and the contentious story that the word list doesn't show like an effect of word surprising.
Ole: Like, have you tried mo like actively modulating the supprisal of words in the word list, like in the sense that you know. If I have like a list where I say, like
Ole: house, cathedral, bridge, elephant, or something like that, like the word elephant, would be, would be surprising in that scenario. And maybe that would be like a way we could disentangle sort of narrative surprising from this sort of simplistic word model. That just word by word probability. Is that something you've looked into?
Marlies: It's not something I've looked into. But it's very interesting.
Marlies: because I think the and 400 ish response that you will get using
Marlies: your like word lists
Marlies: which are actually semantically re related, and as certainly unrelated. Those will yield a way bigger response and narratives.
Marlies: I
Marlies: that's my my gut feeling because
Marlies: those like
Marlies: you are really manipulating it in order for the participant to be surprised.
Marlies: while
Marlies: it is not really natural like, no one is going to say to you that he? Just 8% which would send which with elephant. So
Ole: yeah, I mean, II gave an extreme example now, of course. But I mean, you could use language models to like carefully sort of titrate the amount of surprise in the word list. So it kind of matches, that of a of a natural story, and then.
Marlies: compared with you. And then what you think the 2 responses would be similar, or there would still be some difference between the 2. Yeah. Yeah. Okay. I I made a wrong interpretation. No, I know. I think then probably the effects
Marlies: hmm.
Marlies: might be similar. But it will be very, very tricky, because you always need to be primed.
Marlies: And
Marlies: so, if you're in word lists like house cathedral
Marlies: those words are will probably have.
Marlies: like similar words, probabilities, or or maybe, like semantically, they're in a in a very same category, and those artificial settings will be also very difficult to recreate in a naturalistic
Marlies: story condition, because in the naturalistic story condition, we really have these grammatical structures, pronouns and verbs.
Marlies: it's it's interesting to think about. And maybe if you're really clever, you can come up with a cool design. That's
Marlies: II think it's
Marlies: where to wrap your head about it. But I'm not 100% sure what the outcome will be, but it would be cool to check it out.
Ole: Thank you.
CNSP: Yeah. I agree. I think that's pretty cool idea. I think there's a little bit of stuff to like think through that. But I think that's a great idea. Thanks, Marley, for that. Or with that, I think we'll we'll thank Morley virtually at least, and move on
CNSP: for the sake of time. I'm gonna steal the screen back from you, Morley.