CNSP: Okay? So what I was planning on doing, I've said this a little bit is
CNSP: I don't plan on taking the entire 2 h for this. I don't really wanna talk for 2 h.
CNSP: I would love if some of you all would talk for some of that time.
CNSP: Especially if it can stimulate some like interactive discussion. What I'm gonna do is sort of given a little bit of an overview of what the Trf. Is, what it does, how it like.
CNSP: how we derive it, and then show some of this script and explain what all the different parts mean
CNSP: hopefully. This
CNSP: won't take too long.
CNSP: but I will start with
CNSP: to my slide.
CNSP: So I
CNSP: I was fortunate enough yesterday to find a talk that I'd given in the past. That sort of details, all of this. at least the background of stuff that I wanna cover today.
CNSP: so a lot of this like we might get into some of the slides. I don't actually apply here. And so just please just bear with me as I skip through things that are important things that are not important.
CNSP: So in this, like brief, I think it lasts maybe 15 min. When I gave it the first time we'll talk about
CNSP: sort of the basis of the Trf. Which is built upon linear regression. I'll I'll go from linear regression. The ordinary least squares approach to it, which is what you know is just standard linear regression.
CNSP: I'll give some
CNSP: some context about the Trf. And what we're fitting and why we need to not use ordinarily squares regression. And sometimes I'll tell about how we fit the temporal response function and what that means, and talk about how we
CNSP: validate that model, and they give a few examples of like what we can do with the Trf. A few findings
CNSP: that, some of the group has
CNSP: found some of these things like envelope tracking. I'll cover in the the code
CNSP: and some of the model comparison stuff. But some of the things I won't
CNSP: like, for example, the cocktail party decoding, and some multisensory stuff in here that I won't
CNSP: compare, just to give you a sense of like one what you can do with the
CNSP: with the toolbox, and show you how we do that, but also to to
CNSP: give you a little something that you can do on your own and figure out yourself. So I'm not giving you all the code. There's a lot that the toolbox can do.
CNSP: So essentially the the Mtrf toolbox is
CNSP: just ordinary linear regression with a Ridge regularization. And so here, let me actually get a
CNSP: pointer
CNSP: so that you can see. So here's like a pretty standard view of linear regression. You've got your input X, it gets multiplied by some coefficient or some weighting parameter beta. Here we add some error term, and then we get the output y for the trf, the X in the forward direction at least, the X represents your stimulus feature
CNSP: and the output. Y represents your Eeg signal. We sort of ignore this error term.
CNSP: and then what we do is we fit this beta coefficient?
CNSP: We can estimate that coefficient using the ordinary least squares estimator just by taking the design matrix of X
CNSP: transposing and multiplying by itself. This gives us an auto covariance matrix of our stimulus. And then what we do is we take that, and we divide that
CNSP: from the XTY. So the transpose of the stimulus against its output response. Y, so this is really doing all the business here, and then we divide out the auto covariance of X
CNSP: to get our coefficient. Once we have this coefficient or set of coefficients, as it were. We can then
CNSP: apply those coefficients to some other, or to whatever stimulus parameter, and get some estimate of what the Eeg activity on that to that stimulus would be.
CNSP: This is sort of the basis of the Trf fit and predict.
CNSP: So, as I said, the Y
CNSP: represents the Eg. Signal. The X represents
CNSP: the stimulus, and that stimulus could be really anything. Most of the time. We'll be talking about the stimulus envelope, but it can be a spectrogram.
CNSP: It can be labeled Onsets of phonemes. It can be, as shown here, the label Duration of phonemes.
CNSP: They can be a semantic predictor. That's a stick. And at the beginning of each word with the stick height, representing like how surprising that where it is.
CNSP: and then we're gonna from that we're gonna derive some beta coefficient. But first a little bit about X
CNSP: and and about what it looks like. So the X is a stimulus representation
CNSP: that we're going to
CNSP: input into the Trf, but because the stuff happens in the brain after it happens in the environment.
CNSP: We need to see what happens for lagged versions of X. So
CNSP: here is the original envelope.
CNSP: And so what we can do in this X design matrix is to shift the envelope upward sample by sample so that we can see the response before the stimulus. So here, this would be your standard linear regression.
CNSP: Here we're pairing the second sample of X to the first sample of Y, so this gives us the response before stimulus. So like the second stimulus is happening
CNSP: before the brain response, which is not physiologically possible, generally speaking, with respect to predictions, maybe. But this is gonna this side going to give us a response before the stimulus.
CNSP: Conversely, when we shift the
CNSP: envelope, or whatever stimulus feature downward.
CNSP: we're matching up
CNSP: the first element of the stimulus with later elements of the response. So more physiological, what you would expect according to a feed-forward model.
CNSP: So to the right of this one, we're going to get the responses after the stimuli. And so each one of these columns of X is going to represent the stimulus feature at different lags.
CNSP: We can leave we can pad the edges with zeros.
CNSP: or we can just cut them out of it. The Trf. Can do both of those operations, but but introducing all of these lags gives us a little problem.
CNSP: So each one of these columns is treated as a separate coefficient, essentially in the Trf
CNSP: and when we do the xt we do get some of the auto covariance between these. But we do need to like regularize this because of the multi-colini. And so oh, I forgot to mention. We do have an intercept term. We usually like Z-score all of our stuff. So that is centered at 0. But this will help.
CNSP: you know, any centering issues.
CNSP: So because we introduced a lot of these lags that are essentially perfectly correlated with each other.
CNSP: Aside from the lags, we need to account for that. And the way we do that is, by transitioning to ridge regression and the Ridge regression solution adds 2 terms here
CNSP: to be added to the that covariance matrix
CNSP: lambda, which is our regularization parameter. And I, which is an identity matrix. This is why we call it a ridge, because the I is the one across the ridge of the diagonal
CNSP: the lambda, or regularization parameter will.
CNSP: We'll tell how much we want to emphasize that main diagonal which includes, you know.
CNSP: time, timelag, 0 with time lag 0.
CNSP: So we're going to emphasize those that main diagonal. And that's going to do what's called regularization. It's going to
CNSP: control for overfitting. It's like what tends to happen is that the the slopes will vary wildly because there's a lot of noise in the sit in the signals. And so this will just sort of control that you'll see more about this tomorrow, and some in some of the tutorials.
CNSP: So once we fit these things, we get
CNSP: the Beta coefficients across time lags, which is actually the temporal response function.
CNSP: I remember when I was a grad student reading some of Ed Lawlor's papers
CNSP: about temporal response functions. I had no idea what they were. They were magic, and in fact, they were magic to me for the first, you know, couple of months that I was in the lab. And then and then I realized one day. I don't know why I realized this and why it took me so long to realize. But these are just linear regression coefficients. These are just slopes.
CNSP: So what this point right here means is that this long after the stimulus the Eeg tends to be high when the stimulus goes up so like when a word onsets you get a big influx of the envelope.
CNSP: and then probably about a hundred milliseconds later. The each tends to be high. That's all that means it's just a slope.
CNSP: But of course we can't just fit in
CNSP: and predict on the same data, because that'd be hell overfitting. So what people end up doing for these sorts of experiments. And the data that we're gonna use today is like this is, you'll have someone in an Eeg cap that's just pay. It's paying attention to some stimulus. In this case. It's gonna be some audio book, and they're just listen to audio book chapters. They're asked to answer some comprehension questions, but
CNSP: they're just listening to a book.
CNSP: For all they know. The book is divided into segments that are like manageable times 3 min
CNSP: or so per trial. But we get
CNSP: some speech segment that they listen to for each of these trials. We also get their Eeg for those trials.
CNSP: and we can use some of those trials to fit a model.
CNSP: so we can take some of the stimulus features, and we can take some of the Eg channels. And we can input those into what we call N Trf. Train, and we can fit a forward or a backward decoder model.
CNSP: And we can take
CNSP: somehow and backwards. Then we can take some other trials
CNSP: after we have this fit model.
CNSP: and we can use them to test that model. So we'll take a stimulus feature and pass it through the model and get some Eeg prediction.
CNSP: or if we're going in the backwards direction, we'll take the Eeg, pass it through our backwards, model our decoder model, and get some stimulus reconstruction. Of course we'll take the the other signal
CNSP: and correlate to that
CNSP: right. So this pathway here
CNSP: represents Mtrf train.
CNSP: This pathway here represents Mtrf. Predict we'll go over these functions later.
CNSP: But this process altogether what we typically do is, take a bunch of trials, fit a model and train on the left out trial, and we'll rotate this left out trial.
CNSP: And this process is Mtrf. Crossval. which I'll talk about later, too. Once we have a fit and optimized model. We can look at that model and look at its weights. And so here
CNSP: are the weights. And if we look across time at a channel at the center of the scalp, we see what looks like a nice
CNSP: yeah erp, except it's not exactly an erp
CNSP: but it's different fit to a continuous stimulus can also look at the backwards model decoder weights. This is apparently not a very well regularized decoder weight.
CNSP: but it's there to illustrate a point that we don't typically
CNSP: interpret these backwards weights. You can interpret these forward model weights
CNSP: as actual Eeg signal. This is what happens in response to a stimulus
CNSP: these are a little bit different, because what happens is that all of the Ag. Channels can go together to help predict stimulus. So you get some sort of antagonistic
CNSP: interaction between the channels, and so, like some channels. I'll show you a little bit better that later. But some channels can actually
CNSP: get fitted with weights that are not physiological, but serve to sort of
CNSP: increase the sensitivity of that signal. We can transform those weights into forward looking weights.
CNSP: and that's cool.
CNSP: skip that so
CNSP: we can use the Mtrf toolbox to track ongoing speech. So here is an example of something that I won't talk about during
CNSP: the tutorial. But that's something that is part of the toolbox, and you can do yourself. The data are out there, and the tutorial code is free.
CNSP: In this experiment we were interested in understanding what are the
CNSP: consequences of coherent audio visual stimulation. So we had people watching visual speech, listening to audio only speech and then watching audio visual speech.
CNSP: and we can fit decoder models on the visual and the audio speech
CNSP: and on the audio visual speech. And if, again, these are tested and trained on their own data set. Then the Trf. Has this other function called Ntrf multi cross Val.
CNSP: that can fit a decoder based on the visual and the auditory only, and test on the audio visual. So this is getting at a sort of multisensory phenomenon that we use where we test the audio visual
CNSP: responses against audio plus visual unisensory responses. And so what we find with these is that when we're reconstructing the stimulus.
CNSP: the visual model, and the audio only model, they do fairly well, but when you combine them they get better at reconstructing the stimulus.
CNSP: and when you combine that, and when you
CNSP: fit them on audio visual stimuli, it gets even better still. So V is fit and evaluated to VA. Is fit and evaluated to a a plus v are fit
CNSP: on a and V, and then test it on. AV, so what we're doing here is we're trying to see if there's any unisensory things going on in the audio visual condition. And then AV is fit and tested on audio visual. So we've got an A B decoder that can't predict audio visual
CNSP: data as well as an audio visual decoder, indicating that there's some sort of nonlinear process going on in there that's only captured by the audio visual decoder.
CNSP: And the improvement of this performance is, we take as evidence of audio visual integration.
CNSP: We will talk about this to some degree later, but we can also look at different features using the Mtrf. Full box. So far we've talked about only the envelope model, but speech can be divided into different features like it's spectrogram, which is the envelope that's just resolved over frequencies.
CNSP: We can also label the occurrence of phonemes
CNSP: we can combine those phonemes into phonetic features so like
CNSP: certain features or specific certain phonemes. And that's what this does. And then we can also have a joint model that we use to fit and predict the stimulus.
CNSP: So these are the examples of those trfs. Here's the envelope here that you've seen so far.
CNSP: Here's a spectrogram. Trf, and so what you see here is the different colors correspond to the amplitude, and then the different rows correspond to the different frequency bands same thing for phonemes and phonetic features with the colors and the rows. But these can be used to predict left out data with the idea that what we're gonna try to do is see if
CNSP: if the spectrogram and phonetic feature model can predict different aspects of data.
CNSP: So one way that we can do that is, we can fit a spectrum spectrogram model.
CNSP: We can fit a feature model. And we can look at their like representations how well they can represent data. And the idea here is that they should have some shared contribution because they're correlated.
CNSP: and they should have some unique contribution kind of just made it up here. And if things go well, this is what the Fs model should predict.
CNSP: and that's exactly what we find, or at least what Joe Bonnet found. That. And he concluded that phonetic features were represented uniquely in brain activity above and beyond the spectrogram. So when you listen to narrative speech, you're getting the both the acoustics and you're getting both the linguistic content.
CNSP: I'm gonna skip that it's more of the same. It's just my stuff that I wanted to plug. One last thing that the
CNSP: Mtrf. Toolbox has been used for quite a bit is attention decoding. This is one of our most cited papers.
CNSP: I think it's about 500 citations at this point, and what people were doing here is, they were listening to 2 stories. One was in the left ear, one was in the right ear. One of the stories was Journey to the center of the earth, and the other service to 20,000 leagues under the sea, and on each trial they were asked to pay attention to one or the other.
CNSP: and what we can do is we can take the attended envelope and the unattended envelope, and we can fit decoders to both of those. So they were getting both of these stories in their heads. We can see how their brain responds to each story
CNSP: at the same time, and the brain does respond to both stories, we are able to reconstruct the attended and the unattended decoder.
CNSP: But what we do then, is we correlate
CNSP: these reconstructions with the actual
CNSP: stimulus, and we do the reconstruction of the attended decoder for the attended and unattended story, and the unattended decoder the same way, and we can look at how well
CNSP: the R attended.
CNSP: was, represent the attended stimulus represented as compared to the unattended stimulus, and on a single trial level. The UN. The attended stories almost always represented more strongly in brain activity.
CNSP: And so this has become the basis of a lot of different like attentional decoding paradigms. And it's you know, we have, like our own conferences like Cog here. That's just almost entirely attention decoding.
CNSP: Let's skip some of that. There's some other analyses that I won't talk about
CNSP: You can find the
CNSP: Mtr. Toolbox here if you haven't already gotten gone to it. I forgot to put it in here, but we do have a new python version of the Mtf. Toolbox, called Mtr. Although we'll be giving a tutorial about that tomorrow.
CNSP: so I'll stop there. That's all for the background. I'll stop in case people have questions
CNSP: to see some faces.
CNSP: any questions
CNSP: you don't need to raise your hand for this. You can just unmute yourself and interrupt me at any point that you want. By the way.
CNSP: seeing as there are no questions, I'll just. I'm up.
CNSP: So now I'll turn to the
CNSP: the code. So this code is can be
CNSP: execute it all at once. But I'm going to go through section by section
CNSP: and just control. Enter each section. So if you're following along with me and I'm going too fast, please let me know. I'll slow down. I'll try to be as slow as possible. But it's gonna take all of our inputs to make this thing last 2 h.
CNSP: So the first thing I'm doing is just doing some dependency issues. I'm making sure everything's closed. Giovahi must have written this one. I don't use clear all anymore. You can
CNSP: run this or not run this. I run this because I have my own startup file. I have
CNSP: a bunch of dependencies already in Matlab so II use this to get rid of those. You will need to run this to add all the dependencies. Oh, look at that! I'm not in the right folder, so let me explain the folder. So you should have
CNSP: your data. C and D in Lawlor natural speech here.
CNSP: You should have this
CNSP: code in the example scripts file.
CNSP: and that's why it didn't run, because I wasn't in there.
CNSP: Let me do that again.
CNSP: But you need to run this. Add path dependencies.
CNSP: because we do have these. As Giovanni mentioned. We do have these dependencies that we
CNSP: used in the tutorials and all Cnsp scripts.
CNSP: Then I will execute this section that's just going to go through and
CNSP: set some
CNSP: parameters. So if you want to change, if you can download multiple data sets, you can change this line here
CNSP: to whatever data set you want
CNSP: here. You can also change the reference type or the bandpass filter. Right? Actually, these don't
CNSP: run right now. I'm not doing any preprocessing. I'm using already preprocessed data that we're in that data set.
CNSP: And the
CNSP: preprocessing script I'm using looks something like this, where
CNSP: actually, I just use the same thing. It's just the average reference filtering from one to 8 hertz down, sampling to 64
CNSP: hertz sampling rate. And then in there we're doing a a removal of bad channels by looking at neighboring channels. All of those functions are available
CNSP: in the Cnsp
CNSP: libraries
Stephanie Haro: asking for a friend. How do we find the seen, the
Stephanie Haro: data C and D folder
Stephanie Haro: like what, URL, should we have downloaded the data from?
CNSP: That will be on the cnsborshop.net. Slash resources there are.
CNSP: Let me go to that website so I can see what's on there.
Stephanie Haro: Well.
CNSP: do you have a browser here? Now?
Stephanie Haro: Yeah.
CNSP: So the resources tab here.
CNSP: The natural speech data set is here. It will download a zip file. It's it's really big. It probably won't be done downloading by the end of the tutorial if you start it now.
Stephanie Haro: Okay, great
CNSP: but inside of that will be the data Cnd file
CNSP: with the C and D data.
CNSP: So I've set a couple of parameters, and I'll go and talk about some of them
CNSP: little bit here.
Chelsea Blankenship: I have a quick question, too. So the the example scripts that I have all have, like example 1, 2, 3, 4. It's named differently than what you have pulled up. Or perhaps I'm looking in the wrong spot.
CNSP: Did you get the email
CNSP: with the tutorial instructions?
Chelsea Blankenship: Yes, and I downloaded those. And I'm in like the Cns resources
Chelsea Blankenship: data sets example scripts.
CNSP: Yeah. So take the.
CNSP: So you should have this script and this mat file or a mat, Matt file from that email. And just put it in this folder example. Scripts. Okay, thank you.
CNSP: Sure.
CNSP: So the first 2 parameters I'll talk about Teamin and Tmax. These are going to correspond essentially to what you see
CNSP: in a in an erp. So if you're looking at an erp, you've got the time along the X axis, and this will define the limits of those axis. So what we're gonna do is we're gonna fit a Trf.
CNSP: starting 200 ms before the stimulus came on to see what was going on in the Eeg before the stimulus even happened all the way to about 600 ms after the Eeg happened.
CNSP: I like to use negative lags
CNSP: and a lot of lags after what I think is going to be the bulk of the Trf. Just to get a sense of one. The baseline that's going on before before the stimulus happens to see how much noise there is in the in the Trf. And then also toward the end to see the Trf. Return back to Baseline. So 200, minus 200 to 600 will give us that duration.
CNSP: We got 2 options for Lambda. Here we got one that has fewer lambdas that will make this go a lot quicker. This was actually a vestige from one of the previous codes that I took that does this on multiple subjects. This, the code works runs pretty well using this
CNSP: set of lambda. But if you're finding that your code is running slowly, you can comment this or uncomment this by pressing Ctrl, T on a PC. Comment. This one by pressing control, are on a PC. I don't know what the equivalent is in Mac, or you can just manually take out and insert this percent sign.
CNSP: But I'm gonna roll with this large Lambdas cause. That's what our large list of land is, cause that's how I ran the the tutorial in my practice.
CNSP: And then also the direction of the Tf, so we're Gonna start off with the forward model.
CNSP: I'm going to now load and prepare the ag and stimulus features, and I'll go through a little bit about what's going on here while it loads. It shouldn't take. Actually, it's already loaded.
CNSP: So there's a down sample if necessary. I think the stimuli are possibly saved in 128 Hertz. The Eeg is saved in 64 hertz. So that's what this is doing. Then we're gonna extract some of the stimulus features of interest.
CNSP: So as Giovanni said, we're keeping all of these in C and D format, which means that we're gonna have our main structure here with some fields.
CNSP: So I'm going to look at the stem E, and these are gonna have
CNSP: names, some indices for, like what stimulus. And so this is a very simple experiment. The indices are going to be the actual number
CNSP: of the index, because they were presented in order. But you might have an experiment where participants had something in different order. And that's where those would go
CNSP: same thing. Here, if you have different conditions, you would put it there. We're not going to use these, because
CNSP: these are all in order.
CNSP: And then you have data which is a cell array that is, a one by number of trials.
CNSP: and within that it's going to be time by features. So if I just plot.
CNSP: it's one thing.
Wow!
CNSP: Off screen that is so dumb, why is it doing that? But you can see the envelope here. I'm just going to get rid of that because I don't like that
CNSP: course. Other
CNSP: features are going to have different amount of
CNSP: columns. This one has 19 columns, because we're looking at 19 phonetic features.
CNSP: so on and so forth. And then we're going to have the Eeg data that's organized. Very similarly.
CNSP: it has a similar
CNSP: order, variable that you can use if you had to present in different orders.
CNSP: And then data is again a one by number of trial salary.
CNSP: Oh.
CNSP: great Mac is command plus slash and command, plus t for comment and on comment. Nice. Thanks.
CNSP: and in those data you have time by channels.
CNSP: We also want to make sure that all of the the stimuli and the each year the same duration.
CNSP: that's specifically, we're not specifically, but that's important for the natural speech data set, because the way that the data were epochs. It goes from the start of each trial to the start of the next trial, and the stimuli stopped a little bit before that. So you have to stop, or you have to make sure that they're all the same size. We also
CNSP: normalize the stimuli and the Ag.
CNSP: We usually will take the Z score of the EGAZ. Score of any continuous stimulus, or if using in a an envelope, I tend to like to divide by the standard deviation. So none of the envelope falls below 0. But it actually doesn't really matter. So like the Z score and center.
CNSP: Oh.
CNSP: so apparently there were no, none of the preprocessed.
CNSP: There's a question I'm unable to find. Pre data sub 10, only data sub 10. So apparently there's no pre-process data in there, you should be able to change everything to say. Data sub.
CNSP: 10. So line 107. You can just delete
CNSP: free. and it'll work just the same. You might have some like
CNSP: worse results, because it's not pre-processed. I'm sorry. I thought that there were
CNSP: I thought that there were the preprocessed data in there.
CNSP: So moving on
CNSP: Let me know if you have any problems changing that, or if that creates any headache for anyone
Giovanni Di Liberto: or Aaron, you want to show them how to run the preprocessing code for one subject, for example.
CNSP: Yeah, sure, I could do that.
CNSP: Actually, I mean, let me make sure that that's in.
CNSP: Send this. Make sure this is being sent to everyone
CNSP: possible to send a link. Send a file.
CNSP: They're not.
CNSP: I'm not sure if this script is in there, but
CNSP: you can. If it is.
CNSP: you can just specify.
Giovanni Di Liberto: I don't see it. I don't see it in the email.
Giovanni Di Liberto: I just see the Csp tutorial and Cnsb.
Giovanni Di Liberto: and there are 2 obs tutorial.
CNSP: Alright. Well, give me a second. We've got 2 h here.
Maya מיה مايا: I'll send this out real quick.
Giovanni Di Liberto: Yeah, you share it in a second, can you hear me?
CNSP: Yep.
Maya מיה مايا: so I actually downloaded this stuff and ran the pre-processing and like
Maya מיה مايا: Created the files for the Cnsp example, one.
Maya מיה مايا: So basically, the one of the first sections of that file, the pre-processing, no box. Yeah.
CNSP: Yeah. So you can actually just open this file right here. Example
CNSP: one for a Trf, or maybe any of these examples. But I always I just use example one. And that's actually where I got the I made the Pre processing script apologies everyone. I'm sorry. I thought that I attached that to the email. What I would do is just for sub equals 10,
CNSP: and then you probably want to go down here and set a break point so that it
CNSP: clear
CNSP: you can see what's going on here. Why is it doing? 180, I know what's going on.
Giovanni Di Liberto: and because the subjects are loaded in alphabetical order, and numbers are treated as not as Digits
CNSP: Yup, so you might need to just run all of the subjects.
CNSP: Maya. How long did it take you to do that?
CNSP: I don't know. Maybe 10 min something like that cool. So if if it's important for you all, then, like you can run that file, I would just put a breakpoint at line 147, so that you don't go through and analyze
CNSP: the Mtrf. For all of the subjects.
CNSP: This should import the the data sub file and and save it back into the same folder.
CNSP: I will
CNSP: soon as I figure out how to
CNSP: reply. Also all the blind carbon carbon copies in the email. Send out the C. And D preprocessing script. But I'll continue on with the tutorial, for now.
CNSP: thanks for that, Maya, by the way.
CNSP: So just running the dependencies again.
CNSP: and the parameters again.
CNSP: Loading. Subject 10.
CNSP: Again. You can delete this for now, or you can run the preprocessing script on your own.
CNSP: So now we have our
CNSP: our subject data loaded. So from here. after you get this section loading and preparing agn stimulus features.
CNSP: any subsequent section can be run independently of of any other section. But I'm going to go in order because it makes sense.
CNSP: So the first thing that we do is we want to fit a trf, and so we do that by taking some stimulus feature
CNSP: stem E, which is the envelope and the corresponding Eeg
CNSP: and jam set through Mtrf. Train here, I'm selecting a random trial. You're gonna get a different random trial. It's also gonna look a little different because you're not using preprocessed data. Possibly
CNSP: but so what I'm gonna do is I'm going to highlight and press F, 9 on. PC, I don't know what this is on, Mac. You can also just do the control enter or whatever the equivalent Mac function is just to execute sections.
CNSP: Actually, I'm just gonna do that here just to get both of these.
CNSP: Damn it!
CNSP: Why is it? Oh, I know what's going on.
CNSP: It's my display settings.
CNSP: I'm gonna have to
CNSP: lower my screen layout again. So apologies if people can't see
CNSP: the text so small.
CNSP: There we go
CNSP: alright. So here we go. These are the 2 things that popped up. This is a single trial. Trf. this is a multi-trial trf averaged across trials.
CNSP: I can see the single trials here. I've changed.
CNSP: If we execute this code
CNSP: that have to highlight all of it.
CNSP: Notice that it changes. I'm just gonna execute it a couple of times. You see that there's some consistencies there, and that's really good. I think this subject is is a little.
CNSP: maybe maybe too good for to illustrate some of the things that we're talking about today. But each trial has some consistency to it. You see a little peak in the Gfp. Around 150 ms and one around 80 ms. It's pretty consistent across time. That's great. That's what the Trf actually latches onto. So
CNSP: usually so we can
CNSP: do that.
CNSP: So and also, you see that the the multi-lag is just a better fit lower noise across trials. But the whole point, or one of the whole points here, is to predict EG.
CNSP: From this model, so we could take this model
CNSP: and then use Mtrf. Predict
CNSP: to predict the Eeg. And I'm gonna run it like this.
CNSP: And that's how long it takes to run the output of predict and the output of Mtrf. Cross. Val will be the statistic structure that's got 2 model fit metrics in it. The piercing correlation between the predicted signal and the actual signal on that trial. Here we've got 20 trials by 128 channels.
CNSP: and it's going to be like that because there are no lambdas in involved here, sometimes in a cross value, you'll have a lambda
CNSP: dimension. So let's look at that correlation.
CNSP: Oh, point 5 pretty small.
CNSP: That's about normal for Mtrf,
CNSP: But the problem with doing what we just did is that since this is a
CNSP: this is an average model across all trials. And we're going to use that to predict each trial. It's using some of its own data to predict itself. So we're going to get a little bit of overfitting
CNSP: to that. And so the way that we
CNSP: combat overfitting a little bit here is that we'll take some of the trials. Mtref will do 19 or sorry N. Minus one trial will fit a model on those trials, and then we will take
CNSP: the fully fit model from those 19 trials and predict the left out trial with it.
CNSP: So to illustrate that I'm going to change some of our hyper parameters.
CNSP: I'm changing. I'm removing the pre stimulus and some of the post stimulus time here. Because what? When we do prediction, we really want to focus on the lags that are predictable, and by that I mean the lags that have lots of components to it. So here I might say that it's 0 to 300. Just by looking at this
CNSP: trf, but I chose 400 at the, you know, when I started this tutorial. So we're just gonna
CNSP: press on with that
CNSP: this actually just makes it easier to predict. It's not trying to predict what is probably noise out in front of the stimulus, and what is probably a little bit less signal out after the stimulus.
CNSP: and then we can run an Mtrf. Cross, Val. It takes a little bit longer to run than train and predict.
CNSP: But what this does is it's going to cross-validate or optimize a model and tell us which lambda produces the best predictions for those data. So here we've got our best lambda, which is lambda 4.
CNSP: Just a little bit of something about the land is here.
CNSP: So this is our range of lambda. The higher the lambda. The more regularization of the more emphasis that's placed on the diagonal of the xt X matrix. Usually the lambdas that work really well, or you know in this range, and that's what we found here. It's 10. It's a little low, for you know what I'm used to. I'm used to seeing 100 or 1,000. But
CNSP: whatever is whatever for each subject, and that's kind of the point of lambda optimization.
CNSP: We wanna look at the correlation coefficient of that
CNSP: for the prediction accuracy. we see that it's a little lower. That's probably because of some of the overfitting
CNSP: that occurs when we use the stimulus to predict itself. But we still do have a little bit of overfitting here, because we're optimizing. We'll come back to that later.
CNSP: I'll fit
CNSP: a model with that best lambda
CNSP: and then visualize it.
CNSP: If you notice here, they're very similar, they're slightly different, because they use different lambdas.
CNSP: That just means for this subject. This model does slightly better than this model. which is fine. That's what the point of optimization is.
CNSP: But, as I said, there's still a little bit of overfitting going on in there. It's because of the optimization step. What actually happens is that
CNSP: that's not what I wanted to see.
CNSP: I've got to squeeze.
CNSP: Oh.
giorgia cantisani: I don't. We have a question in the chat?
giorgia cantisani: So from Camilla. Can you explain, lambda? A bit slower? Are you testing many lambdas per subject trial model? What's the what statistic do you use to to select the best one
giorgia cantisani: up?
giorgia cantisani: And yep.
CNSP: yeah. So, Lambda, let me see if I can find good picture.
CNSP: So
CNSP: here's an image of some XY data, weight and size, whatever that means.
CNSP: But
CNSP: we can imagine all of these thoughts relate to our data are are from the data. But we might want to like cross validate our data. So we're going to use like a small subset of data. Or maybe we don't have a lot of data.
CNSP: But anyway, so the the fitting data in this is actually just the red dots
CNSP: right? So we're gonna fit a line to those red dots.
CNSP: and this line is just what linear regression would say is the best fit for that one. But the real best fit is actually blue. We don't know that, though what lambda does and and why they would use ridge regression in this example is that adding more, lambda will actually flatten out this red line until it matches the blue line
CNSP: right?
CNSP: But you don't know which one's the best yet. So you fit a bunch of lambda, or you can search for it or optimize it. But you fit a bunch of lambda and then predicted, on, you know, some left out data. So these green dots.
CNSP: and the lambda that pops up or is the most optimal, will be the one that matches the red line
CNSP: to something closer to the blue line
CNSP: that make sense.
CNSP: because we don't know it. We don't know where this red line is gonna be. So we could have
CNSP: green these 2 green dots be
CNSP: what we're fitting on. And so now the slope. it's too low.
CNSP: so we don't know which direction lambda is. Gonna go. It all depends on the fitting data. So
CNSP: we that's why we would optimize Lambda individually. Let me see your question again.
Mick Crosse: And Aaron, did you just
Mick Crosse: just to add to that.
Mick Crosse: Can you hear me?
Mick Crosse: fitting lambda, and
Mick Crosse: individually. or even say within one subject
Mick Crosse: on individual conditions?
Mick Crosse: just to kind of a note of caution.
Mick Crosse: If you don't have
Mick Crosse: very much data. So say you're working with maybe 5 to 10 min of data or a particular condition or subject. The lander that you will,
Mick Crosse: I suppose, converge on.
Mick Crosse: and the model that you will have as a result
Mick Crosse: may not look very. How would you say, trf, like or have components that resemble neurophysiological components?
Mick Crosse: And the reason for that is because you don't have enough data or your data are too noisy. And so III had someone email me recently where they were basically pointing out
Mick Crosse: how for some
Mick Crosse: conditions, they had these beautiful looking tear apps for others. They were very noisy, a lot of kind of
Mick Crosse: high frequency isolating components.
Mick Crosse: and that was most likely due to noise in their data. If you're
Mick Crosse: trying to fit a model, if you're trying to fit Lambda just for the purposes of visualizing your trs, to be able to
Mick Crosse: make some sort of inference about the neurophysiology of your data.
Mick Crosse: then oftentimes the best way to do that is. if you can't ensure that you have a lot of data
Mick Crosse: is to
Mick Crosse: actually hand. Pick a lambda that enforces enough smoothing
Mick Crosse: so that
Mick Crosse: the resultant model weights begin to look
Mick Crosse: tierraflic.
Mick Crosse: Otherwise, if you try to fit using cross-validation, it'll probably just over fit to the to the data, because you don't have enough.
Mick Crosse: So you know, we use cross validation for the purposes of
Mick Crosse: predicting new data, whether you're talking about forward or backward models.
Mick Crosse: But when you want to
Mick Crosse: you know, visualize the weights. And do you know, time, frequency analysis? On different components.
Mick Crosse: you have to be careful about using cross validation to do that. If you don't have enough data.
Mick Crosse: So we talk a little bit about that in the frontiers paper. But that's just kind of another caution. When you are fitting lambda tier data.
CNSP: Thanks, Beck, just to add on to the add on
CNSP: the high frequency oscillating components that we usually see on the over fit data that is explainable directly by this. I think so. When you don't have enough data, the fit tends to vary wildly like there's lots of excursions from the true mean. And so
CNSP: I think that's probably why it tends to not be smooth, because the the best fit to noisy data can be above or below. And it's sort of, you know, this, this line just goes back and forth. It's pretty unstable.
CNSP: This is actually taken from a really really great video
CNSP: by a dude on Youtube, whose channel is named Statsquest. who has a really great example of ridge regression and actually like demonstrates how you can fit.
CNSP: Using one data point I'm putting that into.
CNSP: Oh, wait. I'm putting that into the the chat. Now.
CNSP: you also get a bonus song for each video. This guy writes these really dumb songs that are hilarious. So I highly recommend that channel. Really good explanations for
CNSP: complex statistical things.
CNSP: To finish answering your question.
CNSP: we do fit for each subject. It takes multiple trials to fit. So we we include all trials at once, and usually will fit different lambdas, or like optimize lambda for different models.
CNSP: and like, I showed. yeah. these are the 2
CNSP: metrics that we use to select the best lambda we have Pearson's correlation and then mean squared error.
CNSP: They both serve different functions. I tend to use the the Pearson's correlation because it's invariant to scale and whereas error is not so error will enforce that the signal is
CNSP: at the same scale, and our won't.
CNSP: But what we usually do is to
CNSP: actually, I'm going to have to. I'll show you.
CNSP: BA,
CNSP: I don't know what I'm doing
CNSP: anyway. What you would.
CNSP: What I you should see is these 11 lambdas will each have like a different.
CNSP: Oh, I know what I did. I've got best lambda in there. What a dumb mistake!
CNSP: There we go!
CNSP: So! Lambdas along the x-axis and the prediction correlation along the y
CNSP: it looks like it's an asymptote, but it's actually not an asymptote. So if we zoom in here.
CNSP: it actually gets marginally better. They're not exactly sure if this constitutes the best lambda for this person.
CNSP: but it's sufficient, and this is kind of a common
CNSP: look for a tuning curve. Another, probably more common, is an actual tuning curve that's just like an inverted hue. So we just usually
CNSP: choose the the highest value there. There's some like issue with like maybe choosing like the knee point of it that might produce a bit better prediction accuracies, especially if you're like getting towards higher Lambdas, but it's usually just the Max of the of this middle dimension, whichever one is 11 here.
CNSP: So I think that pretty much answers all of your questions.
CNSP: So like I said, there's still a little bit of overfitting there, because we're optimizing
CNSP: if it's a good fit, it's not gonna over fit very much, and that's what this will. That's what this will illustrate. So I'm just going to start executing here. So
CNSP: so the optimal number of trials we, I think Ola might talk about this tomorrow, but like
CNSP: we found that like 3 to 4 min of data like where you start seeing an asymptote. So if you like data by like prediction, accuracy, it starts to ask them to and like this knee points different for each individual. But it's usually like like 4 to 5 min of of data.
CNSP: So to get rid of this like optimization,
CNSP: overfitting what we do is we we withhold some data from the cross validation predict procedure. So here I am, putting all of the data into train.
CNSP: our train and S train, and then I'm removing
CNSP: the current trial. And then I'm making the test data. That current trial. So I've got 2 sets here, one with 19 trials and one with one trial. Then we're going to do a cross validation on 19 trials find the best lambda
CNSP: record that cross validation prediction. Then we're going to fit a full model with all of those trials and the best lambda.
CNSP: And then we're gonna use that model. predict those that left out trial.
CNSP: And we're gonna just run through all the trials rotating, which one is the test trial.
CNSP: and then so it should. Have all of those test trials in both the Cross Val and the left out data. So that's what I'm plotting here. And we see these. This is a pretty well fit data set. There's enough data here.
CNSP: and maybe actually, this is back to where this might be too good of data. I was, I tried a version of this where I only use like
CNSP: 3 or 4 min of data to fit. And I could. I could just never get this to work. It only got better. So like the left out data is slightly better than the cross validated. And that's probably just because there's very little, if any, overfitting that's going on in this step, because there's so much data.
CNSP: So really, you only like this is a
CNSP: probably more important whenever you are experiencing like noisy data, or if you have less data than you know, what might be a well fit model.
CNSP: So now we have a model. Now we have a cross-validated and a left out like evaluation of that model.
CNSP: And it's doing it's performing at some level. We need to know
CNSP: how well it's performing. And one of the ways that we do that is to just
CNSP: determine its noise floor.
CNSP: If we have a fitting procedure. That's not matching stimulus. At least this is the way I think about it. There's other ways of thinking about it. We have stimulus and response data that are not matching. How well can we, you know, fit a Trf to those
CNSP: to those data. predict left out data. So that's exactly what we're doing here.
CNSP: The first thing that I'm doing is to ensure all the trials are an even link, because they're not stimulus and response are matching, but across different trials they were all different links, because they were cut to be to end at a like a reasonable spot in the story. So it wasn't jarring for the participants. So each trial is a little bit dur different duration. And I'm fitting a different cross validation on that.
CNSP: And then I'm going to run a whole lot of iterations.
CNSP: I'm not gonna actually do this. I'm going to just preload this which I've also sent to you. Here's a little trick for people in Matlab. You type at 0, and it won't execute that. But I'm gonna run a whole lot of different iterations where I
CNSP: fit a model and then predict a left out model.
CNSP: On its own data to see how well, it can do.
CNSP: And then we're going to compare that distribution to what we actually observed on that model or observed.
CNSP: And the data.
CNSP: So I just really quickly ran a thousand iterations of the Mtrf and that's what this is. This is a thousand
CNSP: trf models that have been evaluated on shuffled data. So we just shuffle the data labels that's done
CNSP: like this just with a Rand. Perm.
CNSP: that's it. And then we compare that to what we observe. So like our observed, is still again about OP.
CNSP: Point 5, and the p-value on that are just tiny. So we can get the p-value by seeing how many of the null responses are bigger than the actual responses?
CNSP: If we look at the p-values.
CNSP: actually, a thousand iterations was not enough to get the actual p-value for some of these channels.
CNSP: There's one that was channel 50. Sorry you are a loser.
CNSP: Once you have this model validation metric, you can use it to. You can like average across channels and see if those channels were better than a null model. You could do some cluster statistics on these you can enter these into like an Fdr. Correction. But here's your p-values.
CNSP: Correct them. However, you feel the need.
CNSP: and that will tell you how good the model is. This model, of course, is very good.
CNSP: There's like no samples above that that are
CNSP: from the null model.
CNSP: Of course we might be interested in how different models compare
CNSP: And those models can be from the same data, or they could be from different data. So we might be interested. What I'm showing here is like, how well different speech features can predict the Ag data. We might be interested in
CNSP: different
CNSP: trials predicting data. Or we might be interested in different conditions or different people or something. But.
CNSP: generally speaking, what we'll do is we can fit a model for each of those things that we're interested in.
CNSP: And so what I've done here is I've fit a model to the envelope, set a model to the spectrogram, set a model to the phonetic features. And I've combined the phonetic features in the spectrogram model into a joint model
CNSP: so that we can look at how well these models. Compare, I'm basically trying to replicate Giovanni's 2,015 current biology paper.
CNSP: which I actually did. Here we see that the ex spectrogram performs better than the envelope that makes a lot of sense, because it's resolved better over frequency. It's the same data, but just more resolution.
CNSP: The phonetic feature model does
CNSP: perform a little bit better than spectrogram. But that's not really the comparison that we're after here, like I mentioned earlier in the slides. What we're actually interested is in whether the spectrogram and the phonetic feature model perform like uniquely, if they are like actually like finding unique parts of the Eeg. And to do that we can combine them and compare that to the constituent pieces.
CNSP: And that's what we see here. So when yet when we add spectrogram and feature together, we see that it performs better than either model alone.
CNSP: implying that not everything that's in both of these is shared variance. There's some unique piece of variance in there.
CNSP: Of course, sometimes we need to take a little bit more care to make sure that we're not comparing models that are drastically different in terms of their numbers of parameters.
CNSP: And so one way that we can do that is to employ the permutation randomization. Approach again.
CNSP: Again, we're I have to make sure that the links are the same, because we're gonna do some shuffling.
CNSP: This takes a long time. So I didn't actually execute it. But so here, I'm only gonna do 50 iterations. Because we're not interested in the actual distribution from each subject. We're only interested in
CNSP: a single point for each subject, and we can compare groups of subjects together. But what we're going to do is we're going to fit another model
CNSP: that's similar to this one. It's going to have the same number of parameters in it.
CNSP: but only one of its features is going to be intact. and it's going to be the feature that we care about. And then this model is going to form, like basically a baseline model, a null model that has
CNSP: all of the information in it that we don't care about.
CNSP: So for me. Here that would be
CNSP: the spectrogram!
CNSP: So what we're trying to do here is we're trying to see how well the features encode eeg better than the spectrogram. So we have the spectrogram as part of this model, but we also have the phonetic features.
CNSP: But we're going to disrupt this phonetic feature somehow. Here I'm shuffling across the feature space.
CNSP: You can also shuffle across the trials.
CNSP: You can also shuffle by like reversing the
CNSP: the feature and time you can circularly shift, which means to take some part at the end and bring that to the beginning, and then, you know, make the beginning of that ending part the actual beginning of the stimulus, just any way to disrupt the
CNSP: the features.
CNSP: And so we will then compare
CNSP: Spectrogram model
CNSP: to the spectrogram plus feature model.
CNSP: So this spectrogram for plus feature model obviously performs better than the spectrogram model. But when we add those extra nuisance regressors in there that are just shuffled feature model. It does get slightly worse. It's pretty marginal. Some people go up, some people go down. But generally this is about the same. So now we compare these 2,
CNSP: if we really are like worried about like number of feature parameters. And and this will be a more fair comparison with respect to that.
CNSP: So so far we've gone in the forward direction.
CNSP: and I like going in the forward direction, because I understand it a little bit better. I know what's going on.
CNSP: I can look at this and know this is the actual, easy response. It's physiologically interpretable.
CNSP: But one of the issues with going in the forward direction is that there's not much sensitivity.
CNSP: And so we can turn to a different direction
CNSP: to improve our sensitivity. And I'm gonna change a few hyper parameters to do that again. I'm looking at a model. So I want to.
giorgia cantisani: There is a little bit. There is a question
giorgia cantisani: from Maya. Yeah. Yep.
Maya מיה مايا: Take can you hear me?
giorgia cantisani: Yes.
Maya מיה مايا: okay, thanks. So sorry. Before. Just continue on to. Yeah. Okay, I wanted to make sure. I understand something about like the different models.
Maya מיה مايا: So, evaluation of the different models with the different predictive, like predicting features. So like, if I understand correctly, like
Maya מיה مايا: Line 3, 87 of the code.
Maya מיה مايا: So the combination of the spectrum phonetic feature model.
Maya מיה مايا: So that's like,
Maya מיה مايا: an additive model, in a way of these 2 features. Modeling the eager response.
Maya מיה مايا: so like with the yeah.
Maya מיה مايا: but if, like, maybe in this combination of features, it wouldn't really make any sense. But there are cases where you would like to look at an interaction of 2 features and modeling response. So
Maya מיה مايا: like.
Maya מיה مايا: yeah, you can make a third type of a feature. That sort of combines the 2, I guess. But like, would you have anything to say about that?
Maya מיה مايا: I've done that before?
CNSP: I don't know how feasible it is.
CNSP: Generally
CNSP: The way I did it is I had.
CNSP: I was reporting, eg. When people are watching visual speech. And then I had some stimulus features from the speaker, so one of the stimulus features we had.
CNSP: we're points on the mouth. So we track these 2 points and we track these 2 points. I'm gonna move
CNSP: where I can see your face. I'm actually looking toward you. We track these 2 points and we track these 2 points. We took the distance between those, and we have these 2 like vectors, for, like how wide the mouth is at any given point, how tall the mouth is at any given point.
CNSP: and they were like yoked sample by sample. and we did
CNSP: 2 things. One is, we fit a 2 feature model to the Trf. Put the horizontal and the vertical in a separate features and predict it, and it does its own thing. And then another thing we did is we multiplied the 2 vectors, and then did a one feature model.
CNSP: That's the best way that. So I did this in response to someone asking about
CNSP: mouth area and not mouth width and height, because a lot of things in the literature has to do with like mouth area that was the best approximation that I could
CNSP: like get was just a multiply those things. But like, essentially, when you're doing linear regression, you can just multiply your input features
CNSP: and get their interaction. I don't know if I would feel comfortable like saying that
CNSP: you're going to get a perfectly reasonable
CNSP: and an equivalent to like a linear regression interaction. If you do that with the T. Rf.
CNSP: Nick might have like more thoughts on that, or Giovanni, or someone who's more expert in like the Trf. Than I am. But I've done that now, like with these models you've got
CNSP: you got a spectrogram.
CNSP: which is a time series. But there's 16 of those time series, or however many frequency bands you have. You also have a phonetic feature model. That's a different type of stimulus. It's either a one or a 0. It's one. If it's on.
CNSP: If that phone name is occurring, or friend of feature is occurring at that time point where 0, if it's not so, most of the
CNSP: most of the features is is zeroes.
CNSP: So I'm not sure how you would do that practically with this type of model, where you have either a stick or an impulse, or like a discrete model
CNSP: and a continuous model. Or if you have multi variant models, you're gonna have to do like pairwise
CNSP: interactions, it seems.
CNSP: But I mean other than that there's no real way to do this with. Trf, it's
CNSP: something that's interesting and something that I think that I would like to see the answer, for I'm not sure if
CNSP: I can't envision like how you would want to do that, or why you would want to do that other than the question that I had from the reviewer.
CNSP: But I'd be very interested in like those cases, and like what we can do with that? Yeah.
CNSP: if you have like thoughts on that email, the Csp Gmail, with like some thoughts, I'd love to hear them
CNSP: might be something that we can add to the toolbox
Maya מיה مايا: cool. Thanks very much. So actually, your example really answers a lot of things, and sorry for taking a second more of time, but
Maya מיה مايا: like in the way you could imagine that there would be like an effect for the like X axis y axis, and an additional effect like
Maya מיה مايا: for the like. The the interaction between them right? So
Maya מיה مايا: like would running a model with these 3
Maya מיה مايا: features be feasible, or you think not
Maya מיה مايا: cause. If you subscribe to situation where just running the direction and just the other 2.
CNSP: if if I was truly looking for an interaction
CNSP: between 2 external like feature spaces
CNSP: on the output variable. I would probably run the features
CNSP: separately
CNSP: and with sorry. Let me rephrase that. So I'm not some clear. I would run a 3 feature model.
Maya מיה مايا: both main effects and the interaction term.
CNSP: Yeah, but like running the interaction on like by itself can also make sense. I wasn't running an interaction, I mean, I was implicitly doing. I wasn't intending to find an interaction
CNSP: so like my mouth area.
CNSP: whatever it was, was not intended to be an interaction. But I mean, it essentially was.
Maya מיה مايا: okay. Great. Thanks very much.
CNSP: Yeah. Don't worry about
CNSP: wasting time. We're like two-thirds through here, and there's still 45 min left. Lots of time to talk.
Camila Zugarramurdi: So I have a question
Camila Zugarramurdi: before you go into the backward model when you compute the forward model. So you try different lags, as you show in the matrix.
Camila Zugarramurdi: do. The legs have, like physiological validity. In a sense. Do they mean something about how this subject is processing the stimuli. Does that make sense? And is it saved for each subject somewhere? Or you can actually use it as a say, dependent, variable.
CNSP: So
CNSP: this is saved. So this is just how let me go to the code for that.
CNSP: So when you fit a model, you just use Mpr. Train
CNSP: and it spits out
CNSP: this variable model.
CNSP: No variable looks like this. It's got
CNSP: the weights.
CNSP: This one here, I believe, is for features. So if you you're doing a forward spec. Actually, I could. Just no, I don't have that yet. If you're doing a forward spectrogram, this should be 16
CNSP: and then this is your
CNSP: your bias term, which is your intercept term. We usually just ignore this, and it's just kind of
CNSP: we don't really do much of anything.
CNSP: the time variable
CNSP: and then other stuff. So here's, you know, the important part of the model, and that gets fit
CNSP: on
CNSP: whatever you input into it. So whatever this is, and this is is what that model represents. In this case, it's a single subject.
CNSP: This can be saved. And in fact, like, if I was like
CNSP: something like that. I could something like that
CNSP: and save all the models in a model structure. And then we have
CNSP: what's called in the
CNSP: libraries or somewhere, and probably not. There is an Mtrf model average that you can input something like that into. And it will average across subjects or across whatever iterations
CNSP: I would fit for each subject.
CNSP: And then you can then take
CNSP: this can be analyzed just as the same as a an Erp. So in terms of like the Gfp, we can like take the Gfp here, where we can take the actual like on specific channels. We can take the the channel weights. We can find a peak here and see how far that is past 0.
CNSP: You can analyze these in any way that's like similar to the
CNSP: similar to an erp. Something that I would be like caution against is to
CNSP: fit trfs or to optimize truffs and then compare T. Rs. so the Trf waits. If you notice I've put here that it's an arbitrary.
CNSP: Why did they do arbitrary units?
CNSP: that's because the the lambda. Actually, it will like make the it will move
CNSP: the the slope.
CNSP: So the more lambda you have there, the more the slopes moved and the flatter things get, and the the shorter the the less amplitude the Trf has. So if you compare, like my trf to your trf, and my Trf has a higher regularization value. Yours is going to be way bigger than mine.
CNSP: and mine's going to be waste more spread out in time. So what I usually do. If I want to compare to your S is, I've applied one single regularization parameter to everyone.
CNSP: Just I might optimize everyone, then take like the mode, or median, or mean, or something like that. but I think it's pretty important to get like
CNSP: all of the T. Rs. In the same like amplitude space, and that. And you do that by using
CNSP: the same lambda? Can you normalize the weights? What do you mean by normalize the weights you mean like, put them on a scale between 0 and one. If that's the case. I mean, you could do that
CNSP: before you average them together. Like, you know? Yeah.
So to maintain the topography for each subject and utilize them, then you averaged them.
CNSP: Yeah, you can do that. But then you're getting rid of a lot of the amplitude information, because that the amplitude is always
CNSP: are the same. Normalizing before averaging
Emilia Fló: within a subject should maintain right? Like N, one type, p, one, like
Emilia Fló: the components should be.
Mick Crosse: II wouldn't
Mick Crosse: advise, you know, normalizing before you average. II think you're better off normalizing
Mick Crosse: the input data to your model.
Mick Crosse: your Eeg data, your your stimulus features? and then
Mick Crosse: whatever comes out, the other end in terms of weights average.
Mick Crosse: I mean the the scale on the on the axis, your y-axis for your weights. It's it doesn't really have much meaning.
Mick Crosse: it's more so just to be able to.
Mick Crosse: Look at the relative amplitude compared, you know, across conditions or subjects. So
Mick Crosse: personally, I'd advise against normalizing weights.
Giovanni Di Liberto: Yeah, have a comment. There! Sorry I'm I'm on the train now, so my
Giovanni Di Liberto: my microphone is terrible, I agree. Make in general, but there are some situations where you can have, maybe have a little data.
Giovanni Di Liberto: and you have some subjects that if you if you optimize the lambda
Giovanni Di Liberto: on the single subject, you can have some subjects that really explode.
Giovanni Di Liberto: So there there might make sense to normalize. But if you are doing what Arnold was saying before, would you select one number for all the subjects. Then it doesn't make much sense to normalize anymore.
Emilia Fló: Okay, thank you. Yeah. I was thinking of this, you optimize a model per subject, and then you have, like a topography for each one, for the weights. So you can normalize it. Take the maximum value per subject, and you have this normalize the Trf per electrode. And then you can average across subjects like that was.
CNSP: But again, you're throwing out your amplitude data because everyone's yeah. It's relative, it would be relative. So if you have 2 conditions, maybe it's you wouldn't mind.
CNSP: Yeah, I just wouldn't do any any statistics or any like inference on the Trf. Waits themselves at that point. But yeah, you could totally do that. And in fact, we I've played around with like z-scoring
CNSP: trf waits
CNSP: when doing, like generic data or sorry generic models. That's where you would combine models from like N minus one subjects and then predict data from the left out subject with that generic model which is useful in cases where you might have, like 3 or 4 min of data from everyone. So like fitting an individual. Trf.
CNSP: probably not
CNSP: not feasible with 3 to 4 min, especially in the folks that you have to deal with, 3 to 4 min of data. So like.
CNSP: you know, people like babies? So you can
CNSP: utilize all the data that's there from all the other subjects, and then average that all together, and so like get more, more time, as it were, in your trfs by using other
CNSP: participant time. And then then, I think normalizing across subjects actually helps, especially if you have one or 2 subjects that drive
CNSP: the Trf, so you can get a better generic model. But yeah, like what we were talking about, though, was mostly for and analyzing the Trfs themselves, which I would not recommend. Normalization
CNSP: cool
CNSP: again. Feel free to interrupt me. I like interruptions
CNSP: so like, I said, this is in the forward direction the
CNSP: the ability to predict Eeg is hard. Eeg is noisy. and we're also using a one dimensional stimulus. In this case the envelope to generate all of those predictions.
CNSP: the reverse direction we get to use all of the 128 responses.
CNSP: To optimize a prediction of one thing, the envelope. So we can really
CNSP: improve our signal to noise ratio.
CNSP: So we're going to change a few hyper parameters just to visualize the model. Do a control entered here?
CNSP: You see, first of all, that it takes a lot.
CNSP: Excuse me while I deal with this here.
CNSP: Oh, huh!
CNSP: Sorry about that. Fix all this stuff that I did.
CNSP: Sorry at.
CNSP: Cool.
CNSP: So you can see it takes a lot longer to do backwards model. And this is part of why we have that short set of lambda. So if if this is taking a long time for you,
CNSP: then feel free to use that short set of lambda. You might want to change some of the lambdas to something higher. The lambdas that are in the short set are like minus 2 is 2
CNSP: and backwards model, tended like
CNSP: Hi Lambdas. You might want to change this from like 4
CNSP: to 8, or like 6 to 10 or something like that.
CNSP: Anyway, you can see it was still running. There
CNSP: is a good time for questions, or
CNSP: whatever, if there are any.
CNSP: there we go.
CNSP: So here's your backwards model. You notice that it goes backwards because the way that pops out it's it's in stimulus time. So this it's almost like a spike triggered, averaging, or or like
CNSP: where you're
CNSP: averaging the Eeg responses that occur as a result of the so averaging the stimulus responses that are occur as a result of the Ag. You see that they're like mostly mirror images
CNSP: in the sense that you've got these channels here. Notice that this red, purple, yellow, blue. correspond to these channels here red, purple, yellow, blue. Those are the same channels. But you notice something that happens
CNSP: close to the baseline is that over here in the forward model, we've got this nice looking like dipole, where, like the the responses are sort of like separating in in like potential space but over here, because of the like, the interactions of each
CNSP: across channels in the backwards model where the backwards model is able to actually use some of these channels antagonistically to like sort of like, increase the sensitivity. And
CNSP: when we compare what this model can do to this model, it's it's like, not a question. So we've been looking at the average forward model prediction scores, which is a little bit misleading because the prediction scores have a
CNSP: have a topography. They are distributed across the scalp with the biggest one or the highest predictions, usually right here, right over where auditory responses are in Eeg.
CNSP: But we're just taking the average across the scalp, which also includes things around the back of the skull and the sides of the skull.
CNSP: So we're averaging a lot of like bad stuff in there. So if we look at the maximum prediction, that's just the channel that can do the best.
CNSP: and it still like doesn't even touch the backward model. So if you really want to like, increase your sensitivity, that's what the backwards model is for takes a lot of time. So you have to be careful about, you know, running it on a
CNSP: a machine that's, you know, doesn't have as much memory or as a slower processor. If you
CNSP: run into those issues.
CNSP: there is some
CNSP: I running something. I am running something. Was it running?
CNSP: Oh, it's just that section
CNSP: gonna cancel that
CNSP: if you have like memory issues, there is a
CNSP: an input that you can modify that will make it go slower. But it also requires less. Input. So the the default requires more memory, and it's faster.
CNSP: And what that does is it pre-computes all the covariance matrices and stores them in memory, and it doesn't have to compute them when it's doing the leave leave one outcross validation.
CNSP: Otherwise it does them
CNSP: while you're doing the lead one out cross-validation. It's a bit slower.
CNSP: We also might be interested in when the response
CNSP: is most evident in brain activity. So and II don't mean, like the Eeg fluctuation, but, like which time lags are, are best able to predict the Eeg.
CNSP: And so we, the Mtf. Toolbox, can also fit what's called a single lag analysis by including this.
CNSP: And this is it.
CNSP: By including these 2 inputs. And what this does is it will fit and evaluate the
CNSP: Let's hear fit each and time. Lag individually.
That's all right.
CNSP: So I'm running that
CNSP: give me a second to figure this one out.
CNSP: I know what's going on here
CNSP: just need the time series
CNSP: I can use model for actually
CNSP: sorry about that.
CNSP: This is what happens when you when you are.
CNSP: when you run this. not expecting to run in the in the tutorial.
CNSP: So yeah, in 5 46, exchange model to model 4. We already have the those time points from
CNSP: this model that we fit there. I was trying to use this model, but since I didn't run, it's there.
CNSP: and we can see the blue here is is using the Mtrf. Cross Val with the single lag.
CNSP: something that I sometimes like to do, especially with some of our multi sensory stuff.
CNSP: because we like to. We we sometimes see a little bit of smearing in time, and we have sometimes see
CNSP: these little bumps that occur like before 0. And sometimes we want to be able to see if they are, you know, temporal smearing or something else. So what I also like to do is, I have this little trick that I do with the Mter toolbox.
CNSP: and that I can I will fit a regular multi lag model, and then I will convert it to a single lag model by changing these 2 variables, and then predict
CNSP: on a
CNSP: the the
CNSP: individuals lags. So sometimes this can like reduce some temporal smearing, you can see like a little, possibly a hump. There, that's maybe like fleshed out. This is not a great example, but it's, you know, it's there to illustrate that there's a little bit more temporal resolution if you fit all the the lags
CNSP: at once.
CNSP: So the last thing I want to talk to you about is
CNSP: didn't know if I would have time to get here. That's why it's at the end, instead of up towards beginning. We're going to go back to the forward direction
CNSP: is is. It's a way that we're developing. It's still not done. It needs to be done. But we're still like concerned about some of the issues with it. As a way to actually remove
CNSP: the effect of some of the features before predicting other features, and by that I mean, like partialing features.
CNSP: So in the case of
CNSP: I don't have it up yet here anymore.
CNSP: Do have it here. In the case of these. where we're looking at comparisons across different features.
CNSP: We looked at 2 different ways of doing that. One way is we can compare the joint model with each of its constituent models.
CNSP: where we can compare it with another baseline model in which we've shuffled a feature of interest.
CNSP: And then, finally, one way that we can do is we can just remove that feature entirely from the aging.
CNSP: So that's gonna involve using the predictive power of the Trf
CNSP: by fitting a model, predicting a teg, and then simply subtracting that prediction from the actual Eeg.
CNSP: So this loop here will do that.
CNSP: And then we have all of this Eeg, New Eeg that does not have the responses to the spectrogram.
CNSP: So I'm gonna run this. See how long it takes.
CNSP: So here it's just fitting and training a model on each trial.
CNSP: removing the effects of that feature from the
CNSP: trial.
CNSP: And then we move on. And we're going to optimize several models. One, we're going to try to optimize to the spectrogram
CNSP: when we have no spectrogram in the model. And then we're gonna optimize to the phonetic features when we have no spectrogram in the model.
CNSP: and that's what this is looking at. So if we look at the Fs model, it's the same Fs model we've looked at. If we look at the F model, it's the same F model we've looked at, and same thing goes for the same S model that we've looked at. But then here we have an F and an S.
CNSP: Predictions that we have fit to Eeg that has no spectrogram responses in it. And so here is a better look at the unique contributions
CNSP: of this phonetic feature model. I emphasize the word unique here because it's going to remove, or this process is going to remove most of the responses that can be correlated correlated between either of the 2.
CNSP: So
CNSP: this is going to have mostly F in it, but some of the F. Isn't going to be there.
CNSP: This is going to have.
CNSP: mostly nothing in it. So this should have like no spectrogram, and then we fit spectrogram. This is just to illustrate that the process works. We're removing this and much information from the Eeg. Yet we still have this much information left in the Ag that could be fit to the feature model, but not the spectrogram model.
CNSP: We also have been working with a different approach to this Farhema Farhen Ahmed in our lab, who's just defended her Phd.
CNSP: And some others in the lab have come up with a different way of doing this, and that involves predicting all of the features that you care about.
CNSP: including the ones that you're trying to control for, and the ones that you're trying to measure, and then using
CNSP: Matlab's or whatever functions partial correlation. to get the responses that are
CNSP: only correlated between these 2 while controlling for these 2. So that involves
CNSP: again, we're going to fit and predict on different trials. So we're going to fit a model to
CNSP: fanatic features.
CNSP: And then we're going to predict Eeg to phonetic features fit a model to the spectrogram. And we're going to predict EG. Related to the spectrogram. And then we're just going to go through all of the channels.
CNSP: and we're going to correlate the Eeg data on that channel to the phonetic feature prediction on that channel.
CNSP: And then we're going to also account for the predictions of the spectrogram on that channel. And this is what that looks like in this example.
CNSP: They're not equivalent, because what I'm doing here is not optimal. In fact, most of these analyses and the tutorial script is not optimal, because I have. I've cheated a little bit.
CNSP: not really cheated. I've made it go a little faster.
CNSP: I fit a single model up here with Mtrf. Train to the stimulus or to the stimulus envelope.
CNSP: and or sorry, rather, I've done that here, and then optimized lambda for that one model.
CNSP: So just this is optimized for the
CNSP: the envelope bottle. But I've used that best lambda
CNSP: all across the analyses. So there's a lot of stub optimality in here. But in in optimal environment, when we test this, these 2 numbers are almost identical. There's a little bit of rounding error, and maybe some other minor differences. But, generally speaking, these 2 numbers are pretty much the same.
CNSP: So there's 2 different methods for accounting for unique portions of the Eeg depending on what you're doing. This might be more easy if you need some
CNSP: spectrogram for Eeg to do some other Trf analysis. But if you really just want what is the unique contribution of us? This is probably faster. cool.
CNSP: And so that's like the end of what I have prepared.
CNSP: And so I'll open it up to questions that go beyond this or questions that are
CNSP: related to this and what I covered.
CNSP: So everyone gets run all of that perfectly without the preprocessed data. Great
Mick Crosse: Aaron, I have a quick question. The multi versus single agonals as you were shown there, I think you have still have it open. Figure 10 where you were kind of doing a sanity check to see
Mick Crosse: what the profile looks like when you use multiple lags
Mick Crosse: are, are you like?
Mick Crosse: Kind of taken an arbitrary window and just sliding it along. No, it's the it's the whole.
CNSP: It's all of the lags are fit at once.
CNSP: Let me pull up the code.
Mick Crosse: So how? How does the
Mick Crosse: as in like from
Mick Crosse: minus 200 to 600. Yep.
Mick Crosse: So how? How does the reconstruction accuracy then pertain to a particular lag.
CNSP: change it to a single lag model
Mick Crosse: confused now.
CNSP: So when you fit a single lag model, did I save this? I don't think I did.
CNSP: I wonder how long this is gonna take
CNSP: that'd be fast.
CNSP: So
CNSP: oh, that's not right. I should actually be doing this one.
CNSP: What is that?
CNSP: I don't like this new Matlab? It's throwing up like stuff, and then, when I keep typing it like messes up.
CNSP: hate it. My!
CNSP: See if I can turn that off or go back to an old version.
CNSP: long is it going to take
CNSP: the team at T. Max?
CNSP: Probably good enough. Alright. So oh, I just overwrote that, too. That was dumb as hell
CNSP: model A and model B, and do that instead.
CNSP: Oh, and also, that's just one trial. So maybe that's why it's taken
CNSP: a long time. So model A and model B have, like very similar
CNSP: structures in the structure
CNSP: that was really silly sounding. Really, the only difference is that the single lags. B
CNSP: has one for each channel, and then the multi lagged. B has one for each time point.
CNSP: and also the type is different.
CNSP: So actually did I?
CNSP: Yep.
CNSP: So I just. I just made it compatible. using the same like. B variable. And yeah, it runs.
Mick Crosse: But what I'm wondering is the the output. You have the reconstruction values.
Mick Crosse: How can you have more than one if you do a multilag analysis?
CNSP: Well, I just fit a multi-lag model. and then I'm predicting the single lag model.
Mick Crosse: Right? Right? Okay. So when you, when you do a multi lag fit like this.
CNSP: it goes this time point, what does that fit it, predict it this time point, what does that fit it predict at this time point? What does that fit it? Predict it? As these aren't the the weights obviously just on that for everyone else. On these are the output, the reconstruction, accuracy.
CNSP: what this does is. It fits all of these models together right? But when you do Mtrf. Train with like type single.
CNSP: it'll fit this, and it'll fit that, and it'll fit that. It'll fit all of these right, and then put them all together in the model structure, and then output that.
CNSP: And so that output model structure is like very nearly identical to
CNSP: the multi lag model structure.
Mick Crosse: I'm not sure if the sorry go ahead.
Mick Crosse: I'm just gonna say it probably makes more sense to do it the way you're doing it with a multi fit than a single fit
Mick Crosse: like in general. Maybe it's it's a lot longer, though the the single fifth off
Mick Crosse: was always had the issue of
not being able to account for autocorrelation
Mick Crosse: and the and the and the signal.
Mick Crosse: And so this kind of gets around that. That's why you you always had that smeared profile in the single fit, so maybe maybe should go
Mick Crosse: update the the toolbox.
CNSP: I mean, I think that like each one has their like utility. Right? Like this one is super fast. And there's not that much, and it's something that you know we've been looking at a lot of like multi sensory data, and some of the stuff that far he's been like doing. And we're seeing a lot of stuff that occurs before 0. And
CNSP: that one paper that's in Neuro image now, where she took your decoder models and and and used them and Ashley's Eve dropping paper. There's a lot of stuff that we chopped up the temporal smearing, and we put that in the discussion. I think this like, maybe it's temporal smearing. Maybe it's something else. We don't really know what it is. But then we we kept getting some new, like multi sensory data. And it seems like there's actually some stuff that happens before 0,
CNSP: like
CNSP: long time before 0. And at least in the multi sensory case.
CNSP: something that's occurring like out around here. it's probably the visual stimulus.
CNSP: because, like, you know, Chandra Sakar and the visual stuffs like starting to move. You know a couple of 100 ms before the you hear it. So it's maybe that. But we're also seeing some stuff that's like 300 ms before onset time.
CNSP: And like that, seems like it's not
CNSP: like within the realm of the temporal smearing.
CNSP: So, yeah, I mean it. It does help.
CNSP: How much does it help? I mean, you still have like these 2 components that are from each of the
CNSP: the 2 peaks, and in the Gfp.
CNSP: You still have this something laid around 400 ms, that's, you know, mirrors like what's going on at 4 and a milliseconds in the Trf.
CNSP: I don't really see that this is actually helping a lot in in this particular situation, and I think that's because there's a lot of good data in in this. There's like, 20 min of actually no, it's more like 60 min of data.
CNSP: Anyway. Yeah, I'll stop belaboring the point. I do like my way, obviously.
CNSP: But I like the fastness of your way.
CNSP: Anything else.
CNSP: Just just yell out.
CNSP: How about like.
CNSP: if someone has like ideas, if they're wanting to do with the Trf
CNSP: and you're you're trying to use your use the trf, to analyze your data. And you have a, you know, some issues, maybe tell us what you're doing with your data. And
CNSP: you know any questions that you might have about that. It doesn't have to be about the tutorial.
CNSP: And this is a like tutorial about the Trf toolbox. But
CNSP: I mean, mostly it's just we're here to help you.
CNSP: And so, if, like, the tutorial was enough great. But you know, while we're here
CNSP: might as well grill us.
CNSP: I'm gonna stop sharing
CNSP: so that we're all in the call.
CNSP: I'll ask a question. Please feel free to answer here, too. It's not my show. Sorry. Go ahead.
Camila Zugarramurdi: Yeah. So I have a data on on speech processing in children.
Camila Zugarramurdi: And it's longitudinal data. So I really want to use the data because it took a lot of effort to take. But I had technical issues. So I had a lag
Camila Zugarramurdi: between the stimulus and the response which I can. I don't know. Well, it's variable, and I don't know how long it is. So I was wondering whether
Camila Zugarramurdi: given that you're like shifting the steamer and the response, could you just shift it until you find the best that possible, and then just use that. Overcome that technical
Camila Zugarramurdi: deficit.
CNSP: Yeah, it it can. I think
CNSP: got to be a little careful there, I guess.
CNSP: But I mean you can. The the Trf
CNSP: doesn't really care about that lag. Actually, it's just gonna find whatever lag is corresponds to like some relationship between. Oh, this happened in the Eeg. And this happened in the in the stimulus, and that is like really consistent across trials.
CNSP: So that's really all it cares about. The only issue would be from my point of view is. if there is a variable lag across trials.
CNSP: it might take some clever cleverness, and also maybe some like human, like hand holding of the Trf
CNSP: function. So like.
CNSP: for example, you don't want to just do an Mtr. F. Cross valve. If it's
CNSP: if the the time lag is variable across trials, because what will happen is you'll get a trf of the component here. It's your F of the component. There will average them together. We'll get a smaller T rf, that's like kinda has 2 humps in it. And we're gonna try to predict some other data
CNSP: where that hump should be very believed somewhere else.
CNSP: What I would imagine doing is running a bunch of Mtrf. Trains on some different trials.
CNSP: maybe doing a cross-validation between.
CNSP: Oh, sorry! A cross correlation between the Ntrf. Weights between each trials, and you might have like a one standard trial. And then you to that. You say this trial. What's the maximum cross correlation.
CNSP: this trial? What's the maximum cross correlation? And you can use the lags from the cross correlation to maybe align the Trfs.
CNSP: And then, once you have them aligned, you might be able to use that
CNSP: alignment to then go back and do an Mtr. F. Cross vow after you adjust your
CNSP: your eeg like stimulus like alignment?
Camila Zugarramurdi: Does that make sense? Make sense sort of. I would have to try it. But yes, but like if if you, if I were to take stimulus in Eugene.
CNSP: and they're time aligned perfectly. and I am tired of trainer
CNSP: and then I take same stimulus, same, eg. And I like add a lag. Mtrf. Train it, and I go back and add another lag. Mtrf. Train it, and II record these lags. I can
CNSP: apply that lag to the Trf. And then they will. Should
CNSP: Mick Joe, Steph, someone corrects me. If I'm wrong, they should align perfectly
CNSP: given. But you have the same regularization parameter. It's the same trial all the same signals that should align.
CNSP: Yeah, perfectly.
CNSP: That's a that's a really cool. I'm I'm really sorry about that like situation that that's like an ag person's worst nightmare. but it's a pretty cool situation. I like from my perspective, because I don't have to deal with it
CNSP: in a in a way that, like the Trfs might be able to help you, and it's just like, I don't know. I think, that
CNSP: it'll be fine.
CNSP: I think that you can like fix that with something. And I don't even think it's gonna need to be a Trf like the Trf. Here is just a stand in for something that can that is invariant to
CNSP: thailack
CNSP: you could use other things. So we've got a guy on the lab gendo who is
CNSP: looking at semantic responses. So the response, and of course I'm just waffling now. So like, if anyone has an actual, real question, please ask it. If you just like to listen to me talk great. So Jen has.
CNSP: He's looking at semantic responses which we usually index by the N. 400. So what happens in 400 if you're not familiar with it. If I tell you that Dennis told me to brush my, you have a prediction, and then I say free.
CNSP: and then prediction is violated, and that violation of of the prediction is evident in a larger negative component. 400 ms after I said tree
CNSP: so we studied the N. 400. But we always always had a question of whether
CNSP: the N. 400 was optimally fit by a stick model at the beginning of the trial, or at the beginning of the word, because you're listening to a word. If it's short word, you're going to get the meaning of it almost immediately.
CNSP: If it's a long word, you might have to sit there and listen to a few phoneings. And it's not enough time to like really affect behavior and like conversation. Maybe it is, if you're, you know, have a hard time if you're speaking a site, whatever but
CNSP: those
CNSP: you know 100 ms that it takes you to understand. The word might not be a big deal behaviorally, but normally, that's a long time. So Jen has developed a a neural network that can
CNSP: fit semantic trfs by varying
CNSP: the The timing of that's the impulse that corresponds to each word.
CNSP: and he gets better predictions. They're marginally better, because you know the semantic like predictive prediction. Accuracies aren't that high, anyway? But we do get significant
CNSP: cross validated improvements
CNSP: by like varying the the stick. All of that, just to say that you might be able to talk to someone who can fit a Trf using a neural network
CNSP: where time is a parameter.
dannapinto1: I have a few questions.
CNSP: Oh, okay.
dannapinto1: 2 different questions.
dannapinto1: The first one in our projects, we
dannapinto1: well, our previous projects. We usually have 2 stimuli that are playing concurrently.
dannapinto1: and we usually do different analysis to see like in attend selective attention divided attention paradigms like that. And we've always debated if to enter the 2 stimuli envelopes into a multivariate model, or if to, if we want to compare the responses between them, if to put them into the same model, or if to put them in 2 different models.
dannapinto1: and we were wondering how
dannapinto1: because of the issue of cross-validation, if
dannapinto1: if that the if we're, for example, comparing attended versus unattended, if if we put them in the multivariate if that will affect, if if one's responses are lower, if that will affect
dannapinto1: the total performance of the model.
dannapinto1: if the difference between them is big.
CNSP: I don't think so.
CNSP: I ran an experiment where I
CNSP: had noise.
CNSP: and then I randomly changed its envelope.
CNSP: And then there is a like a visual flash on the screen, and it's brightness was changed by a different envelope, and I can tear f them separately and together, and when I tear f them together, the the performance improves because you're accounting for more data.
CNSP: right? There are some responses in the brain
CNSP: that are responding to your unattended stimulus and some that are responding to your attended stimulus
CNSP: and if you input both into and in into a multivariate Drf, you should be able to do better.
CNSP: Their weights might be affected by the multivariate
CNSP: approach.
CNSP: But I would guess probably not much, since they're probably not correlated.
CNSP: Probably the biggest interaction there is actually what's going on in the brain
CNSP: rather than what's going on in the modeling, modeling procedure you've got like neurons that are that are unable to respond to one because they're like you're attended to. The other one.
CNSP: am I getting close to like your question? I'm not sure if I quite understand your question, you're worried about whether the modeling procedure will, or whether entering them together, will affect like, make a worse model.
dannapinto1: If, yeah, if, if, for example, in our experiment, one of the
dannapinto1: Stimuli was a very.
dannapinto1: It was a very rhythmic stimuli, and we found that when we put them in 2 separate models.
dannapinto1: the model was
dannapinto1: like there was no speech tracking at all for the rhythmic stimuli. And we we wanted. That's a whole different question that we wanted to look into. But then we were wondering if, when and then, when we put them together, when we looked at the, at the model the multivariate model, it seemed to look worse
dannapinto1: then, when we would just look at the
dannapinto1: con like the continuous speech or the other stimuli by itself, and we were wondering if there was something, maybe, that we should done, maybe
dannapinto1: to control or like to to do something in between, so that we could
dannapinto1: enter them into the same model, instead of looking at them in 2 separate models.
CNSP: So let me just like summarize what I think that you did give speech A and B
CNSP: should have modeled to a you got a TRF. For a. not the reconstruction or not, the prediction accuracies, but the Trf. Itself.
dannapinto1: Get a model to be. Get it to your F for B,
CNSP: you fit a tear off for A and B together. and then you're gonna take the A B model, separate it into model a model. B,
CNSP: right? And you're looking at model A from the joint model and model a fits similarly side by side.
dannapinto1: and the same thing for B.
CNSP: Both of those look worse.
dannapinto1: I mean the B. The B was bad either way, but
dannapinto1: rhythmic speech, the continuous speech, no, A is the regular continuous speech, and B. Is like rhythmic like onset of words, or come at a very specific rate.
CNSP: Feel like something is going on here. That's not well, maybe not. I don't know. Are you using the actual envelope for the rhythmic speech?
dannapinto1: Both of them. Yeah.
CNSP: are the what is being spoken? Words?
dannapinto1: Yeah. Yeah. I mean words in like, in phrases.
dannapinto1: I think it's just the the onset of the words is just regular regular
dannapinto1: regularized.
CNSP: Well, what I was gonna say, which I don't. I'm less concerned about now is that the Trf. Does not like rhythmic things.
CNSP: and, as Ed would say, it would shit itself
dannapinto1: But that that applies more to things like sinusoids and things that are like regularly rhythmic.
CNSP: And I don't think that's a Trf issue. I think that's just a linear time, invariant system issue, which I say those words, I don't know really anything about them.
CNSP: So if you want to hear more of those words, ask Nick. Probably. Giovanni, I don't know.
CNSP: But yeah, I'm not sure that's actually the problem.
dannapinto1: Hmm.
dannapinto1: and then I have another question sort of about let me follow up on that, though. Email us that and think about that a little bit more.
dannapinto1: Okay, okay.
CNSP: But go ahead with your other question.
dannapinto1: So another another thing that we're doing now is also, we're looking at this like once set of one stimuli, the envelope. But we're also adding the other other parameters. So other features, such as word on set phony on set things like that. And we were just wondering you might have spoke about it today. But we were trying. I was trying to run the code, so I wasn't always with you. But how do? How like? How would you
dannapinto1: specify exactly if I want to understand. If adding, these features improved my model. what what baseline, or what criteria should I set so that I can understand
dannapinto1: whether each criteria where they're adding each of these features, actually.
dannapinto1: made my model substantially better like, how do I know between these r values. If
dannapinto1: if this is better than just looking at the envelope.
CNSP: we usually just do like a paired samples test
CNSP: between the 2 like,
dannapinto1: would you separate models? Would you do like separate models for just the envelope, and then
dannapinto1: like with each feature? And then compare.
CNSP: yeah. So here's the thing. If you put features together in one model, you're only gonna get one output.
dannapinto1: So each m 2, I'm gonna
CNSP: do this again.
CNSP: Right? Do we have, Matlab?
dannapinto1: Yes, we see it.
CNSP: Yeah.
CNSP: So here's kind of hold on just a second.
CNSP: I'm gonna actually show you show some data. And
CNSP: you see a paper.
CNSP: So this is from your man. Giovanni just left the call
CNSP: had to get home.
CNSP: So this is what he did, that envelopes spectrogram.
CNSP: bonemes, phonetic features and
CNSP: joint model with phonetic feature and spectrogram. He pretty much can phonemes
CNSP: from the outset, because Finneg features a smaller model, it does better
CNSP: or and it does better generally.
CNSP: so this is what he does. So this is an Mtr. F. Cross valve that's an Mtr. F. Cross valve that each one of these is a different Mtr. F. Cross valve for different people. So, however, many people he has in here times 5 is the number of models or cross validations he did for this.
CNSP: And that's what this is right here.
CNSP: So oh, one thing I actually didn't mention is that like, I'm only gonna do an Mtr. F cross vow. For most of this I should have mentioned this at the very beginning.
CNSP: that's my cab. By the way.
CNSP: this is really the way to go, unless you're pretty sure that you're not overfitting is to have like a separate train and test set. So even though like it would be one of
CNSP: these for each
CNSP: of those 5 conditions. But here I'm just
CNSP: doing that for each condition. So we get a value here, a value there, value there and a value there.
CNSP: Or actually, we get 128 values. You can do different things. But I'm just taking the average across the entire scalp
CNSP: but
dannapinto1: and you wouldn't do combinations.
dannapinto1: you would you have to do both
CNSP: forgot. I took the
CNSP: damn!
CNSP: That's my own function, and I cleared my own variable space. I'm not able to show you the
CNSP: Topo plots, but they do have like topographies. Actually. No, I do have
CNSP: this
CNSP: to show you. So they they do have topographies. and so like.
CNSP: This is like, not very good. This is, or not very optimal, so you can like choose whichever like channels you want.
CNSP: like something like that.
CNSP: if you have to find some channels and that will improve it. But you have to do this. One is an envelope, one feature spectrogram, one feature phonemes, one feature or phonetic features, one feature.
CNSP: This is a phonetic feature of spectrogram. and it was made with 2 features. Got it.
CNSP: I will actually just show you this just to illustrate what it looks like.
CNSP: So
CNSP: this is time 11,365 by one.
CNSP: That's the envelope model. the phoneme, or phonetic feature model. I keep calling them Phonemes.
CNSP: 1,000 or 11,365 time points by 19 features. I call this one feature model. It has 19 parameters in it, 19 features. So it it gets entered in as 19 coefficients. But we treat this as like one set of features
CNSP: the same thing. With the this.
CNSP: the spectrogram, 16 channels of 16 frequency channels
CNSP: and the Fs is now 35.
dannapinto1: How does it go
dannapinto1: like just from just this is like, maybe a like a right like a code question. But how does it know in this case to treat all of what's cause? This looks exactly like how I would enter the like when I enter my s
dannapinto1: my stimuli, and I want to enter the different features. I enter them like I enter time times.
dannapinto1: which is like the 4 features. So how does it know in this case that doesn't between them as one.
CNSP: It will always treat each of these
CNSP: as separate, but it will add them together at the end.
CNSP: Ish.
CNSP: So at the end, we're going to add 16 features together for the spectrogram model. At the end. We're gonna add 19 features together to get the ultimate
CNSP: prediction in the feature model, the phonetic feature model.
CNSP: And then we're gonna add both of those features together at the end. So you're gonna have, like.
CNSP: essentially. And I'm I'm kind of like bending the truth a little bit. Here you're gonna have, like 16 spectrogram predictions and 19 phonetic feature predictions.
CNSP: And so the spectrum predictions are only going to be those 16. The Fedex features are only going to be those 19, and the combined joint model is going to be all 35. So that's how you get the yep.
CNSP: but it it's not. It's not smart, actually, and in terms of like treating columns differently. And it doesn't really need to be
CNSP: we do have. we have worked on something called Banded Ridge regression.
CNSP: And that's where we do treat
CNSP: bands of features separately, as we don't have this yet, and I've I've got it written, and it's I'm just not happy with it. So we aren't releasing it yet.
CNSP: But
CNSP: what it does is it treats bands of like features like, say, these 16 spectrograms, bands, or 19 phonetic features, as one set and regularizes them altogether the same.
CNSP: but potentially different from the other features.
CNSP: And so like. And that's that's useful because
CNSP: the amount of regularization depends on a lot of things so like things that are not like pertinent here, like the the data quality, how much data you have and the and stuff like that. But pertinent to this. Another thing that matters is like the type of feature. So like, if you've got a sparse model where you've just got a bunch of zeros. And then somewhere in there, there's like once, and most of your data are zeros.
CNSP: or if you have a feature that is
CNSP: substantially different temporal dynamics.
CNSP: So, for example, one of our old grad students, Adam Bedner. did an experiment where you had someone
CNSP: listening to speech that moves in space. So someone's talking. And it's just sitting here moving like like this across space. And their goal is to track that in space. Right?
CNSP: Well, it moves way slower than it's speaking. So I'm like speaking at normal speed. And I'm moving my hand about how fast it would move in space. So this like feature, like compared to like the envelopes is really slow.
CNSP: And so if I were gonna fit, a a model with that envelope of the motion and an envelope of the speech energy I would might want to like regularize them differently because of different speak frequency characteristics.
CNSP: So that's one way that we can use the trf, to like
CNSP: treat things differently. But that doesn't really do anything
CNSP: other than
CNSP: change the regularization. But that is that provides the optimal lambda. If you fit a spectrogram model, and if you fit a phonetic feature model, you'll see most of the time. The phonetic feature model requires a higher lambda because of probably because of like this sparseness.
CNSP: but otherwise you don't really need things to be treated differently, because it's one model, one prediction.
CNSP: How well can it do?
dannapinto1: Thank you so much. You're welcome.
CNSP: And 17 troopers here just trying to like, hear all my waffle.
Maya מיה مايا: Okay, thanks everyone very much. 10 pm.
Maya מיה مايا: yeah. So like, that's all I have, and I'm only here like you need extra help.
Maya מיה مايا: Thanks.
CNSP: Yeah.
CNSP: I'll give someone. I'll give someone like another couple of minutes to speak up if you're shy, but after that, like we'll we'll call it a day.