WEBVTT

00:19.829 --> 00:21.940
This week we are pleased to host Niko

00:21.940 --> 00:25.190
Krigaskte to discuss task performing

00:25.190 --> 00:27.590
models , comparing predictions of

00:27.590 --> 00:29.368
representational geometries and

00:29.368 --> 00:31.920
topologies . Doctor Krigaskor is a

00:31.920 --> 00:33.799
computational neuroscientist who

00:33.799 --> 00:36.549
studies how our brains enable us to see

00:36.549 --> 00:38.959
and understand the world around us . He

00:38.959 --> 00:40.792
is a professor of psychology and

00:40.792 --> 00:42.903
neuroscience at Columbia University ,

00:42.903 --> 00:45.070
affiliated member of the Department of

00:45.070 --> 00:47.237
Electrical Engineering , and principal

00:47.237 --> 00:49.919
investigator and director of cognitive

00:49.919 --> 00:52.141
engineering at the Zuckerman Mind Brain

00:52.141 --> 00:54.720
Behavior Institute at Columbia . His

00:54.720 --> 00:57.400
talk seeks an understanding of the

00:57.400 --> 00:59.520
brain and computational mechanism .

01:00.110 --> 01:02.630
Underlying cognitive functions focusing

01:02.630 --> 01:04.852
today on two techniques in particular I

01:04.852 --> 01:06.741
think to analyze representational

01:06.741 --> 01:08.300
geometry and topological

01:08.300 --> 01:10.500
representational similarity analysis ,

01:10.790 --> 01:13.709
and he argues that this requires that

01:13.709 --> 01:16.349
we implement our theories in task

01:16.349 --> 01:18.669
performing models and adjudicate among

01:18.669 --> 01:20.790
these models on the basis of their

01:20.790 --> 01:23.069
predictions of brain representations

01:23.069 --> 01:25.402
and behavioral responses . So thank you ,

01:25.510 --> 01:27.566
Professor , for joining us today . I

01:27.566 --> 01:29.566
see your slides , they look great .

01:30.500 --> 01:32.949
Thank you , Kevin . Great to be with

01:32.949 --> 01:36.709
you all . Um , my , my lab studies

01:36.709 --> 01:39.709
vision in humans and primates , and we

01:39.709 --> 01:42.629
also do methods development and for

01:42.629 --> 01:46.190
today's talk , uh , I'm gonna focus on

01:46.190 --> 01:48.440
methods development . I think , uh ,

01:48.449 --> 01:51.779
it's of interest , uh , to some of you ,

01:52.120 --> 01:54.849
um , to analyze brain activity patterns

01:54.849 --> 01:57.389
and to use brain activity measurements

01:57.389 --> 01:59.167
to adjudicate between different

01:59.167 --> 02:02.269
theories . Um , often now we can

02:02.269 --> 02:04.325
implement theories and computational

02:04.325 --> 02:06.269
models that predict detailed

02:06.269 --> 02:08.910
representations of stimuli , and we can

02:08.910 --> 02:11.419
study the representational geometries

02:11.789 --> 02:13.845
and compare models in terms of their

02:13.845 --> 02:15.622
predictions of representational

02:15.622 --> 02:17.900
geometries , and that's what I'm gonna ,

02:18.050 --> 02:21.070
uh , focus on in , in the talk today .

02:21.619 --> 02:23.830
And it would be great to have some

02:23.830 --> 02:26.429
interaction along the way , just

02:26.429 --> 02:29.139
interrupt me when something is unclear ,

02:29.190 --> 02:31.350
and we can talk about it , and I hope

02:31.350 --> 02:33.789
also at the end there'll be um some

02:33.789 --> 02:37.100
time to to to discuss a little bit

02:37.100 --> 02:40.919
further . So my uh title today

02:40.919 --> 02:42.800
is comparing models by their

02:42.800 --> 02:44.830
predictions of Representational

02:44.830 --> 02:48.470
geometries and topologies . And ideally

02:48.470 --> 02:50.479
these models could be brain

02:50.479 --> 02:52.701
computational models , for example , in

02:52.701 --> 02:54.590
vision these could be deep neural

02:54.590 --> 02:57.600
network models that take um visual

02:57.600 --> 03:00.800
stimuli as inputs and then process them

03:00.800 --> 03:04.279
through stages of , of representation .

03:04.759 --> 03:07.360
Uh , across layers and if the recurrent

03:07.360 --> 03:11.309
models , uh , possibly also across time ,

03:11.919 --> 03:15.039
and then the goal is to use the

03:15.039 --> 03:16.928
internal representations in those

03:16.928 --> 03:20.300
models to To adjudicate between the

03:20.300 --> 03:22.699
models , to evaluate them based on

03:22.699 --> 03:24.339
their , their predictions .

03:27.869 --> 03:30.789
So what I'm gonna talk about is

03:31.520 --> 03:35.119
about is implemented in um a Python

03:35.119 --> 03:37.919
toolbox , which we call the RSA 3

03:37.919 --> 03:39.720
toolbox . RSA stands for

03:39.720 --> 03:42.130
Representational similarity analysis

03:42.130 --> 03:45.940
and 3 is the third . iteration

03:45.940 --> 03:49.820
of this uh set of tools started with

03:49.820 --> 03:53.649
RA1 in in 2008 when I was a postdoc

03:54.020 --> 03:57.669
and we had a major um set of new

03:57.669 --> 04:01.369
tools in 2014 and this is the third

04:01.369 --> 04:04.919
um iteration of this . Which we've just

04:05.039 --> 04:07.809
um finished developing so we have the

04:07.809 --> 04:11.130
toolbox online uh it's on GitHub and

04:11.130 --> 04:14.320
it's , it's ready for for application .

04:14.570 --> 04:16.799
We haven't officially released it yet .

04:16.850 --> 04:19.079
There's gonna be a paper on the toolbox

04:19.079 --> 04:22.369
which we're still working on , um , so

04:22.369 --> 04:26.320
this is uh . Quite new , um , but some

04:26.320 --> 04:28.760
of the methodological advances that I'm

04:28.760 --> 04:30.927
gonna talk about that I implemented in

04:30.927 --> 04:34.000
this Python toolbox have been under

04:34.000 --> 04:36.160
development for the past decade or so

04:36.160 --> 04:39.359
and that's all slowly come together and

04:39.359 --> 04:41.690
I'm very excited that now , you know ,

04:41.760 --> 04:44.600
all these things are , are working

04:44.600 --> 04:47.730
together properly and so excited to

04:47.730 --> 04:49.799
tell you a little bit about it . The

04:49.799 --> 04:53.000
lead developer of the toolbox is Aikoch ,

04:53.079 --> 04:56.920
who was a postdoc in our lab and is now

04:56.920 --> 04:58.753
a professor at the University of

04:58.753 --> 05:02.440
Luxembourg . And so he is uh a

05:02.440 --> 05:04.273
methodologist , he's a cognitive

05:04.273 --> 05:05.829
scientist , but he's also a

05:05.829 --> 05:08.390
statistician , and he's super brilliant

05:08.390 --> 05:11.119
and uh he's the lead uh methodologist

05:11.119 --> 05:14.149
and programmer on the toolbox . However ,

05:14.230 --> 05:16.397
there are many other people involved .

05:16.397 --> 05:18.630
Another central person is Jasper van

05:18.630 --> 05:22.299
den Bosch , who , uh , was a postdoc in ,

05:22.390 --> 05:24.390
in our lab a long time ago . We're

05:24.390 --> 05:28.049
still in Cambridge , UK . And uh

05:28.049 --> 05:31.359
he's a major contributor to the , the ,

05:31.410 --> 05:33.690
the software development , making sure

05:33.690 --> 05:36.970
we follow uh professional software

05:36.970 --> 05:39.019
development practices . And then

05:39.019 --> 05:41.075
there's many other people involved .

05:41.075 --> 05:43.186
Another very important contributor is

05:43.186 --> 05:45.899
Jon Deison , a longtime collaborator of

05:45.899 --> 05:48.739
ours , uh , and a motor computational

05:48.739 --> 05:51.660
neuroscientist at Western University in

05:51.660 --> 05:54.299
London , Ontario in Canada . He's also

05:54.299 --> 05:57.049
a statistician and has , uh ,

05:57.059 --> 06:00.339
contributed , um , very substantially

06:00.339 --> 06:02.506
to some of the methodological advances

06:02.506 --> 06:04.672
that are implemented in this toolbox .

06:05.010 --> 06:07.829
And then there are a number of other uh

06:07.829 --> 06:10.940
people who at some time were postdocs

06:10.940 --> 06:13.051
in the lab . Maria Moore , who's also

06:13.051 --> 06:16.070
at Western in London , Ontario , Jan

06:16.070 --> 06:18.950
Share , who's now a professor at the

06:18.950 --> 06:21.059
University of Montreal , Katherine

06:21.059 --> 06:23.309
Stors , who's in New Zealand , Benjamin

06:23.309 --> 06:25.470
Peters , who's just starting his own

06:25.470 --> 06:27.670
lab at the University of Edinburgh ,

06:28.190 --> 06:30.246
and Tal Golan , who's a professor at

06:30.246 --> 06:33.579
Ben Goon University in Israel . Um ,

06:33.619 --> 06:36.299
they all made major contributions to

06:36.299 --> 06:38.730
this , and then there's another , um ,

06:39.179 --> 06:42.239
uh , groups of , of collaborators over

06:42.239 --> 06:44.619
the years , all of whom have made

06:44.619 --> 06:46.980
contributions to the toolbox . So this

06:46.980 --> 06:50.089
is really , uh , a community effort .

06:50.339 --> 06:53.140
Um , a lot of these people are kind of

06:53.140 --> 06:55.779
collaborators of mine at least at some

06:55.779 --> 06:57.980
point , but there's also a lot of new

06:57.980 --> 06:59.980
people coming in and there are some

06:59.980 --> 07:02.313
here on this slide that have never been .

07:02.364 --> 07:05.154
In our lab and we , we hope to develop

07:05.154 --> 07:07.355
this further as an open source

07:07.355 --> 07:10.195
development so maybe some of you want

07:10.195 --> 07:13.315
to join in that um it is on GitHub , it

07:13.315 --> 07:17.035
is open source you can um look at the

07:17.035 --> 07:19.825
discussion forum , you can contribute

07:19.825 --> 07:22.024
to it . You can issue full requests .

07:22.035 --> 07:23.785
We're , we're totally open to

07:24.154 --> 07:26.859
developing this together . With some of

07:26.859 --> 07:30.600
you . So this is a toolbox for a

07:30.600 --> 07:33.269
particular method for

07:33.519 --> 07:36.709
analyzing brain activity data

07:36.869 --> 07:40.200
acquired with um different methods in

07:40.200 --> 07:42.350
neuroscience in humans and animals ,

07:42.399 --> 07:44.399
including neural recordings and

07:44.399 --> 07:46.959
functional imaging techniques such as

07:46.959 --> 07:49.181
functional magnetic resonance imaging .

07:49.559 --> 07:52.950
So , uh , here's my one

07:52.950 --> 07:56.750
slide summary of how RSA

07:56.750 --> 07:58.806
representational similarity analysis

07:58.806 --> 08:02.720
works . Um , in our experiments , we

08:03.559 --> 08:05.226
design different experimental

08:05.226 --> 08:07.480
conditions . So with us , we're , we're

08:07.480 --> 08:09.036
studying visions , so those

08:09.036 --> 08:11.147
experimental conditions correspond to

08:11.147 --> 08:13.690
experimental stimuli . So we might show

08:13.880 --> 08:16.239
a bunch of different stimuli you you

08:16.239 --> 08:18.750
see a hand and an umbrella . And we

08:18.750 --> 08:22.109
show those exact same images to the

08:22.109 --> 08:24.609
brains of our subjects who could be

08:24.609 --> 08:26.910
monkeys in a functional magnetic

08:26.910 --> 08:30.579
resonance imaging scanner or um .

08:32.989 --> 08:36.379
Monkeys with electrodes in their brains

08:36.379 --> 08:38.268
whose brain activity recording in

08:38.268 --> 08:41.848
different cortical areas . We also

08:41.848 --> 08:44.039
present uh the same stimuli to our

08:44.039 --> 08:46.261
models , and these could be deep neural

08:46.261 --> 08:47.650
network models that have

08:47.650 --> 08:50.369
representations across layers , um .

08:51.400 --> 08:53.622
And in each of their layers , they have

08:53.622 --> 08:57.140
internal uh activity patterns . And so

08:57.140 --> 08:59.409
we get in the brains of our subjects

08:59.409 --> 09:01.989
and and the layers of our models is

09:01.989 --> 09:03.950
activity patterns , one activity

09:03.950 --> 09:06.830
pattern for each of our stimuli , and

09:06.830 --> 09:08.830
we want to interpret those activity

09:08.830 --> 09:10.789
patterns as representations of the

09:10.789 --> 09:13.099
stimuli , right ? So the goal is to

09:13.429 --> 09:16.190
evaluate whether the set of activity

09:16.190 --> 09:19.630
patterns elicited by the images in a

09:19.630 --> 09:23.030
cortical area like area B4 in a human

09:23.030 --> 09:26.159
subject . Um , whether they are in some

09:26.159 --> 09:29.780
sense similar or related to the

09:30.039 --> 09:32.919
activity patterns elicited in let's say

09:32.919 --> 09:35.440
layer 4 of the deep neural network

09:35.440 --> 09:38.280
model , and this comparison is

09:38.280 --> 09:41.830
non-trivial because we don't know ,

09:42.299 --> 09:44.750
uh beforehand the correspondency

09:44.750 --> 09:46.880
between the units of the deep neural

09:46.880 --> 09:49.880
network model and the neurons that

09:49.880 --> 09:52.280
we've recorded from in a monkey or the

09:52.280 --> 09:55.750
voxels that we've . Um , we've measured

09:55.750 --> 09:59.140
activity for using fMRI in a in a human

09:59.140 --> 10:02.619
subject . So a direct comparison at the

10:02.619 --> 10:04.940
level of the response patterns would

10:05.460 --> 10:08.969
require . Fitting some kind of model

10:08.969 --> 10:10.890
that defines this correspondency

10:10.890 --> 10:13.057
mapping between the units of the model

10:13.057 --> 10:15.679
and the responses that we've actually

10:15.679 --> 10:18.169
measured , and 11 way to do that is

10:18.169 --> 10:21.119
using linear encoding models where we

10:21.770 --> 10:24.169
try to predict each voxel in the brain

10:24.169 --> 10:26.280
or each measured neuron in the monkey

10:26.280 --> 10:28.336
brain as a linear combination of the

10:28.336 --> 10:30.369
units of the neural network model .

10:31.929 --> 10:34.250
However , in RSA we take a different

10:34.250 --> 10:36.760
approach . We , uh , abstract from

10:36.760 --> 10:39.239
these , these actual response patterns

10:39.609 --> 10:42.530
by viewing these response patterns as

10:42.530 --> 10:44.809
points in the multivariate response

10:44.809 --> 10:48.010
space where there's one dimension for

10:48.010 --> 10:50.890
each neuron or each measured response

10:50.890 --> 10:53.679
channel , for example , a voxel . And

10:53.679 --> 10:56.559
then a response pattern is a point in

10:56.559 --> 10:58.989
that space where the coordinates define

10:59.440 --> 11:01.440
the activities across the different

11:01.440 --> 11:03.719
neurons . So every stimulus corresponds

11:03.719 --> 11:06.479
to a point in that space . And so when

11:06.479 --> 11:08.919
we have 3 stimuli , we have 3 points

11:08.919 --> 11:11.359
here , and what we care about are the

11:11.359 --> 11:14.239
distances between those points and the

11:14.239 --> 11:17.479
distance matrix . Defines what we call

11:17.479 --> 11:21.419
the representational geometry and that

11:21.440 --> 11:23.559
reflects to what extent the brain

11:23.559 --> 11:25.799
region we're looking at cares about the

11:25.799 --> 11:27.950
distinctions between the stimuli ,

11:28.320 --> 11:30.209
right ? So if we have a big if we

11:30.209 --> 11:32.320
assume that there is some noise on

11:32.320 --> 11:34.431
these patterns , and let's assume for

11:34.431 --> 11:36.729
the moment that the noise is isotropic

11:37.119 --> 11:39.989
and the further away two patterns are ,

11:40.200 --> 11:42.311
the more distinct they are , the more

11:42.311 --> 11:45.200
decodable they are . And so the

11:45.200 --> 11:48.320
representation of geometry in this uh

11:48.320 --> 11:50.559
context of isotropic noise as we

11:50.559 --> 11:53.900
measure the Euclidean distances , we uh

11:54.280 --> 11:56.770
have essential information about the

11:58.159 --> 12:00.326
The information about the stimuli that

12:00.326 --> 12:04.090
the neural population encodes . And

12:04.090 --> 12:07.840
so this motivates characterizing

12:07.840 --> 12:10.530
a brain representations or cortical

12:10.530 --> 12:13.450
areas representational geometry in

12:13.450 --> 12:16.530
order to characterize the the code in

12:16.530 --> 12:19.169
that area . And we can do that for a

12:19.169 --> 12:21.640
cortical area and a monkey or a human ,

12:21.969 --> 12:25.409
um , and we can do that for a layer of

12:25.409 --> 12:28.530
the deep neural network or some other

12:28.969 --> 12:30.191
computational model .

12:33.489 --> 12:36.340
So the representation of geometry can

12:36.340 --> 12:39.270
be characterized by the distance matrix .

12:40.719 --> 12:43.869
We use a slightly more general term of

12:43.869 --> 12:46.299
representation dissimilarity matrix

12:46.909 --> 12:50.270
where um I'm gonna show you later how

12:50.270 --> 12:52.419
dissimilarity um

12:53.700 --> 12:57.289
Includes estimators for distances that

12:57.289 --> 12:59.489
are not themselves distances in the

12:59.489 --> 13:02.880
mathematical sense . So for example ,

13:03.039 --> 13:05.869
they can be cross validated estimators

13:06.239 --> 13:09.599
and they can Sometimes return negative

13:09.599 --> 13:11.640
values which enables them to be

13:11.640 --> 13:13.807
unbiased . For example , when the true

13:13.807 --> 13:16.450
distance is zero , an unbiased

13:16.450 --> 13:20.299
estimator has to be able to um return

13:20.299 --> 13:22.559
negative values so that the values of

13:22.559 --> 13:24.726
returns from the true distance is zero

13:24.840 --> 13:28.400
can be symmetrically distributed uh

13:28.400 --> 13:32.400
about the true value of 0 . So

13:32.609 --> 13:34.665
these representational dissimilarity

13:34.665 --> 13:36.831
matrices characterize representational

13:36.831 --> 13:40.010
geometries and they're square matrices

13:40.179 --> 13:42.969
indexed by the stimuli horizontally and

13:42.969 --> 13:46.690
vertically . And since they're indexed

13:46.690 --> 13:48.746
by the stimuli for the brain and the

13:48.746 --> 13:50.968
model , they're very straightforward to

13:50.968 --> 13:53.359
compare without fitting any parameters .

13:53.369 --> 13:56.849
So this obviates the need for fitting

13:56.849 --> 13:59.016
linear encoding models at the level of

13:59.016 --> 14:02.119
the response patterns because by

14:02.369 --> 14:05.250
computing these distance matrices ,

14:05.280 --> 14:07.809
we're abstracting from the particular

14:07.809 --> 14:10.750
units that encode the patterns or the

14:10.750 --> 14:14.080
particular neurons . That define the

14:14.080 --> 14:16.960
multivariate response space and so

14:16.960 --> 14:19.127
while we might have a different number

14:19.127 --> 14:22.309
of units in the model than uh we have

14:22.309 --> 14:24.960
neurons that we've measured for our

14:24.960 --> 14:27.520
brain region of interest and we don't

14:27.520 --> 14:29.880
know the 1 to 1 correspondency by

14:29.880 --> 14:32.400
focusing on the geometry we can make

14:32.400 --> 14:34.919
straightforward comparisons um between

14:34.919 --> 14:38.090
different representations . Can I ask a

14:38.090 --> 14:40.690
quick question ? Sure . Hi , I'm sorry

14:40.690 --> 14:42.746
to interrupt . I wonder if you could

14:42.746 --> 14:44.270
give us a sense of the uh

14:44.270 --> 14:46.909
dimensionality of the space that's

14:46.909 --> 14:49.190
represented in the geometry on the

14:49.190 --> 14:51.150
right there . Uh , is it high

14:51.150 --> 14:53.372
dimensional or low dimensional ? Yeah ,

14:53.489 --> 14:55.890
this is high dimensional . So here , um ,

14:55.900 --> 14:58.122
what I'm drawing is a three dimensional

14:58.122 --> 15:00.011
space because that's where we can

15:00.011 --> 15:02.609
intuit , but um there will be one

15:02.609 --> 15:04.831
dimension for every neuron . So we , we

15:04.831 --> 15:08.770
might , we , we

15:08.770 --> 15:11.809
might have an array in a cortical area

15:11.809 --> 15:14.210
in a monkey's brain and maybe we're

15:14.210 --> 15:17.580
recording from 100 neurons . And then

15:17.580 --> 15:21.179
we'd have a 100 dimensional space or

15:21.179 --> 15:23.012
in functional magnetic resonance

15:23.012 --> 15:25.690
imaging , we might be looking at some

15:26.059 --> 15:28.580
functional area , maybe in visual area

15:28.580 --> 15:32.359
before and um if we use high

15:32.359 --> 15:34.081
resolution functional magnetic

15:34.081 --> 15:35.970
resonance imaging , we might have

15:35.970 --> 15:39.049
hundreds of voxels um that define the ,

15:39.059 --> 15:41.170
the response pattern in that region ,

15:41.170 --> 15:43.281
so then the space would have hundreds

15:43.281 --> 15:47.109
of dimensions . Appreciate that . I

15:47.109 --> 15:48.887
just , uh , I heard you mention

15:48.887 --> 15:51.539
Euclidean , uh , distance earlier , uh ,

15:51.549 --> 15:53.549
which sort of breaks down at higher

15:53.549 --> 15:55.549
dimensions , and I wondered how you

15:55.549 --> 15:57.438
dealt with that . Yeah , um , the

15:57.438 --> 16:00.119
Euclidean disc , so that's , uh , uh ,

16:00.299 --> 16:02.609
there's a very important , um , point

16:02.609 --> 16:05.409
there . The Euclidean distance doesn't

16:05.409 --> 16:07.242
necessarily break down in higher

16:07.242 --> 16:09.242
dimensions . It depends on what the

16:09.729 --> 16:12.007
response patterns actually are , right ?

16:12.039 --> 16:14.261
So for example , if we measure response

16:14.261 --> 16:17.840
patterns , um , from populations of

16:17.929 --> 16:19.651
neurons in the monkey brain to

16:19.651 --> 16:21.880
different Images from different

16:21.880 --> 16:24.429
categories , there's a lot of structure

16:24.429 --> 16:27.239
in those distance matrices , and the

16:27.239 --> 16:30.919
high dimensionality doesn't doesn't

16:30.919 --> 16:34.559
automatically or mean that Euclidean

16:34.559 --> 16:36.630
distances cannot be informative .

16:38.979 --> 16:40.923
Interesting . I have to think more

16:40.923 --> 16:43.257
about that . I appreciate your response .

16:43.469 --> 16:47.359
Yeah . Yeah , this is a very , very

16:47.359 --> 16:49.070
interesting issue indeed .

16:51.969 --> 16:54.729
So , and for the , for the models , um ,

16:55.609 --> 16:58.429
Usually the numbers of units and the

16:58.429 --> 17:02.229
models were not limited by arrays

17:02.229 --> 17:04.430
for recording or by the limited

17:04.430 --> 17:06.349
resolution of functional imaging

17:06.349 --> 17:08.698
techniques . We have all the

17:08.698 --> 17:10.920
information , right , and we might have

17:10.920 --> 17:13.254
thousands or tens of thousands of units .

17:13.254 --> 17:15.031
So then it's really a very high

17:15.031 --> 17:17.178
dimensional space and in deep neural

17:17.178 --> 17:20.499
networks as well , these geometrical

17:20.499 --> 17:22.979
characterizations of representations at

17:22.979 --> 17:26.218
different levels , uh , often contain a

17:26.218 --> 17:28.385
lot of interesting structure , so they

17:28.385 --> 17:31.058
make quite , quite detailed predictions

17:31.338 --> 17:34.589
despite , um , The fact that there are

17:34.589 --> 17:37.229
scenarios where Euclidean distances are

17:37.229 --> 17:41.040
not um Um , not that helpful in ,

17:41.140 --> 17:43.251
in high dimensional spaces . It's not

17:43.251 --> 17:46.770
true in general . So ,

17:46.819 --> 17:50.390
um , so this technique involves , um ,

17:52.290 --> 17:54.800
A number of choices that we're going to

17:54.800 --> 17:57.640
discuss one by one . The first is how

17:57.640 --> 17:59.640
to estimate the RDM , the

17:59.640 --> 18:01.862
representational dissimilarity matrix .

18:01.862 --> 18:04.199
What should be a way of getting from

18:04.199 --> 18:06.032
the activity patterns that we've

18:06.032 --> 18:08.829
measured to these dissimilarity

18:08.829 --> 18:10.989
estimates . So that's the question of

18:11.510 --> 18:15.130
the RDM estimator that we choose . And

18:15.130 --> 18:18.209
a second decision is the RDM comparator

18:18.209 --> 18:20.410
when we want to compare the model

18:20.410 --> 18:21.910
predicted representational

18:21.910 --> 18:24.439
dissimilarity matrix to the measured

18:24.439 --> 18:26.661
representational dissimilarity matrix .

18:26.810 --> 18:28.810
How do we compare these two ? Do we

18:28.810 --> 18:30.921
compute a Pearson correlation ? Do we

18:30.921 --> 18:33.920
compute a mean squared error

18:34.250 --> 18:37.369
or a spinman correlation ? There are a

18:37.369 --> 18:39.609
number of different choices there . And

18:39.609 --> 18:43.099
then a third . Decision involves the

18:43.099 --> 18:45.599
model comparative inference . How do we ,

18:45.900 --> 18:48.770
um , once we have , you know , a number

18:49.020 --> 18:51.219
that characterizes how well each model

18:51.219 --> 18:53.660
predicts the brain representation , how

18:53.660 --> 18:57.180
do we do statistics on this and compare

18:57.459 --> 18:59.969
different models and in order to

19:00.219 --> 19:02.099
determine whether one model is

19:02.099 --> 19:04.321
significantly better than another model

19:04.339 --> 19:07.540
or whether any differences are just in

19:07.540 --> 19:10.859
the range of Um , what we would expect

19:10.859 --> 19:12.780
on the basis of the noise and the

19:12.780 --> 19:16.640
intersubject variability . So

19:16.640 --> 19:18.807
let's talk about these one by one . So

19:18.807 --> 19:20.680
the first one is the , the RDM

19:20.680 --> 19:24.239
estimator . Um , how do we estimate

19:24.239 --> 19:28.199
these representational distances from

19:29.439 --> 19:32.579
Activity patterns that we've measured .

19:33.400 --> 19:37.239
One thing to keep in mind is that

19:37.239 --> 19:39.461
whenever we measure something , there's

19:39.461 --> 19:41.599
some noise and in the case of

19:41.599 --> 19:43.821
functional magnetic resonance imaging ,

19:43.821 --> 19:46.189
there's a lot of noise actually . And

19:46.189 --> 19:48.133
in the case of neural recordings ,

19:48.133 --> 19:51.479
there's also uh significant noise and

19:51.479 --> 19:54.280
trial to trial variability uh in the

19:54.280 --> 19:57.310
measurements . And if we

19:57.699 --> 20:00.780
naively measure the distance ,

20:01.500 --> 20:03.760
The representational distance by

20:04.250 --> 20:07.329
computing the distance between noisy

20:07.329 --> 20:10.310
activity patterns , then um

20:11.000 --> 20:14.979
We Suffer from something . It's called

20:14.979 --> 20:17.449
a positive bias . So these distance

20:17.449 --> 20:20.750
estimates of noisy data are positively

20:20.750 --> 20:23.609
biased . So let's talk about that for a

20:23.609 --> 20:25.819
moment . So we have this multivariate

20:26.099 --> 20:28.819
neural response space . Um , I've

20:28.819 --> 20:32.219
labeled two axes here , neuron 1 and

20:32.219 --> 20:35.050
neuron 2 , and I've drawn two more

20:35.050 --> 20:37.219
arrows in order to indicate that there

20:37.219 --> 20:39.219
are many more neurons . So it's not

20:39.219 --> 20:42.089
just 4 , but it's , it's hundreds as we

20:42.579 --> 20:45.810
discussed . And we have two particular

20:45.810 --> 20:49.160
response patterns , R I and RJ , and

20:49.400 --> 20:52.449
these dots are meant to indicate the

20:52.449 --> 20:54.560
mean response patterns that you would

20:54.560 --> 20:57.130
get if you presented these two stimuli .

20:57.849 --> 21:01.089
I and J many times over and and average .

21:01.170 --> 21:04.810
So these are the 22 response patterns .

21:08.550 --> 21:11.180
And what we want to estimate is the

21:11.180 --> 21:14.800
distance DIJ between the The two

21:14.800 --> 21:16.959
true response patterns , so the

21:16.959 --> 21:19.181
response patterns that you would get if

21:19.181 --> 21:21.348
you , if you measured indefinitely and

21:21.348 --> 21:25.040
average the patterns . However ,

21:25.329 --> 21:27.760
in practice , we get these noisy

21:27.760 --> 21:30.369
estimates of the patterns . So what we

21:30.369 --> 21:33.410
actually have is this RI hat instead of

21:33.410 --> 21:37.270
the RI . And this I hat is

21:37.270 --> 21:38.859
affected by some error .

21:41.079 --> 21:43.540
Of norm epsilon here , so it's

21:43.540 --> 21:45.930
displaced in this multivariate space

21:46.219 --> 21:48.699
and because the the space is very high

21:48.699 --> 21:50.900
dimensional , so this displacement is

21:50.900 --> 21:54.849
gonna tend to be roughly orthogonal to

21:55.219 --> 21:59.180
um other . Directions in the space ,

21:59.540 --> 22:03.280
including the direction um . Of the

22:03.280 --> 22:05.391
difference between these two response

22:05.391 --> 22:08.729
patterns . And then response pattern J .

22:09.680 --> 22:11.791
Similarly , it's going to be affected

22:11.791 --> 22:13.958
by noise , so our estimate of response

22:13.958 --> 22:16.239
pattern J is going to be somewhat off

22:16.619 --> 22:19.380
and it's going to be displaced , and

22:19.380 --> 22:21.491
since the noise process , we're going

22:21.491 --> 22:23.602
to assume it's the same between these

22:23.602 --> 22:25.547
two patterns . The displacement is

22:25.547 --> 22:27.819
going to be of a similar magnitude as

22:27.819 --> 22:31.380
for pattern RI . And the

22:31.380 --> 22:34.520
direction , if the direction of the ,

22:34.530 --> 22:36.697
the , the distribution of the noise is

22:36.697 --> 22:39.540
isotropic in this uh response pattern

22:39.540 --> 22:41.596
space , the direction is going to be

22:41.596 --> 22:43.318
roughly orthogonal both to the

22:43.318 --> 22:46.479
connection line between RI and RJ and

22:46.819 --> 22:49.979
to the error displacement for the , the

22:49.979 --> 22:51.579
other pattern RI .

22:55.489 --> 22:57.949
So the distance that we would naively

22:57.949 --> 23:00.800
measure if we just compute the distance

23:00.800 --> 23:03.078
between our measured response patterns .

23:03.699 --> 23:07.689
RI hats and RJ hats would be this D

23:07.689 --> 23:11.380
hat IJ . And this is in

23:11.380 --> 23:13.500
general going to be longer than the

23:13.500 --> 23:15.689
true distance DIJ .

23:19.020 --> 23:22.040
And more precisely , it's gonna be .

23:23.349 --> 23:26.900
Roughly true that the square of the

23:26.900 --> 23:30.849
DA IJ , the estimated naively estimated

23:30.849 --> 23:33.569
distance , is gonna be the sum of the

23:33.569 --> 23:37.130
square true distance . Plus 2

23:37.130 --> 23:41.089
times the squared error magnitude , and

23:41.089 --> 23:43.145
this is because these components are

23:43.145 --> 23:45.256
roughly orthogonal to each other in a

23:45.256 --> 23:47.959
in a high dimensional space . So these

23:47.959 --> 23:51.469
um These squares add up according to

23:51.469 --> 23:53.920
the Pythagorean theorem .

23:57.760 --> 23:59.871
And here I've , I've plotted the true

23:59.871 --> 24:02.579
distance versus the estimated distance

24:02.579 --> 24:06.160
for a little . Simulation where I've

24:06.160 --> 24:09.040
added Gaussian noise to the the true

24:09.040 --> 24:13.000
patterns , R I and RJ and

24:13.000 --> 24:16.819
then uh Naively use the

24:16.819 --> 24:20.099
Euclidean distance to estimate the the

24:20.099 --> 24:22.266
distance between the patterns . And as

24:22.266 --> 24:24.209
you can see that the estimated

24:24.540 --> 24:27.489
distances along the vertical are all

24:27.900 --> 24:30.260
larger than the two distances , and

24:30.260 --> 24:33.959
that's What we call the positive bias

24:33.959 --> 24:37.800
of distance estimates . Questions on

24:37.800 --> 24:38.800
this part ?

24:43.010 --> 24:45.010
So this just reflects the fact that

24:45.010 --> 24:47.121
there's error in the measurements and

24:47.121 --> 24:49.849
these errors tend to also push activity

24:49.849 --> 24:52.250
patterns apart . You can think of the

24:52.250 --> 24:55.469
special case also where RI and RJ

24:55.819 --> 24:59.199
are , uh , have a true distance of zero ,

24:59.250 --> 25:01.528
so they're actually the same . Pattern ,

25:01.528 --> 25:03.639
right ? You always have noise and the

25:03.639 --> 25:05.806
distance you measure naively is always

25:05.806 --> 25:08.028
going to be positive . So when the true

25:08.028 --> 25:10.160
distance is zero under noise , you're

25:10.160 --> 25:12.438
always going to get positive distances ,

25:12.438 --> 25:14.382
and that's another very simple and

25:14.382 --> 25:16.493
intuitive way of understanding this ,

25:16.520 --> 25:20.430
this positive bias . So how can we

25:20.430 --> 25:24.270
remove the bias ? Well , we can

25:24.270 --> 25:27.119
think of this slightly more general um

25:27.119 --> 25:30.180
scenario where we we're not looking at

25:30.180 --> 25:32.689
the Euclidean distance , but we're

25:32.689 --> 25:35.130
looking at the Malanobus distance . So

25:35.130 --> 25:38.849
this is just um adding the inverse of

25:38.849 --> 25:41.599
the noise covariance matrix in between

25:41.599 --> 25:45.040
these two pattern differences .

25:45.369 --> 25:47.729
Um , we take the , the inner product of

25:47.729 --> 25:50.449
the pattern differences uh would be the

25:50.449 --> 25:52.760
square Euclidean distance .

25:54.369 --> 25:57.770
Which is equal to the uh sum of squared

25:57.770 --> 26:00.729
deviations between the two patterns and

26:00.729 --> 26:03.689
for the Malanobus distance we uh do a

26:03.689 --> 26:06.369
multivariate noise normalization and

26:06.369 --> 26:09.810
this can be achieved by inserting in

26:09.810 --> 26:13.010
the middle of this uh this inner

26:13.010 --> 26:15.369
product here , the inverse of the noise

26:15.369 --> 26:18.410
covariance matrix . So this would be um

26:18.410 --> 26:20.577
the square mile over distance and this

26:20.577 --> 26:22.688
would be affected by the same kind of

26:22.688 --> 26:24.632
bias that I introduced you to just

26:24.632 --> 26:27.469
before . In order to remove the bias ,

26:27.540 --> 26:30.689
we can use a cross validation approach

26:30.689 --> 26:33.930
where um we just use a different data

26:33.930 --> 26:37.439
set . In , um , the second term here

26:37.439 --> 26:39.800
for um the difference between these two

26:39.800 --> 26:42.520
response pattern vectors . When we do

26:42.520 --> 26:46.479
that , um , we have these test vectors

26:46.479 --> 26:50.439
here , um , affected by noise as these

26:50.439 --> 26:52.479
two vectors here are . So these are

26:52.479 --> 26:55.910
both um noisy estimates . However , the

26:55.910 --> 26:58.439
noise is different now . The noise is

26:58.439 --> 27:00.272
pointing in different directions

27:00.272 --> 27:03.069
because it's a different data set . And

27:03.069 --> 27:06.670
uh the effect of that is that um

27:07.680 --> 27:09.880
When the true distance is zero , the

27:09.880 --> 27:13.239
noise vectors are approximately

27:13.239 --> 27:16.040
orthogonal , and so the estimate is

27:16.040 --> 27:19.199
distributed symmetrically about 0 and

27:19.199 --> 27:22.390
the bias is is gone . You can also

27:22.390 --> 27:24.880
think of this as um

27:25.979 --> 27:28.540
Using the Fisher linear discriminant as

27:28.540 --> 27:30.780
a linear decoder and the fissure linear

27:30.780 --> 27:33.099
discriminant weights of this , this

27:33.099 --> 27:35.989
first , um , part of this equation here ,

27:36.060 --> 27:40.020
so the RI minus the RJ transpose times

27:40.020 --> 27:42.260
the inverse of the noise covariance ,

27:42.339 --> 27:46.250
that's the weights vector W um that is

27:46.250 --> 27:48.472
known as the . linear discriminant , if

27:48.472 --> 27:51.775
we use that as a linear decoder , we

27:51.775 --> 27:54.854
could project data points from a

27:54.854 --> 27:57.045
different data set onto that decoder ,

27:57.135 --> 27:59.574
and that's what we would usually do in

27:59.574 --> 28:02.494
a linear decoding analysis to determine

28:02.494 --> 28:05.055
the accuracy with which we can decode

28:05.055 --> 28:07.175
which of the two stimuli has been

28:07.175 --> 28:09.084
presented to the subject .

28:10.709 --> 28:12.750
And because we're using a different

28:12.750 --> 28:16.750
data set , um , we , uh ,

28:17.189 --> 28:19.979
have an unbiased estimate , for example ,

28:19.989 --> 28:22.780
of decoder accuracy , um , which is not

28:22.780 --> 28:26.069
biased by overfitting the decoder to to

28:26.069 --> 28:29.349
a given training data set . And

28:29.349 --> 28:33.349
if instead of determining the accuracy

28:33.349 --> 28:35.910
of the decoder , we just project the

28:35.910 --> 28:37.910
difference between the two patterns

28:37.910 --> 28:40.780
onto the decoder , we get uh

28:41.819 --> 28:43.763
Another way of looking at why this

28:43.763 --> 28:46.969
provides an Unbiased estimate

28:47.420 --> 28:50.390
um of the distance . So in the

28:50.390 --> 28:52.869
particular case that the two patterns

28:52.869 --> 28:55.550
are truly the same and they're only

28:55.550 --> 28:58.290
different due to the noise , the

28:58.790 --> 29:01.030
decoder discriminant direction would

29:01.030 --> 29:03.619
point in a random direction , and then

29:03.790 --> 29:05.901
we'd have the difference for the test

29:05.901 --> 29:07.846
data set and that would point in a

29:07.846 --> 29:09.630
random direction , and then the

29:09.630 --> 29:11.686
projection of these two vectors onto

29:11.686 --> 29:14.270
each other um would have an expected

29:14.270 --> 29:17.069
value of 0 . And since the true

29:17.560 --> 29:20.319
distance in this scenario is 0 , that's

29:20.319 --> 29:22.880
what we want that uh ensures that we

29:22.880 --> 29:25.359
have an unbiased distance estimate .

29:26.810 --> 29:29.300
So here's how this this works out in a

29:29.300 --> 29:32.569
in a simulation . So we have different

29:32.569 --> 29:36.089
uh simulated true distances here that

29:36.089 --> 29:39.869
you see . Um , encoded in

29:39.869 --> 29:42.810
these different shades of gray . If you

29:42.810 --> 29:44.810
can see my pointer .

29:47.150 --> 29:49.180
Can you see my pointer .

29:52.900 --> 29:55.859
Your cursor , I , I think so . Yeah .

29:57.170 --> 29:59.900
So looks good . Yeah , my , um , view

29:59.900 --> 30:03.630
of the slide is , um . Darkened in

30:03.630 --> 30:07.160
the In the video conferencing , but you

30:07.160 --> 30:10.089
can see it clearly , right ? Yeah , I

30:10.089 --> 30:12.311
see the shades of gray you're referring

30:12.311 --> 30:14.311
to . So there are the the shades of

30:14.311 --> 30:16.367
gray here , these are different true

30:16.367 --> 30:18.478
distances . The light one is the case

30:18.478 --> 30:20.630
where the true distance is zero . And

30:20.630 --> 30:22.829
then the dark one is a case where the

30:22.829 --> 30:26.790
true distance is 2 , and if we don't

30:26.790 --> 30:29.270
use cross validation for estimating the

30:29.380 --> 30:32.189
the distance , we get the true

30:32.189 --> 30:34.300
distances when there's no noise , but

30:34.300 --> 30:36.380
the more noise there is , the bigger

30:36.780 --> 30:40.569
the the positive bias becomes . And

30:40.569 --> 30:42.689
when we use the cross validated

30:42.689 --> 30:45.849
distance estimator , then for each uh

30:45.849 --> 30:49.489
true distance , we get an expectation

30:49.969 --> 30:52.650
the right estimate here and as the

30:52.650 --> 30:55.410
noise goes goes up , we don't get

30:55.410 --> 30:57.739
bigger and bigger estimates we just get

30:59.089 --> 31:01.489
Noisier and noisier estimates , right ?

31:01.569 --> 31:04.069
But the , the mean , the expected value

31:04.329 --> 31:08.290
of the , the estimate is um the true

31:08.290 --> 31:10.839
value . And that's the definition of

31:10.839 --> 31:11.920
unbiasedness .

31:15.579 --> 31:17.246
Are there questions on this ?

31:21.170 --> 31:23.810
So this is um this is a bit of a

31:23.810 --> 31:26.290
technicality , but it's , it's useful

31:26.290 --> 31:28.520
to be able to measure these

31:28.520 --> 31:30.920
representational distances without bias

31:31.369 --> 31:33.729
because that gives us a more accurate

31:33.729 --> 31:36.449
representation of the geometry and

31:36.449 --> 31:38.390
that's what we , what we need to

31:38.390 --> 31:41.209
adjudicate between models . The next

31:41.209 --> 31:44.310
question is , Once we've estimated

31:44.310 --> 31:46.989
these representation of geometries by

31:47.359 --> 31:49.581
getting estimates of the representation

31:49.581 --> 31:52.810
of the similarity matrices . We have

31:52.810 --> 31:55.099
representational dissimilarity matrices

31:55.380 --> 31:57.859
predicted by each model , and we have a

31:57.859 --> 32:00.081
measured representational dissimilarity

32:00.081 --> 32:03.199
matrix . Now how do we compare these ?

32:03.239 --> 32:06.020
How do we assess how good the

32:06.439 --> 32:09.459
prediction is for a given model ? So

32:09.459 --> 32:13.050
this is about accounting , um , so what

32:13.050 --> 32:14.979
one challenge in this context is

32:14.979 --> 32:17.170
accounting for the dependency among

32:17.170 --> 32:20.089
dissimilarity estimates , and we can

32:20.459 --> 32:22.300
account for this dependency by

32:22.300 --> 32:25.140
whitening . But first , um ,

32:26.099 --> 32:29.939
Let's , let's talk about um the

32:29.939 --> 32:33.060
dependency between uh the dissimilarity

32:33.060 --> 32:36.099
estimates . So once we have an RDM

32:36.579 --> 32:39.900
this RDM lives in a space where we have

32:39.900 --> 32:42.500
one dimension for each pair's pair of

32:42.500 --> 32:45.739
stimuli . So because we , we estimate

32:45.739 --> 32:48.140
the distance in the distance matrix for

32:48.140 --> 32:52.040
each pair of stimuli , so , uh . The

32:52.040 --> 32:55.390
distance matrices are points in a space

32:55.800 --> 32:57.856
where there's one dimension for each

32:57.856 --> 33:00.479
dissimilarity , and there's 11

33:00.479 --> 33:02.757
dissimilarity for each pair of stimuli ,

33:03.040 --> 33:05.262
so we have a dimension for each pair of

33:05.262 --> 33:09.199
stimuli . However , um , The

33:09.199 --> 33:12.449
dissimilarity estimates are not

33:13.150 --> 33:16.310
independent of each other . Consider ,

33:16.400 --> 33:19.199
for example , two dissimilarity

33:19.199 --> 33:22.989
estimates . That share a stimulus .

33:23.150 --> 33:25.039
So for example , you might have 3

33:25.039 --> 33:28.229
stimuli A , B , and C , and you have 1

33:28.650 --> 33:31.030
dissimilarity estimate between stimulus

33:31.030 --> 33:33.189
A and stimulus B , and another one

33:33.189 --> 33:36.780
between stimulus B and stimulus C . Um ,

33:36.939 --> 33:39.540
so since these two dissimilarities

33:39.540 --> 33:41.760
share one stimulus , stimulus B .

33:43.689 --> 33:45.856
The estimates of these dissimilarities

33:45.856 --> 33:48.119
are going to be dependent because as

33:48.119 --> 33:50.439
our estimate of the response pattern

33:50.439 --> 33:54.000
for stimulus B moves around ,

33:54.569 --> 33:56.989
um , so let's say um this is my

33:56.989 --> 33:59.045
estimate for the response pattern of

33:59.045 --> 34:01.560
stimulus A and this is my estimate for

34:01.560 --> 34:03.989
the response pattern of stimulus C .

34:04.319 --> 34:06.760
and now here's stimulus B , as this

34:06.760 --> 34:09.800
moves around , it's gonna affect the

34:09.800 --> 34:13.320
estimates of the dissimilarities . Both

34:13.320 --> 34:16.550
between A and B and B and C

34:17.090 --> 34:20.080
and so they're gonna be uh correlated

34:20.080 --> 34:22.302
and in general they're gonna tend to be

34:22.302 --> 34:24.709
positively correlated . So we have this

34:24.709 --> 34:26.879
correlational structure in our

34:26.879 --> 34:29.449
dissimilarity estimates and we can

34:29.449 --> 34:31.209
model that as a multi-normal

34:31.209 --> 34:33.810
distribution in the space of all

34:34.169 --> 34:38.129
RDMs . And so um what I'm , what I've

34:38.129 --> 34:41.040
indicated here with these um . These

34:41.040 --> 34:44.750
red distributions is the the sampling

34:44.750 --> 34:48.020
distributions of the the estimated um

34:48.270 --> 34:51.699
RDMs , and that can be um

34:51.949 --> 34:54.989
approximated as a multi-normal

34:54.989 --> 34:56.989
distribution in the space of all

34:56.989 --> 35:00.760
possible RDMs . So imagine we have this

35:00.760 --> 35:03.560
scenario where we have um the data

35:03.560 --> 35:07.120
RDMD and two model predicted

35:07.120 --> 35:09.669
RDMs , M1 and M2 .

35:10.590 --> 35:13.070
However , we also know that there's a

35:13.070 --> 35:15.620
dependency structure indicated by these

35:15.790 --> 35:18.350
ellipses , where each ellipse is an

35:18.350 --> 35:21.550
isoprobability density contour of one

35:21.550 --> 35:24.629
of these uh multi-normal distributions .

35:25.429 --> 35:29.300
So if we naively um compared these

35:29.870 --> 35:33.229
um these RDMs , we might conclude that

35:33.229 --> 35:36.510
model 2 is a better model of the

35:36.510 --> 35:38.454
representation in the brain region

35:38.454 --> 35:40.939
we're looking at because the model ,

35:40.989 --> 35:44.570
model . Two predicted RDM is

35:44.570 --> 35:47.969
closer in a Euclidean distance sense to

35:47.969 --> 35:49.929
the data RDMD .

35:52.169 --> 35:54.909
However , if we take The

35:55.419 --> 35:58.929
structure . Of the dependency

35:59.219 --> 36:01.739
of the the the similarity estimates

36:01.739 --> 36:05.500
into account , we see that model

36:05.500 --> 36:09.100
M1 is actually a better model of the

36:09.100 --> 36:12.860
data because uh the likelihood , the

36:12.860 --> 36:16.379
probability of the data given model M1

36:16.379 --> 36:19.100
is higher than the probability of the

36:19.100 --> 36:21.659
data given model M2 .

36:23.780 --> 36:25.969
And we can account for this by

36:25.969 --> 36:29.080
estimating . The dissimilarity

36:29.080 --> 36:32.239
estimation error covariance the .

36:34.500 --> 36:37.820
And this is a um An

36:37.820 --> 36:39.949
estimator of this the similarity

36:39.949 --> 36:42.005
estimation error covariance has been

36:42.005 --> 36:45.639
derived by uh young Edison and Haikoch .

36:47.719 --> 36:51.229
And by taking this uh covariance into

36:51.229 --> 36:53.820
account , we can widen this RDM space

36:54.209 --> 36:56.699
and after whitening we can measure the

36:56.709 --> 36:59.780
the Euclidean distance , and that will

37:00.229 --> 37:02.989
then correctly reveal that model one is

37:02.989 --> 37:06.510
actually in this case a better model of

37:06.510 --> 37:08.909
the data . So we can think of this as

37:08.909 --> 37:12.159
using the Malanobus distance in RDM

37:12.159 --> 37:14.620
space instead of the Euclidean distance .

37:15.100 --> 37:16.899
Or we can think of it as first

37:16.899 --> 37:18.889
whitening the whole space and then

37:18.889 --> 37:20.945
using the Euclidean distance . Those

37:20.945 --> 37:24.780
two are um equivalent . Hey Nico .

37:25.159 --> 37:27.310
Sure . Can you , um , this is Kevin ,

37:27.399 --> 37:29.520
can you just describe the whitening

37:29.520 --> 37:32.290
thing ? I , I , I wasn't , I didn't

37:32.520 --> 37:34.464
catch that here what it is , and I

37:34.464 --> 37:36.520
would , I don't read this stuff very

37:36.520 --> 37:38.631
often , so in the paper , I , I , can

37:38.631 --> 37:40.798
you clarify that a little bit ? Yeah ,

37:40.798 --> 37:43.879
so whitening means that you're linearly

37:43.879 --> 37:47.600
transforming the entire space such that

37:47.600 --> 37:50.719
the noise will be isotropic . That's

37:50.719 --> 37:53.469
the The definition of whitening , right ?

37:53.590 --> 37:55.870
So in this case we uh

37:57.820 --> 38:01.149
The noise is uh well approximated as a

38:01.149 --> 38:04.149
multi-normal , and the noise is the

38:04.149 --> 38:07.939
same for different points in the space .

38:08.469 --> 38:09.719
So by

38:12.169 --> 38:15.370
Stretching and squeezing linearly by a

38:15.370 --> 38:18.800
linear transform of the original axis

38:18.800 --> 38:22.270
of this the space , we can render these

38:22.270 --> 38:26.000
ellipses circles . Making the

38:26.000 --> 38:29.600
noise isotropic . And the particular

38:29.600 --> 38:32.040
linear transform that is needed to

38:32.040 --> 38:35.719
achieve this is related to the

38:35.719 --> 38:38.399
covariance matrix V that characterizes

38:38.399 --> 38:40.879
the shape of these uh multi-normal

38:40.879 --> 38:42.601
distributions , right ? So the

38:42.601 --> 38:44.212
multi-normal distribution is

38:44.212 --> 38:46.959
characterized by its covariance matrix

38:46.959 --> 38:50.639
V . And if we apply it

38:50.639 --> 38:53.870
to all the points in the space , A

38:53.870 --> 38:56.260
particular linear transform and the

38:56.260 --> 38:58.482
transform that's needed is to the power

38:58.482 --> 39:02.290
of minus 12 . Then after that linear

39:02.290 --> 39:04.750
transform we've sort of stretched and

39:04.750 --> 39:07.899
squeezed , um , we can with the linear

39:07.899 --> 39:09.843
transform you can rotate , you can

39:09.843 --> 39:11.843
stretch , you can squeeze , you can

39:11.843 --> 39:14.080
shear , but it's all linear and uniform

39:14.080 --> 39:16.780
across the entire space . After you do

39:16.780 --> 39:20.419
that , the noise is isotropic and then

39:20.419 --> 39:22.770
just measuring the Euclidean distance .

39:24.219 --> 39:27.449
Reflects the degree to which

39:27.620 --> 39:31.090
um Different

39:32.370 --> 39:35.010
Models can account for the data .

39:36.020 --> 39:38.629
Better than without doing this because

39:38.629 --> 39:41.899
we're we're taking the The structure of

39:41.899 --> 39:43.288
the dependency among the

39:43.288 --> 39:45.010
dissimilarities into account .

39:47.669 --> 39:49.725
So in , in other words , here , um ,

39:49.725 --> 39:51.558
you know , the structure of this

39:51.558 --> 39:53.780
distribution is . You can think of this

39:53.780 --> 39:57.340
as in some directions in this RDM space ,

39:58.030 --> 40:00.510
our our estimate is noisy . So if we

40:00.510 --> 40:02.510
have a large displacement in that

40:02.510 --> 40:05.810
direction . This could easily be due to

40:05.810 --> 40:08.570
noise as it is for model one . So model

40:08.570 --> 40:12.129
one is actually the best model because

40:12.129 --> 40:13.740
even though there is a large

40:13.740 --> 40:15.462
displacement to the data RDM ,

40:15.649 --> 40:17.649
displacement , this displacement is

40:17.649 --> 40:20.649
very consistent with having occurred

40:20.649 --> 40:22.770
due to the noise because the noise is

40:22.770 --> 40:25.209
large in that direction , right ?

40:25.530 --> 40:28.689
Whereas the smaller displacement um

40:28.689 --> 40:30.856
that we would have to assume to be due

40:30.856 --> 40:34.540
to noise if model 2 were true . It is

40:34.540 --> 40:37.820
nevertheless less likely because the

40:37.820 --> 40:41.689
noise is as as smaller variants

40:41.939 --> 40:45.659
in that direction from model 2 to the

40:45.659 --> 40:46.419
data RDM .

40:51.239 --> 40:53.295
That was , that was helpful . Yeah ,

40:53.295 --> 40:57.169
thank you . So

40:58.659 --> 41:01.500
Using this approach , we can define RDM

41:01.500 --> 41:05.120
comparators that take the structure of

41:05.120 --> 41:07.760
the dissimilarity estimation error

41:07.760 --> 41:11.500
covariance into account um . Such

41:11.500 --> 41:14.020
as the whitened , here's an RDM

41:14.020 --> 41:18.020
correlation and the whitened cosine RDM

41:18.020 --> 41:19.870
similarities . So these are two

41:19.870 --> 41:21.540
alternative estimators .

41:24.860 --> 41:27.560
And it turns out that these estimators

41:28.000 --> 41:30.280
generalize the linear centered kernel

41:30.280 --> 41:33.060
alignment , which is a very popular

41:33.379 --> 41:35.101
method in the machine learning

41:35.101 --> 41:37.040
literature for comparing

41:37.040 --> 41:40.510
representations in neural networks . It

41:40.510 --> 41:43.429
generalizes this linear kernel uh

41:43.429 --> 41:45.750
alignment to unbiased distance

41:45.750 --> 41:48.429
estimators . So the linear kernel

41:48.429 --> 41:51.500
alignment is the , the special case of

41:51.500 --> 41:54.780
these whitened Pearson and cosine RDM ,

41:55.149 --> 41:58.129
um . Similarities

42:00.729 --> 42:03.750
For bias distance estimators .

42:04.530 --> 42:08.169
And uh And the the these new tools

42:08.169 --> 42:09.949
we've we've generalized them to

42:09.949 --> 42:12.360
unbiased distance estimators , which is

42:12.360 --> 42:14.629
very useful when you're dealing with

42:14.629 --> 42:17.750
noisy data as we do as scientists .

42:22.659 --> 42:26.469
Another The thing that we can do by

42:26.469 --> 42:30.250
choosing a different RDM comparator is

42:30.250 --> 42:32.760
focus on the topology of the

42:32.760 --> 42:35.639
representation rather than the geometry

42:35.639 --> 42:37.919
of the representation . So I've said in

42:37.919 --> 42:40.949
the beginning that we can characterize

42:40.949 --> 42:43.399
the geometry of the representation by

42:43.399 --> 42:47.030
the distance matrix . We can

42:47.030 --> 42:50.310
characterize the topology of the

42:50.310 --> 42:53.669
representation by transforming the

42:53.669 --> 42:57.020
distance matrix nonlinearly . So let me

42:57.429 --> 43:00.540
illustrate this for you . Imagine you

43:00.540 --> 43:04.370
have a set of stimuli along

43:04.739 --> 43:07.780
a single dimension , such as um the

43:07.780 --> 43:11.030
orientation of a grating . Or the

43:11.030 --> 43:14.290
direction of motion in a 2D

43:15.110 --> 43:19.060
plane . So you have a , a bunch

43:19.060 --> 43:21.929
of different um stimuli and you have

43:21.929 --> 43:23.959
them encoded in a set of response

43:23.959 --> 43:27.280
patterns in the multivariate uh space .

43:27.620 --> 43:31.540
So we usually think of uh the set of

43:31.540 --> 43:34.340
response patterns that encode some low

43:34.340 --> 43:38.179
dimensional , um , Set

43:38.179 --> 43:41.060
of stimuli where the stimuli are

43:41.060 --> 43:43.379
described by a small number of

43:43.379 --> 43:45.649
parameters in this case one parameter .

43:45.939 --> 43:48.260
We think of that as the neural response

43:48.260 --> 43:50.570
manifold . So what I'm showing you here

43:50.860 --> 43:53.429
is a toy example where we have

43:53.820 --> 43:57.090
different neural response manifolds ,

43:57.340 --> 43:59.340
and they're one dimensional because

43:59.340 --> 44:01.396
there's one intrinsic dimension here

44:01.620 --> 44:03.842
and in some of them there is a crossing

44:03.842 --> 44:07.590
here , so this is actually um . Um , an

44:07.590 --> 44:09.757
intersection , which is not consistent

44:09.757 --> 44:12.310
with the , the definition of the term

44:12.310 --> 44:15.909
manifold , but it's still a possibility

44:15.909 --> 44:17.798
that a neural representation of a

44:17.798 --> 44:20.709
single variable intersects itself at

44:20.709 --> 44:24.090
some point . And

44:24.600 --> 44:27.239
we have on the left side here two

44:27.239 --> 44:29.010
scenarios , two possible

44:29.010 --> 44:31.121
representations where there is such a

44:31.121 --> 44:33.360
crossing . And on the right side here

44:33.360 --> 44:37.080
we have two scenarios where there's not

44:37.080 --> 44:39.120
such a crossing . So in the two

44:39.120 --> 44:41.870
right-hand scenarios , um , the neural

44:42.639 --> 44:44.830
representation truly is a manifold .

44:44.879 --> 44:48.590
You could call it a representational uh

44:48.590 --> 44:52.310
neural manifold . And on the left

44:52.310 --> 44:55.219
side here , um , you have

44:55.669 --> 44:58.389
um representations where there is this

44:58.389 --> 45:02.090
crossing . So You have 4

45:02.090 --> 45:04.449
different representations here , and

45:04.449 --> 45:06.850
you have a different structure when you

45:06.850 --> 45:09.639
care about the geometry . So , um ,

45:09.649 --> 45:12.479
this first one here and the third one ,

45:12.649 --> 45:15.649
they're geometrically similar . If you

45:15.649 --> 45:17.482
just look at the shape of them .

45:18.169 --> 45:20.391
They're quite similar in terms of their

45:20.391 --> 45:22.447
geometry . However , one of them has

45:22.447 --> 45:24.502
the crossing , the other one doesn't

45:24.502 --> 45:26.502
have the crossing . And similarly ,

45:26.502 --> 45:28.558
when you look at the second one here

45:28.558 --> 45:30.489
and the 4th 1 , um , they're

45:30.489 --> 45:33.169
geometrically similar , but they're not

45:33.169 --> 45:35.530
topologically similar . So the

45:35.530 --> 45:38.010
topologically similar ones are the two

45:38.010 --> 45:40.320
left ones and the two right ones here ,

45:40.889 --> 45:43.649
um , and the topological similarity

45:43.649 --> 45:45.816
derives from the fact that these two .

45:46.340 --> 45:48.451
Have a crossing , and these two don't

45:48.451 --> 45:52.010
have a crossing . So

45:52.030 --> 45:55.129
imagine we were interested , so if we

45:55.129 --> 45:58.489
do , you know , normal RSA based on

45:58.489 --> 46:01.520
RDMs , we'd be sensitive to those

46:01.520 --> 46:04.010
geometrical similarities , but we

46:04.010 --> 46:06.129
wouldn't be very sensitive to those

46:06.129 --> 46:09.409
topological similarities here . So we

46:09.409 --> 46:12.570
group these four representations

46:12.570 --> 46:14.889
according to their geometry , not their

46:14.889 --> 46:17.800
topology , and that's reflected . In

46:17.800 --> 46:19.744
the representational dissimilarity

46:19.744 --> 46:23.239
matrices , if you look at the RDM for

46:23.239 --> 46:25.295
each of these four representations ,

46:25.295 --> 46:28.560
you see that the first one and the 3rd

46:28.560 --> 46:30.800
1 are similar , and the second one and

46:30.800 --> 46:34.629
the 4th 1 are quite similar . But the

46:34.629 --> 46:37.310
left two are not so similar and the

46:37.310 --> 46:39.510
right two are not so similar .

46:41.830 --> 46:45.139
When we look at uh The

46:45.139 --> 46:48.659
geodesics of those representations , so

46:48.659 --> 46:51.659
when we measure the distances not along

46:51.659 --> 46:53.770
straight lines in this multivariate

46:53.770 --> 46:56.580
response space , but along the manifold

46:56.580 --> 46:59.379
instead , and we get different matrices

46:59.379 --> 47:03.219
that reflect . The topology rather than

47:03.219 --> 47:05.860
the geometry . And in this case , you

47:05.860 --> 47:08.300
can see with the naked eye that um you

47:08.300 --> 47:11.739
get similar matrices um for these two

47:11.739 --> 47:13.979
on the left and similar matrices for

47:13.979 --> 47:17.260
these two on the right . So in order

47:17.260 --> 47:20.989
to um be sensitive to these

47:22.129 --> 47:25.149
RDM comparisons to the

47:25.409 --> 47:28.919
topologies , we can replace the RDM

47:29.449 --> 47:32.050
by the representational geodesics

47:32.050 --> 47:33.689
matrix or RGDM .

47:35.929 --> 47:38.280
And in the next few slides , I'm gonna

47:38.590 --> 47:40.679
tell you a little bit more about how

47:40.679 --> 47:42.629
this works . Can I ask you a quick

47:42.629 --> 47:44.909
question ? Sure . Uh , it's a dumb

47:44.909 --> 47:47.020
question . I think the mathematicians

47:47.020 --> 47:49.187
in the room are eating this stuff up ,

47:49.187 --> 47:52.110
um , but I have , uh , in this , um ,

47:52.879 --> 47:55.750
Crossover here in the manifold , or

47:55.750 --> 47:57.861
it's not really a manifold because it

47:57.861 --> 47:59.917
has this crossover thing that you're

47:59.917 --> 48:02.083
saying , you're saying that can happen

48:02.083 --> 48:04.194
in neuroscience and like neuroscience

48:04.194 --> 48:06.306
or psychology terms , what what would

48:06.306 --> 48:08.899
be happening to cause that ? So there

48:08.899 --> 48:12.780
would be a part of the Um , so two

48:12.780 --> 48:15.580
different stimuli that elicit the same

48:15.580 --> 48:17.580
response pattern , right ? So you'd

48:17.580 --> 48:20.300
have a representation where um there's

48:20.300 --> 48:23.340
particular stimuli that are distinct in

48:23.340 --> 48:25.780
our representation . So here they're

48:25.780 --> 48:28.399
color coded in black and white , um ,

48:28.419 --> 48:31.020
but they're not distinct in the

48:31.020 --> 48:33.242
representation , right ? So they elicit

48:33.242 --> 48:34.853
the same response patterns .

48:37.080 --> 48:40.129
So you're looking at a particular brain

48:40.129 --> 48:42.129
region , let's say , and that brain

48:42.129 --> 48:44.240
region , doesn't have any distinction

48:44.240 --> 48:46.185
in how it's representing those two

48:46.185 --> 48:48.629
different stimuli . Yeah , we often

48:48.629 --> 48:50.851
hear about , you know , selectivity and

48:50.851 --> 48:53.429
invariants . So invariance means you

48:53.429 --> 48:55.762
can change something about the stimulus ,

48:55.762 --> 48:58.096
but the response pattern doesn't change .

48:58.096 --> 49:00.870
The population of neurons doesn't care

49:00.870 --> 49:04.310
about whatever property , um , the

49:04.310 --> 49:05.870
region is invariant to .

49:08.669 --> 49:09.669
Thank you .

49:12.929 --> 49:16.770
So let's dig a little deeper and um

49:16.770 --> 49:18.300
think about um

49:20.989 --> 49:23.629
Geometry and topology for a bunch of

49:23.629 --> 49:26.820
stimuli . So here's a hypothetical

49:27.270 --> 49:30.590
representational space um where each of

49:30.590 --> 49:33.110
those nodes represents the response

49:33.110 --> 49:35.310
pattern elicited by a different

49:35.310 --> 49:38.149
stimulus , and you can think of a a dog

49:38.149 --> 49:40.550
and a cat and a tree and a bus and a

49:40.550 --> 49:43.479
car . And some of them are further

49:43.479 --> 49:45.360
apart than than others .

49:48.229 --> 49:51.820
So The motivation is

49:51.820 --> 49:54.449
that the motivation for caring about

49:54.570 --> 49:56.514
about the topology rather than the

49:56.514 --> 49:59.780
geometry is that once two stimuli are

49:59.780 --> 50:02.060
clearly distinct , further increasing

50:02.060 --> 50:05.379
their distance does not increase their

50:05.379 --> 50:07.659
discriminability . So if the noise

50:07.659 --> 50:09.739
distributions already are not

50:09.739 --> 50:12.179
overlapping anymore , if you push those

50:12.179 --> 50:14.860
stimuli further apart , you're not

50:14.860 --> 50:17.699
adding um information .

50:18.139 --> 50:21.510
Um , with respect to , uh ,

50:21.520 --> 50:24.879
discriminating those two stimuli . And

50:24.879 --> 50:28.600
then secondly , when two stimuli elicit

50:28.600 --> 50:31.969
very similar response patterns . Then ,

50:32.300 --> 50:35.060
um , the differences between them may

50:35.060 --> 50:37.659
be negligible given the noise and the

50:37.659 --> 50:40.260
representation . So the stimuli already

50:40.260 --> 50:43.090
are not discriminable and perhaps those

50:43.100 --> 50:46.260
those small differences uh between

50:46.260 --> 50:49.459
different um dissimilarities , all of

50:49.459 --> 50:52.500
which are small , don't matter so much

50:52.500 --> 50:55.219
to um the nature of the representation .

50:56.540 --> 50:59.870
And then thirdly , The discriminability

50:59.870 --> 51:02.760
sensitively depends on distances when

51:02.760 --> 51:05.129
the distances in some intermediate

51:05.320 --> 51:09.080
range , right ? So , This is a

51:09.120 --> 51:12.770
a way of motivating intuitively that we

51:12.770 --> 51:15.739
might care especially about

51:16.270 --> 51:19.389
Variation of dissimilarities in an

51:19.389 --> 51:21.659
intermediate range when things are very

51:21.659 --> 51:24.070
close , then maybe we can neglect the

51:24.070 --> 51:27.110
tiny differences between these pairs of

51:27.110 --> 51:29.469
stimuli , all of which are very close ,

51:29.590 --> 51:32.219
and when things are very far apart ,

51:32.510 --> 51:34.177
maybe they're just completely

51:34.177 --> 51:36.629
discriminable and we , we don't need to

51:36.629 --> 51:39.149
care about exactly how far apart they

51:39.149 --> 51:42.969
are . So this motivates turning an

51:42.969 --> 51:46.439
RDM into a weighted graph where we

51:46.729 --> 51:48.618
start with the RDM , the distance

51:48.618 --> 51:51.090
matrix . Um , one way to turn it into a

51:51.090 --> 51:53.370
graph would be to apply a hard

51:53.370 --> 51:56.080
threshold on the distances where we say

51:56.330 --> 51:58.552
all the distances that are smaller than

51:58.552 --> 52:01.649
some threshold , they're um zero , and

52:01.649 --> 52:03.816
then all the distances that are larger

52:03.816 --> 52:07.510
than some thresholds are 1 . And

52:07.510 --> 52:11.090
this um threshold of distance matrix

52:11.600 --> 52:13.850
is also related it's a complement of

52:13.850 --> 52:17.169
the adjacency matrix of a graph where

52:17.169 --> 52:19.729
we're connecting the things that are

52:19.729 --> 52:22.810
neighbors and we're disconnecting um

52:22.810 --> 52:26.750
things that are not neighbors . What we

52:26.750 --> 52:30.510
are proposing here is to use a soft

52:30.510 --> 52:34.340
threshold in order to set

52:34.340 --> 52:37.949
all the small distances to zero and all

52:37.949 --> 52:40.870
the large distances to some maximum

52:40.870 --> 52:43.750
value , making this one by convention

52:43.750 --> 52:46.820
here , and then still to be sensitive

52:46.820 --> 52:49.389
to the transition between these two .

52:51.060 --> 52:52.969
This gives rise to a family of

52:52.969 --> 52:55.889
geotopological distance transforms that

52:55.889 --> 52:59.409
are defined by two thresholds L and U ,

52:59.449 --> 53:02.010
a lower and an upper threshold .

53:03.850 --> 53:06.360
And by choosing different values for L

53:06.360 --> 53:09.840
and U , we get different transforms .

53:10.500 --> 53:13.080
Different instances of this piecewise

53:13.080 --> 53:16.030
linear transform of the the distances .

53:16.399 --> 53:19.110
For each of these transforms , we can

53:19.399 --> 53:22.280
apply the transform to the RDM and we

53:22.280 --> 53:23.760
get a different RDM .

53:26.040 --> 53:28.659
And so we get this this family of what

53:28.659 --> 53:31.739
we call geotopological um distance

53:31.739 --> 53:35.739
transforms where in the upper left

53:35.739 --> 53:39.550
corner here . Um , where the L is

53:39.550 --> 53:43.540
zero and the U , the upper bound

53:43.540 --> 53:46.229
is maximum , we're not transforming the

53:46.229 --> 53:48.118
RDM at all , so we have the , the

53:48.118 --> 53:51.189
original RDM as one member of this

53:51.189 --> 53:54.800
family . And then along the

53:54.800 --> 53:58.760
diagonal here we have binary RDMs

53:58.760 --> 54:00.427
where we've applied some hard

54:00.427 --> 54:02.709
thresholds and as we move along the

54:02.709 --> 54:05.080
diagonal we've applied the threshold at

54:05.080 --> 54:07.270
a different level . So we have this

54:07.280 --> 54:09.879
this binary RDM that says that certain

54:09.879 --> 54:12.649
things are just together in a cluster

54:13.000 --> 54:15.056
and other things are just separate .

54:16.770 --> 54:19.709
And then we also have an intermediate

54:19.709 --> 54:22.590
space here where we're uh sensitive to

54:22.590 --> 54:26.360
different aspects of the Geometry and

54:26.360 --> 54:27.120
topology .

54:31.860 --> 54:35.389
So when we apply this to the simple toy

54:35.389 --> 54:38.449
scenario , we first have to decide on a

54:38.449 --> 54:41.010
number of stimuli that that's been

54:41.010 --> 54:42.679
included in this hypothetical

54:42.679 --> 54:45.570
experiment . So I'm gonna replace these

54:45.570 --> 54:49.489
continuous manifolds by uh sort of a

54:49.489 --> 54:52.169
finite number of little balls here that

54:52.169 --> 54:54.002
represent particular positions .

54:56.709 --> 54:59.790
And then we can apply um this

54:59.790 --> 55:02.820
geotopological transform to turn the

55:02.820 --> 55:05.300
RDM into what we call a

55:05.300 --> 55:07.770
representational geotopological matrix

55:07.770 --> 55:11.290
or RDTM . And when we do this ,

55:12.280 --> 55:15.530
As you can see , um , this intersection

55:15.530 --> 55:17.879
point here is reflected in these eyes

55:17.879 --> 55:19.990
in the middle of the upper triangular

55:19.990 --> 55:23.399
and lower triangular , um , part of the

55:23.399 --> 55:26.040
matrix . So you have these , these eyes

55:26.040 --> 55:28.207
here in the middle , and the first one

55:28.207 --> 55:30.262
and in the second one making the the

55:30.262 --> 55:34.040
the left tube . Similar because they

55:34.040 --> 55:37.629
have this this intersection here . And

55:37.629 --> 55:41.330
in the right to scenarios , you don't

55:41.330 --> 55:44.840
get the these eyes in the matrix

55:44.840 --> 55:47.007
because you don't have this connection

55:47.007 --> 55:50.260
here , right ? So once you apply this

55:50.800 --> 55:54.659
geoopological transform , you have a

55:54.659 --> 55:56.826
representation in the form of a square

55:56.826 --> 55:58.899
matrix that better reflects the

55:58.899 --> 56:01.209
topological similarity here .

56:03.580 --> 56:07.000
And this uh matrix characterizes ,

56:07.370 --> 56:11.370
um , The connectivity structure

56:11.370 --> 56:14.199
among neighboring stimuli here because

56:14.570 --> 56:17.810
um we don't care about the uh

56:17.810 --> 56:21.409
exact variation among large distances

56:21.409 --> 56:24.290
anymore . We care only about what's

56:24.290 --> 56:26.457
about the neighborhood relationships ,

56:26.457 --> 56:28.401
what's connected to what along the

56:28.401 --> 56:31.370
manifold , and that's the definition of

56:31.370 --> 56:34.659
the topology . And we can also go a

56:34.659 --> 56:38.260
step further and replace those um .

56:39.389 --> 56:41.439
Nonlinearly transformed

56:41.889 --> 56:45.010
representational distances by geodesics

56:45.010 --> 56:47.850
where we're looking in this graph that

56:47.850 --> 56:50.530
I defined in the previous slide for the

56:50.530 --> 56:53.409
shortest paths between any two stimuli .

56:54.780 --> 56:56.836
And then measure the length of those

56:56.836 --> 56:59.360
paths , and that's what the the lower

56:59.360 --> 57:03.179
row of matrices shows here . So you

57:03.179 --> 57:06.219
can see intuitively how um these

57:06.219 --> 57:09.300
geotopological matrices and then even

57:09.300 --> 57:11.620
more strongly the representational

57:11.620 --> 57:15.169
geodesics matrices reflect the topology

57:15.169 --> 57:18.659
rendering the left to uh cases similar .

57:18.870 --> 57:21.139
The right two cases similar , but the

57:21.139 --> 57:23.459
left two very different from the right

57:23.459 --> 57:26.300
two , whereas when we just look at the

57:26.300 --> 57:28.244
representational distance matrix ,

57:28.340 --> 57:30.500
we're characterizing the geometry and

57:30.500 --> 57:32.860
so the 1st and the 3rd are similar and

57:32.860 --> 57:34.916
the second and the 4th are similar .

57:37.530 --> 57:39.197
Are there questions on this ?

57:47.419 --> 57:51.199
So when we just quantify this with uh

57:51.679 --> 57:54.159
multi-dimensional scaling where we

57:54.520 --> 57:57.989
embed these four matrices in the space

57:58.239 --> 58:01.120
such that the distances in 2D reflect

58:01.120 --> 58:03.360
the similarities between these four

58:03.360 --> 58:06.540
matrices , we get a grouping according

58:06.540 --> 58:09.199
to geometry for the RDM and then for

58:09.199 --> 58:12.669
the RDTM the topology plays a greater

58:12.669 --> 58:15.959
role here and then for the RGDM . Um ,

58:16.030 --> 58:18.679
the , the most prominent factor here is

58:18.679 --> 58:21.620
the topology , so it groups the two

58:22.070 --> 58:25.469
black points together here and those um

58:25.469 --> 58:28.590
correspond to the untangled um .

58:29.709 --> 58:32.550
Flat 8 and bent 8 , so without the

58:32.550 --> 58:36.389
crossing . And the two gray points

58:36.389 --> 58:38.830
here are separate and they correspond

58:39.639 --> 58:42.080
to the flat band aid with the crossing .

58:49.120 --> 58:52.739
We applied this to Human

58:52.739 --> 58:55.620
FRI data where we have um .

58:57.159 --> 59:00.770
Different visual regions . Measured

59:01.040 --> 59:04.959
for different uh visual stimuli . And

59:04.959 --> 59:07.719
we looked at how well we're able with

59:07.719 --> 59:10.709
different geoopological

59:11.280 --> 59:14.679
matrices to identify which

59:14.679 --> 59:17.989
region an RDM from a new subject .

59:20.080 --> 59:23.300
Came from . So I give you and so it's

59:23.300 --> 59:26.520
basically you have , you have a uh A

59:26.520 --> 59:29.439
number of subjects you can estimate the

59:29.439 --> 59:32.159
RDMs for different visual regions in

59:32.159 --> 59:35.419
all of the subjects . Now I give you an

59:35.419 --> 59:39.110
RDM from a new subject and you are

59:39.110 --> 59:41.443
supposed to tell me what region that is ,

59:41.520 --> 59:43.760
right ? This is a very important

59:43.760 --> 59:45.871
problem because we want to understand

59:45.871 --> 59:47.982
the computational differences between

59:47.982 --> 59:50.204
different regions . So we're interested

59:50.204 --> 59:53.040
in using summary statistics such as

59:53.040 --> 59:56.669
RDMs or GDTMs that Uh ,

59:56.709 --> 59:59.669
are good at showing us the

59:59.669 --> 01:00:01.629
computational differences between

01:00:01.629 --> 01:00:05.229
different brain regions and so if we

01:00:05.229 --> 01:00:07.810
had a very useful summary statistic

01:00:08.110 --> 01:00:11.550
then we should be able um to

01:00:11.550 --> 01:00:15.100
identify which visual region the data

01:00:15.590 --> 01:00:17.929
from the new subject came from right

01:00:17.929 --> 01:00:21.469
based on comparing its RDM to the RDMs

01:00:21.469 --> 01:00:23.580
from the other subjects where we know

01:00:23.580 --> 01:00:26.419
which regions they came from . And we

01:00:26.419 --> 01:00:29.020
can do this for all the different

01:00:29.020 --> 01:00:32.870
summary statistics uh in this family of

01:00:32.870 --> 01:00:36.139
geo topological descriptors including

01:00:36.139 --> 01:00:38.969
the RDM in the upper left corner here ,

01:00:39.300 --> 01:00:42.820
and then the hard threshold of matrices

01:00:42.820 --> 01:00:44.979
along the diagonal here and then all

01:00:44.979 --> 01:00:47.979
the other um piecewise linearly

01:00:47.979 --> 01:00:50.620
transformed um versions of the RDM .

01:00:51.370 --> 01:00:53.439
And when we do that , what we see is

01:00:53.439 --> 01:00:56.320
that the best performing summary

01:00:56.320 --> 01:00:58.760
statistic is something in the middle

01:00:58.760 --> 01:01:02.580
here . So this is not the RDM ,

01:01:02.659 --> 01:01:05.979
but it's applying um a combination of

01:01:05.979 --> 01:01:09.229
two thresholds LNU that uh

01:01:10.070 --> 01:01:12.370
Define a piecewise linear transform of

01:01:12.370 --> 01:01:16.260
the RDM . And uh so from just

01:01:16.919 --> 01:01:19.919
Um , the descriptive result here , it

01:01:19.919 --> 01:01:22.830
would seem as though by doing this

01:01:23.320 --> 01:01:25.487
piecewise linear transform where we're

01:01:25.487 --> 01:01:27.209
compressing all the very small

01:01:27.209 --> 01:01:29.431
distances and we're compressing all the

01:01:29.431 --> 01:01:31.629
very large distances , perhaps we're

01:01:31.629 --> 01:01:34.239
reducing the noise and the nuisance

01:01:34.239 --> 01:01:37.600
variability more than we're uh reducing

01:01:37.600 --> 01:01:40.429
the signal and therefore we can then

01:01:40.679 --> 01:01:44.080
distinguish better which region , um .

01:01:45.129 --> 01:01:48.310
The data came from in the new subject .

01:01:50.699 --> 01:01:52.810
However , it's important to note that

01:01:52.810 --> 01:01:55.389
um But we do

01:01:56.780 --> 01:02:00.719
Comparisons here . The geometry , uh ,

01:02:00.729 --> 01:02:04.010
sensitive parts of this , this family ,

01:02:04.199 --> 01:02:07.620
so that's around the RDM perform as

01:02:07.620 --> 01:02:09.787
well as all of the , the other parts .

01:02:09.820 --> 01:02:12.850
So this method does not perform

01:02:12.850 --> 01:02:15.739
significantly better than um using the

01:02:15.739 --> 01:02:19.479
full RDM . But it reduces

01:02:19.479 --> 01:02:21.919
the information quite substantially and

01:02:21.919 --> 01:02:25.120
it can perform equally well in

01:02:25.120 --> 01:02:27.176
different parts of this matrix , the

01:02:27.176 --> 01:02:29.479
dark parts of this matrix or the the

01:02:29.479 --> 01:02:31.989
parts where where it performs very well .

01:02:35.820 --> 01:02:39.040
So here the the shade um

01:02:39.040 --> 01:02:41.580
encodes the brain region identification

01:02:41.580 --> 01:02:44.600
accuracy and the dark parts here are

01:02:44.600 --> 01:02:47.639
the parts where a member of this family

01:02:47.639 --> 01:02:50.590
of summary statistics allows you to

01:02:50.590 --> 01:02:53.679
identify which region the data came

01:02:53.679 --> 01:02:56.679
from in the new subject um very

01:02:56.679 --> 01:03:00.659
accurately . We've also

01:03:00.659 --> 01:03:03.570
done this for layers of neural networks

01:03:03.570 --> 01:03:05.570
where the goal is to identify which

01:03:05.570 --> 01:03:07.949
layer of the neural network generated

01:03:07.949 --> 01:03:11.620
the data . Based on data from other

01:03:11.620 --> 01:03:13.842
instances trained from different random

01:03:13.842 --> 01:03:15.739
seeds of the same neural network

01:03:15.739 --> 01:03:19.139
architecture , so as for people , we

01:03:19.139 --> 01:03:21.379
have different individual neural

01:03:21.379 --> 01:03:23.580
networks here , um , we have a given

01:03:23.580 --> 01:03:27.360
architecture , um , we initialize it

01:03:27.360 --> 01:03:29.754
with . Different random seats and train

01:03:29.754 --> 01:03:32.074
it in each case and you get a trained

01:03:32.074 --> 01:03:34.745
model that does the same visual

01:03:34.745 --> 01:03:37.524
classification task about equally well ,

01:03:37.564 --> 01:03:39.564
but all the weights are different ,

01:03:39.564 --> 01:03:42.064
right ? And now we want to know , um ,

01:03:42.314 --> 01:03:44.475
here's an RDM from layer 3 , but I'm

01:03:44.475 --> 01:03:47.389
not telling you it's layer 3 . Um ,

01:03:47.719 --> 01:03:49.879
compare it to the RDMs from the other

01:03:49.879 --> 01:03:53.110
instances of that model , uh , class ,

01:03:53.199 --> 01:03:55.750
and tell me which layer it came from ,

01:03:55.959 --> 01:03:57.792
right ? If we had a good summary

01:03:57.792 --> 01:03:59.959
statistic , a good characterization of

01:03:59.959 --> 01:04:02.669
the region , we should be doing well at ,

01:04:02.679 --> 01:04:06.300
at doing this task . And

01:04:06.310 --> 01:04:09.030
um here you see that this , this works

01:04:09.030 --> 01:04:12.100
well when you apply appropriate um

01:04:12.350 --> 01:04:14.350
lower and upper thresholds and form

01:04:14.350 --> 01:04:18.270
this linear um . The piecewise linear

01:04:19.060 --> 01:04:22.750
transform on the RDMs . And here is

01:04:22.750 --> 01:04:25.830
a here are multi-dimensional scaling

01:04:25.830 --> 01:04:29.679
arrangements , um . Of the different

01:04:29.679 --> 01:04:32.850
instances um for each of the different

01:04:32.850 --> 01:04:34.919
layers of the networks , so we have

01:04:34.919 --> 01:04:36.808
different colors here , different

01:04:36.808 --> 01:04:38.863
layers layer 1 , layer 2 , layer 3 ,

01:04:38.863 --> 01:04:41.689
layer 4 , and , uh , each dot here is a

01:04:41.689 --> 01:04:44.489
different instance of the , the mobile

01:04:44.489 --> 01:04:47.250
architecture and you can see that when

01:04:47.250 --> 01:04:49.800
we use the representational

01:04:49.800 --> 01:04:53.330
geoopological matrices , we get this

01:04:53.510 --> 01:04:55.649
really nice separation of these

01:04:55.649 --> 01:04:59.159
different layers . And when we use the

01:04:59.159 --> 01:05:02.229
RDM naively without applying any

01:05:02.830 --> 01:05:06.199
nonlinear uh transform to it , um , we

01:05:06.199 --> 01:05:10.110
get more of a Or a tangle where

01:05:10.110 --> 01:05:12.166
um layer 10 is very separate because

01:05:12.166 --> 01:05:14.409
the the last layer that's very , very

01:05:14.409 --> 01:05:16.520
different from all the other layers ,

01:05:16.790 --> 01:05:18.846
but then um some of the other layers

01:05:18.846 --> 01:05:21.350
are somewhat more , more confusible .

01:05:26.689 --> 01:05:30.399
So Let's zoom out a little bit and look

01:05:30.399 --> 01:05:33.929
at what the toolbox offers uh moving

01:05:33.929 --> 01:05:37.000
toward the the end here . Um , so we ,

01:05:37.290 --> 01:05:39.250
we have many different uh distance

01:05:39.250 --> 01:05:42.090
estimators including biased estimators

01:05:42.090 --> 01:05:44.010
and unbiased estimators .

01:05:46.989 --> 01:05:50.010
And some of these are especially

01:05:50.010 --> 01:05:52.520
developed for neural recording data .

01:05:52.610 --> 01:05:54.388
We're very interested in neural

01:05:54.388 --> 01:05:56.489
recording data and supporting them

01:05:56.489 --> 01:06:00.070
fully , including um distances that are

01:06:01.199 --> 01:06:03.520
Especially appropriate for for neo

01:06:03.520 --> 01:06:06.290
recordings such as the croissant KL

01:06:06.639 --> 01:06:08.472
distance estimator and the cross

01:06:08.472 --> 01:06:10.583
validated version of the croissant KL

01:06:10.780 --> 01:06:14.340
estimator . And then we have different

01:06:14.340 --> 01:06:17.129
choices for the RDM comparator

01:06:17.129 --> 01:06:19.459
including cosine similarity , the

01:06:19.459 --> 01:06:22.489
Pearson correlation coefficient , uh ,

01:06:22.739 --> 01:06:25.139
different rank correlation coefficients ,

01:06:25.320 --> 01:06:28.219
topology-based comparators , widened

01:06:28.219 --> 01:06:29.739
RDM comparators .

01:06:32.610 --> 01:06:34.832
So it's good to have many options , but

01:06:34.832 --> 01:06:37.250
it's also confusing and it is important

01:06:37.250 --> 01:06:40.979
um to have a rationale for choosing the

01:06:40.979 --> 01:06:44.219
right option . Uh , fortunately , there

01:06:44.219 --> 01:06:47.959
is , uh , Quite

01:06:47.959 --> 01:06:50.270
a straightforward way of determining

01:06:50.800 --> 01:06:54.239
which combination of RDM estimator and

01:06:54.239 --> 01:06:57.360
RDM comparator is appropriate for a

01:06:57.360 --> 01:07:01.120
particular study . And it's based on

01:07:01.120 --> 01:07:04.469
asking 3 questions about

01:07:04.600 --> 01:07:07.159
your uh experiment .

01:07:10.169 --> 01:07:12.580
The first one is , do the models make

01:07:12.580 --> 01:07:15.939
ratio scale dissimilarity predictions ?

01:07:16.500 --> 01:07:19.520
So , You might have neural network

01:07:19.520 --> 01:07:21.687
models and you might want to interpret

01:07:21.687 --> 01:07:24.959
them as telling you , uh , on a ratio

01:07:24.959 --> 01:07:27.237
scale what the dissimilarity should be .

01:07:27.239 --> 01:07:29.406
So for example , if the model says two

01:07:29.406 --> 01:07:32.080
stimuli are indistinguishable , the

01:07:32.080 --> 01:07:34.302
dissimilarity is zero , and you want to

01:07:34.302 --> 01:07:36.959
take that seriously as a prediction for

01:07:36.959 --> 01:07:39.239
what should be the case in your brain

01:07:39.239 --> 01:07:41.295
region . Your brain region should be

01:07:41.295 --> 01:07:43.295
invariant to the difference between

01:07:43.295 --> 01:07:46.120
those , those two stimuli that . Uh , a

01:07:46.120 --> 01:07:48.800
weaker scenario is where you want to

01:07:48.800 --> 01:07:51.840
say , well , I don't want to take my

01:07:51.840 --> 01:07:54.007
model quite so seriously . I only want

01:07:54.007 --> 01:07:56.320
to predict , for example , the rank

01:07:56.320 --> 01:07:59.040
order of the dissimilarities . So I

01:07:59.040 --> 01:08:01.439
want the model says these two should be

01:08:01.439 --> 01:08:03.389
more similar than these other two

01:08:03.389 --> 01:08:06.159
stimuli . I want that to hold in the

01:08:06.159 --> 01:08:09.320
data , but I don't need the

01:08:09.320 --> 01:08:12.580
interpretable zero point or um

01:08:12.840 --> 01:08:16.069
or the . Exact magnitude of those

01:08:16.069 --> 01:08:19.220
dissimilarities to . To give me an

01:08:19.220 --> 01:08:23.029
accurate prediction . Right , so if the

01:08:23.029 --> 01:08:25.029
answer to this question , do models

01:08:25.029 --> 01:08:26.807
make ratio scale the similarity

01:08:26.807 --> 01:08:28.918
predictions is yes , we want to use a

01:08:28.918 --> 01:08:32.220
ratio scale comparative for the RDMs ,

01:08:32.509 --> 01:08:34.731
and if it's no , then we want to use an

01:08:34.731 --> 01:08:37.910
ordinal scale or topology based um

01:08:37.910 --> 01:08:40.970
comparative . The second question is ,

01:08:41.229 --> 01:08:43.396
are there multiple independent pattern

01:08:43.396 --> 01:08:45.790
estimates from repetitions in each

01:08:45.790 --> 01:08:48.250
condition ? This is just uh useful to

01:08:48.250 --> 01:08:51.709
have if we can repeat the

01:08:51.709 --> 01:08:54.580
stimuli and we have multiple

01:08:54.990 --> 01:08:57.870
independent pattern estimates based on

01:08:57.870 --> 01:09:01.189
repetitions , then , uh , certain

01:09:01.189 --> 01:09:03.300
methods can be applied that cannot be

01:09:03.300 --> 01:09:06.588
applied otherwise . Namely unbiased

01:09:06.588 --> 01:09:08.829
distance estimators that require cross

01:09:08.829 --> 01:09:11.338
validation . So if the answer is yes ,

01:09:11.668 --> 01:09:13.890
then we have the option to use unbiased

01:09:13.890 --> 01:09:16.778
distance estimators . Otherwise we must

01:09:16.778 --> 01:09:19.269
use biased distance estimators because

01:09:19.269 --> 01:09:21.269
we don't have the data to cross

01:09:21.269 --> 01:09:25.229
validate . And the final question

01:09:25.229 --> 01:09:27.838
is , are the errors of the pattern

01:09:27.838 --> 01:09:29.894
estimates for different conditions .

01:09:30.589 --> 01:09:33.240
Independent and identically distributed

01:09:33.240 --> 01:09:36.869
within each partition . And if

01:09:36.869 --> 01:09:40.219
the answer to this one is yes , then

01:09:43.240 --> 01:09:45.580
It means that if we use a biased

01:09:45.580 --> 01:09:47.802
distance estimator , we have the bias ,

01:09:47.802 --> 01:09:50.024
but at least the ranks of the distances

01:09:50.024 --> 01:09:52.136
are still going to be interpretable .

01:09:52.479 --> 01:09:55.419
If the answer is no , then we really

01:09:55.419 --> 01:09:57.890
need an unbiased distance estimator

01:09:58.100 --> 01:10:00.580
even just to get the ranks right among

01:10:00.580 --> 01:10:02.819
the dissimilarities , and this has to

01:10:02.819 --> 01:10:06.220
be kept in mind in our choice of RDM

01:10:06.220 --> 01:10:09.140
estimator and RDM comparator .

01:10:10.080 --> 01:10:12.250
If the answers to all three questions

01:10:12.250 --> 01:10:16.060
are no , then we um suggest

01:10:16.060 --> 01:10:18.759
that you change the experimental design ,

01:10:19.250 --> 01:10:21.370
um , then you're sort of out of , out

01:10:21.370 --> 01:10:23.481
of good options , right ? So it's not

01:10:23.481 --> 01:10:26.330
gonna be a very compelling conclusion .

01:10:27.310 --> 01:10:29.532
So , and then , but for for each of the

01:10:29.532 --> 01:10:32.839
other cases , there is a combination of

01:10:32.839 --> 01:10:36.640
RDM estimator shown in italics here and

01:10:36.640 --> 01:10:39.859
RDM comparator shown in bold here that

01:10:39.859 --> 01:10:41.915
is an appropriate choice that allows

01:10:41.915 --> 01:10:44.200
you to make draw good conclusions and

01:10:44.200 --> 01:10:45.311
model comparisons .

01:10:47.970 --> 01:10:48.899
An important

01:10:51.589 --> 01:10:54.910
Ability to have in these analysis is to

01:10:56.350 --> 01:10:59.390
Allows some flexibility for the RDM

01:10:59.669 --> 01:11:02.459
prediction . So for example , sometimes

01:11:02.790 --> 01:11:06.680
we may have multiple . Sets

01:11:06.680 --> 01:11:09.120
of features like the different feature

01:11:09.120 --> 01:11:11.453
maps of the layer in the neural network ,

01:11:11.453 --> 01:11:13.890
and we may not have a strong prediction

01:11:14.279 --> 01:11:17.279
about their relative prevalence in the

01:11:17.279 --> 01:11:19.899
neural population . So for example , we

01:11:19.899 --> 01:11:23.379
might think that certain features

01:11:23.379 --> 01:11:26.529
um are present in the neural population ,

01:11:26.569 --> 01:11:28.625
but we don't know how prevalent they

01:11:28.625 --> 01:11:30.680
are in the population and if you had

01:11:30.680 --> 01:11:32.847
more of a certain kind of tuning curve

01:11:32.847 --> 01:11:36.169
in the population that would change the

01:11:36.169 --> 01:11:38.058
representation of geometry that's

01:11:38.058 --> 01:11:40.169
predicted , right ? So it's important

01:11:40.169 --> 01:11:43.479
to be able to use um parameters

01:11:43.850 --> 01:11:47.810
in uh modeling RDMs . So this

01:11:47.810 --> 01:11:50.850
means we want flexible RDM models where

01:11:50.850 --> 01:11:54.399
the data RDM is explained . Um ,

01:11:54.580 --> 01:11:57.459
using a number of parameters , and

01:11:57.459 --> 01:11:59.570
there are different kinds of flexible

01:11:59.570 --> 01:12:01.970
RDM models that our toolbox , uh ,

01:12:01.979 --> 01:12:05.129
supports . The simplest is perhaps , um ,

01:12:05.620 --> 01:12:08.479
a selection model where you uh just

01:12:08.479 --> 01:12:11.490
choose among a set of predicted RDMs .

01:12:11.870 --> 01:12:13.870
Another one is a weighted component

01:12:13.870 --> 01:12:15.939
model where you have , uh , in this

01:12:15.939 --> 01:12:19.629
case 3 . RDMs , RDM

01:12:19.629 --> 01:12:21.950
components , and you can predict any

01:12:21.950 --> 01:12:23.950
convex combination of these three .

01:12:25.649 --> 01:12:28.160
Another one is an RDM manifold model

01:12:28.160 --> 01:12:30.200
where the components are in a

01:12:30.200 --> 01:12:32.759
particular order and in this case it's

01:12:32.759 --> 01:12:35.359
a one dimensional manifold and in the

01:12:35.359 --> 01:12:37.919
lower right you see an RDM manifold

01:12:37.919 --> 01:12:40.660
model where there are two dimensions to

01:12:40.660 --> 01:12:42.649
the RDM manifold that the model

01:12:43.359 --> 01:12:46.140
predicts . So all of these can be

01:12:46.140 --> 01:12:48.660
handled by the toolbox in case your

01:12:48.660 --> 01:12:52.259
theory isn't so specific as to

01:12:52.259 --> 01:12:55.379
predict a single fixed RDM , but there

01:12:55.379 --> 01:12:58.379
is some uncertainty about the exact RDM

01:12:58.379 --> 01:13:00.819
that your your theory predicts .

01:13:03.299 --> 01:13:05.859
This flexibility , so if there is no

01:13:05.859 --> 01:13:08.620
flexibility , then the inference is

01:13:08.620 --> 01:13:11.729
particularly easy , but when you do fit ,

01:13:11.979 --> 01:13:14.779
then uh this makes it a little bit more

01:13:14.779 --> 01:13:17.459
complicated to do the model comparisons

01:13:17.459 --> 01:13:20.020
correctly because whenever you fit ,

01:13:20.259 --> 01:13:22.709
you overfit to the data to some extent ,

01:13:22.859 --> 01:13:26.100
and that overfitting can cause

01:13:27.959 --> 01:13:30.720
an optimistic bias to the performance

01:13:30.720 --> 01:13:32.998
of your model , right ? So for example ,

01:13:32.998 --> 01:13:35.220
if you have one model that has a lot of

01:13:35.220 --> 01:13:37.220
parameters , you have another model

01:13:37.220 --> 01:13:39.331
that has no parameters , and the more

01:13:39.331 --> 01:13:41.387
flexible model that's been fitted to

01:13:41.387 --> 01:13:43.553
the data is going to explain the , the

01:13:43.553 --> 01:13:47.080
data that it's been fitted to um better

01:13:47.080 --> 01:13:49.200
in many cases , not necessarily , but

01:13:49.200 --> 01:13:51.560
it , it's in a better position to fit

01:13:51.919 --> 01:13:54.009
the data it's been fitted to better

01:13:54.009 --> 01:13:56.120
because it's been overfitted to those

01:13:56.120 --> 01:13:59.529
data . And in order to compare

01:14:00.640 --> 01:14:03.140
different models fairly , especially

01:14:03.140 --> 01:14:04.973
when those models have different

01:14:04.973 --> 01:14:07.084
numbers of parameters or some of them

01:14:07.084 --> 01:14:09.251
have parameters and the others have no

01:14:09.251 --> 01:14:12.020
parameters , we need to , uh , fit and

01:14:12.020 --> 01:14:14.131
test and cross validation where we're

01:14:14.131 --> 01:14:15.819
using different data for the

01:14:15.819 --> 01:14:18.152
performance evaluation of of each model .

01:14:19.250 --> 01:14:22.049
Unfortunately , this uh slide is a bit

01:14:22.049 --> 01:14:25.850
broken here , um . But the way this

01:14:25.850 --> 01:14:28.859
works is that we designate a set of

01:14:28.859 --> 01:14:32.169
fitting subjects and a set of fitting

01:14:32.500 --> 01:14:36.100
um stimuli shown in blue here . So we

01:14:36.100 --> 01:14:39.140
have a stack of RDMs here and we have

01:14:39.140 --> 01:14:42.500
uh uh one RDM for each subject , and

01:14:42.500 --> 01:14:44.540
then we designate a set of fitting

01:14:44.540 --> 01:14:48.379
stimuli and a set of test stimuli , and

01:14:48.379 --> 01:14:51.890
we also designate uh Set of fitting

01:14:51.890 --> 01:14:55.250
subjects and a set of test subjects .

01:14:56.120 --> 01:14:59.680
We then fit all the models to

01:15:00.779 --> 01:15:03.660
Using only the fitting stimuli and only

01:15:03.660 --> 01:15:06.890
the fitting subjects . And we then

01:15:06.890 --> 01:15:09.540
attempt to predict the dissimilarities

01:15:09.870 --> 01:15:12.310
in the test subjects only for the test

01:15:12.310 --> 01:15:14.430
stimuli , so simultaneously

01:15:14.910 --> 01:15:17.750
generalizing across both subjects and

01:15:17.750 --> 01:15:20.189
stimuli . So that's a very hard

01:15:20.189 --> 01:15:23.410
generalization to require of the models ,

01:15:23.430 --> 01:15:26.589
and this gives us uh unbiased estimates

01:15:26.589 --> 01:15:28.589
of the performance of the different

01:15:28.589 --> 01:15:31.470
models . And then around this cross

01:15:31.470 --> 01:15:34.870
validated . Uh , procedure for

01:15:34.870 --> 01:15:36.814
estimating the performance of each

01:15:36.814 --> 01:15:39.100
model , we wrap a bootstrapping

01:15:39.100 --> 01:15:41.180
procedure that's also a two-factor

01:15:41.180 --> 01:15:44.069
bootstrap where we bootstrap both the

01:15:44.069 --> 01:15:46.870
subjects and the stimuli with a view to

01:15:46.870 --> 01:15:49.470
generalizing across both subjects and

01:15:49.470 --> 01:15:53.390
stimuli with our our inferences . So we

01:15:53.390 --> 01:15:56.029
have implemented in the toolbox a model

01:15:56.029 --> 01:15:58.029
comparative inference strategy that

01:15:58.029 --> 01:16:01.200
uses two-factor cross validation . To

01:16:01.200 --> 01:16:03.430
avoid overfitting to their subjects or

01:16:03.430 --> 01:16:07.169
stimuli . Uh , inside a two-factor

01:16:07.169 --> 01:16:10.919
bootstrap . Loop which

01:16:10.919 --> 01:16:13.720
treats both subjects and stimuli as

01:16:13.720 --> 01:16:16.870
random effects . So this ensures that

01:16:17.120 --> 01:16:19.160
when we say that two models are

01:16:19.160 --> 01:16:21.720
significantly different , we expect

01:16:21.720 --> 01:16:24.160
those differences to hold up when

01:16:24.160 --> 01:16:26.160
someone else tries to replicate our

01:16:26.160 --> 01:16:28.770
experiment using different subjects and

01:16:28.770 --> 01:16:32.330
different stimuli . And so that those

01:16:32.330 --> 01:16:34.649
are the kinds of results that we we

01:16:34.649 --> 01:16:37.410
wanna care about results that don't

01:16:37.410 --> 01:16:41.040
only hold for our subjects but hold for

01:16:41.330 --> 01:16:44.609
um for people in general or for at

01:16:44.609 --> 01:16:47.040
least for people sampled from the same

01:16:47.609 --> 01:16:51.089
population that our sample of subjects

01:16:51.089 --> 01:16:53.649
can be considered a random sample of .

01:16:56.180 --> 01:16:58.180
And then we can do model evaluation

01:16:58.180 --> 01:17:00.124
where we get a bar for each of the

01:17:00.124 --> 01:17:03.669
models and we get um these uh

01:17:03.959 --> 01:17:06.070
indications here at the top that show

01:17:06.070 --> 01:17:09.080
us for each model which other models it

01:17:09.080 --> 01:17:11.359
significantly dominates . So we can say

01:17:11.359 --> 01:17:13.600
this model is significantly better than

01:17:13.600 --> 01:17:16.160
these other models , and we also get a

01:17:16.160 --> 01:17:18.879
noise ceiling which shows us the range

01:17:18.879 --> 01:17:21.339
within which we would expect the the

01:17:21.339 --> 01:17:23.399
true model to fall if the two model

01:17:23.399 --> 01:17:25.600
were among the models we're testing .

01:17:26.390 --> 01:17:29.790
We get these dew drops here uh at the

01:17:29.790 --> 01:17:31.734
lower bound of the noise ceiling ,

01:17:31.734 --> 01:17:33.859
which indicate which models are

01:17:33.859 --> 01:17:36.120
significantly below the lower bound of

01:17:36.120 --> 01:17:38.342
the noise ceiling . So those are models

01:17:38.342 --> 01:17:41.470
that we can reject as not fully

01:17:41.470 --> 01:17:44.290
explaining our data . And then there

01:17:44.290 --> 01:17:46.359
are cases like this one here ,

01:17:46.370 --> 01:17:49.790
conclusional . Layer 2 in this case is

01:17:49.790 --> 01:17:52.950
the , the two model because these data

01:17:52.950 --> 01:17:56.189
are simulated using um this neural

01:17:56.189 --> 01:17:58.709
network and simulated from

01:17:58.709 --> 01:18:01.629
convolutional layer 2 . And

01:18:01.959 --> 01:18:04.479
convolutional layer two , while it's

01:18:04.479 --> 01:18:06.870
not in the noise ceiling , it comes

01:18:06.870 --> 01:18:08.981
close to the lower bound of the noise

01:18:08.981 --> 01:18:11.399
ceiling , and it's evaporated this

01:18:11.399 --> 01:18:13.600
little dew drop here , so it's not

01:18:13.600 --> 01:18:16.319
significantly below the lower bound of

01:18:16.319 --> 01:18:18.152
the noise ceiling in this case .

01:18:19.799 --> 01:18:22.459
Another way to visualize all the

01:18:22.459 --> 01:18:24.459
information in the bar graph on the

01:18:24.459 --> 01:18:27.430
left and some additional information is

01:18:27.430 --> 01:18:30.899
in what we call the model map . So here ,

01:18:31.149 --> 01:18:33.540
um , this is an inversion where we use

01:18:33.540 --> 01:18:37.450
uh We , we plot the data RDM

01:18:37.450 --> 01:18:39.850
at the center of this diagram and then

01:18:39.850 --> 01:18:43.100
we arrange all the model predicted RDMs

01:18:43.100 --> 01:18:46.569
around this uh central data

01:18:46.569 --> 01:18:49.850
RDM such that the distance to the data

01:18:49.850 --> 01:18:53.330
RDM exactly reflects the performance of

01:18:53.330 --> 01:18:57.089
each model . Um ,

01:18:57.330 --> 01:18:59.649
so a larger distance indicates that the

01:18:59.649 --> 01:19:02.049
predicted RDM is further away , so that

01:19:02.049 --> 01:19:05.729
corresponds to a smaller bar on the

01:19:05.729 --> 01:19:07.618
left side here . So it's a little

01:19:07.618 --> 01:19:09.785
counterintuitive . So here being close

01:19:09.785 --> 01:19:12.490
to the data RDM means , uh , you're

01:19:12.490 --> 01:19:15.560
performing well and convolutional layer

01:19:15.560 --> 01:19:17.560
2 is the best performing model , so

01:19:17.560 --> 01:19:20.169
it's closest and by convention plotted

01:19:20.169 --> 01:19:23.500
directly above the data RDM . And then

01:19:23.810 --> 01:19:26.600
um the other models are on orbits here ,

01:19:26.890 --> 01:19:30.370
um , at different radii from the data

01:19:30.370 --> 01:19:34.000
RDM and those radii exactly reflect

01:19:34.410 --> 01:19:36.521
the performance of each of the models

01:19:36.521 --> 01:19:39.100
without any distortion . And then in

01:19:39.100 --> 01:19:41.267
addition , there's a bit of additional

01:19:41.267 --> 01:19:43.211
information in this diagram that's

01:19:43.211 --> 01:19:45.433
missing on the left , which is that the

01:19:45.433 --> 01:19:48.819
angles at which these uh these other

01:19:48.819 --> 01:19:51.740
models are plotted here are chosen by a

01:19:51.740 --> 01:19:55.560
one dimensional variant of MDS so that

01:19:55.740 --> 01:19:57.779
the models that are most similar in

01:19:57.779 --> 01:20:00.140
their predicted RDMs are closer

01:20:00.140 --> 01:20:02.251
together in this diagram . So you see

01:20:02.251 --> 01:20:04.100
this progression from convention

01:20:04.100 --> 01:20:06.609
convolutional layer 12 through

01:20:06.609 --> 01:20:10.459
2345 . And then to the fully connected

01:20:10.459 --> 01:20:13.779
layers here and that's in in this plot

01:20:13.779 --> 01:20:15.835
here you have the same progression ,

01:20:15.835 --> 01:20:18.057
but here it's by convention . It's just

01:20:18.057 --> 01:20:20.279
because the user said this is the order

01:20:20.279 --> 01:20:22.700
of the models . In this case it

01:20:22.700 --> 01:20:25.339
reflects the fact that convolutional

01:20:25.580 --> 01:20:28.459
layer one has an RDM that's similar to

01:20:28.459 --> 01:20:31.939
convolutional layer two's RDM . And

01:20:31.939 --> 01:20:34.390
convolutional layer 2 has an RDM that's

01:20:34.390 --> 01:20:37.589
similar to convolutional layer 3's RDM

01:20:37.589 --> 01:20:39.990
and so on . And so this arrangement

01:20:39.990 --> 01:20:42.046
tells us something about how similar

01:20:42.046 --> 01:20:44.212
these different mole predictions are .

01:20:48.850 --> 01:20:52.680
So two boxes um on GitHub

01:20:53.189 --> 01:20:55.709
and um there's significant

01:20:55.709 --> 01:20:58.470
documentation in the read the docs so

01:20:58.470 --> 01:21:02.060
it's basically ready to go and it's got

01:21:02.060 --> 01:21:04.870
uh a number of new capabilities that

01:21:04.870 --> 01:21:07.350
I've told you about white RDM

01:21:07.350 --> 01:21:09.990
comparators , model comparative

01:21:09.990 --> 01:21:12.109
inference generalizing to populations

01:21:12.109 --> 01:21:14.500
of subjects and and stimuli ,

01:21:14.879 --> 01:21:17.350
topology-based comparators .

01:21:18.020 --> 01:21:20.290
Dissimilarity estimators from neural

01:21:20.290 --> 01:21:22.810
data which are new in the in the past

01:21:22.810 --> 01:21:25.930
we've used sort of rather naive ways of

01:21:25.930 --> 01:21:28.259
comparing neural response patterns .

01:21:28.270 --> 01:21:31.290
Now we've um started working on this

01:21:31.290 --> 01:21:34.169
more seriously and introduced thisroant

01:21:34.169 --> 01:21:36.799
KL RDM estimator and it's cross

01:21:36.799 --> 01:21:39.890
validated variant . We also have more

01:21:39.890 --> 01:21:42.209
efficient rank-based model evaluation

01:21:42.209 --> 01:21:45.060
which I didn't talk about . And ways of

01:21:45.060 --> 01:21:48.020
refuting models by comparison to the

01:21:48.020 --> 01:21:49.964
lower bound of the noise ceiling .

01:21:50.540 --> 01:21:53.359
Better visualization . As well

01:21:54.750 --> 01:21:56.861
Using the pairwise comparisons in the

01:21:56.861 --> 01:21:59.028
bar graph and the mobile map that I've

01:21:59.028 --> 01:22:02.189
showed you . So in conclusion , um ,

01:22:02.200 --> 01:22:06.160
RSA 3 is a powerful set of new tools

01:22:06.160 --> 01:22:08.720
for deriving theoretical , for driving

01:22:08.720 --> 01:22:10.890
theoretical progress by inferential ,

01:22:11.200 --> 01:22:13.959
uh , comparisons of models when the

01:22:13.959 --> 01:22:15.737
models predict representational

01:22:15.737 --> 01:22:18.770
geometries . Estimated from your

01:22:18.770 --> 01:22:21.819
recording on neuroimaging data . And

01:22:21.819 --> 01:22:25.040
the RSA 3 toolbox is documented and

01:22:25.040 --> 01:22:27.359
ready for use , and I hope , uh , some

01:22:27.359 --> 01:22:30.680
of you will join us in developing the

01:22:30.680 --> 01:22:33.430
toolbox and better methods in general .

01:22:33.720 --> 01:22:36.680
Thank you very much . Yeah , thank you .

01:22:36.790 --> 01:22:39.129
I hope uh our team joins you in that

01:22:39.589 --> 01:22:42.549
charge too , and um I just want to say

01:22:42.549 --> 01:22:44.271
that these techniques that you

01:22:44.271 --> 01:22:46.382
presented just made it such a massive

01:22:46.382 --> 01:22:48.493
impact in the field , you know , I've

01:22:48.493 --> 01:22:50.382
only been in the field for like a

01:22:50.382 --> 01:22:52.160
decade , but um Uh , you know ,

01:22:52.330 --> 01:22:54.552
everyone's using these now , and , um ,

01:22:54.552 --> 01:22:56.774
I don't know , it's just really cool to

01:22:56.774 --> 01:22:56.470
see it developing , so I guess thank

01:22:56.470 --> 01:22:59.939
you for that . So , um . Good to see

01:22:59.939 --> 01:23:02.819
you . Thank you so much . Thank you

01:23:02.819 --> 01:23:04.875
guys so much . I'll see you all next

01:23:04.875 --> 01:23:04.149
week .