Self attention based computation of principal eigenvector under certain conditions

Abstract

This code generates the principal eigenvector using self attention under the conditions:

• The input vectors are positive.
• No input transforms are used.

The self attention based eigenvectors do not have the same magnitude as the principal eigenvector, only the same angle. The angle between the vectors is smaller when softmax isn't used. See code for details.

Algorithm

The iris dataset is used alongside randomly generated datasets. For each dataset the adjacency matrix is computed. The eigenvectors and eigenvalues of the adjacency matrix are computed. To compute self attention, the adjacency matrix and the softmax of the adjacency matrix are multiplied by the input dataset. The cosine similarity is then computed between the eigenvectors and the vectors computed with self attention.

Results

Summary

• Principal eigenvector vs self attention 1st vector (works)
• Principal eigenvector vs self attention 2nd vector (works)
• Non-Principal eigenvector vs self attention 1st vector (doesn't work)

The first entry in each table is for the iris dataset. The rest of the entries are randomly generated datasets.

Principal eigenvector vs self attention 1st vector

eigenvalue	mag eigenvector	mag self attention	cosine similarity (with softmax)
9206.530596	1.000000	95.267962	0.979676
153.035448	1.000000	6.677844	0.967490
146.985750	1.000000	6.469487	0.967032
160.615877	1.000000	6.880530	0.965821
145.946463	1.000000	6.241726	0.963127
165.390938	1.000000	6.646201	0.971719
168.939761	1.000000	6.637303	0.967610
173.455692	1.000000	7.324602	0.964891
164.364278	1.000000	7.093303	0.965082
170.809366	1.000000	6.815885	0.964670
168.725411	1.000000	6.963146	0.971783
162.228956	1.000000	6.471354	0.957326
152.398589	1.000000	6.420290	0.964964
152.788177	1.000000	6.011698	0.961383
159.292100	1.000000	6.641088	0.961978
164.603275	1.000000	6.781595	0.970627
165.112983	1.000000	7.025318	0.963097
165.635110	1.000000	6.455346	0.970197
157.807981	1.000000	6.499539	0.966751
177.225237	1.000000	6.971489	0.970760
158.409995	1.000000	6.420343	0.964178
160.217627	1.000000	6.233390	0.964693
170.408037	1.000000	7.133776	0.963515
168.460888	1.000000	6.594924	0.970950
153.034498	1.000000	6.524873	0.967838
169.224380	1.000000	6.500595	0.966955
174.409270	1.000000	7.013595	0.963185
153.403013	1.000000	6.277223	0.969020
157.568193	1.000000	6.493295	0.972137
155.412338	1.000000	6.163024	0.967136
162.250681	1.000000	6.594340	0.967418
162.858646	1.000000	6.511601	0.969240
148.066191	1.000000	6.250871	0.967192
161.412856	1.000000	6.308377	0.965175
154.030834	1.000000	6.039407	0.961977
174.718474	1.000000	6.969898	0.968564
165.243244	1.000000	6.815741	0.963750
163.703212	1.000000	6.490107	0.965378
168.063493	1.000000	6.475724	0.971855
167.906161	1.000000	6.497066	0.960129
167.534931	1.000000	6.784336	0.965059
160.578051	1.000000	6.815965	0.965940
164.235436	1.000000	6.536982	0.967962
155.008998	1.000000	6.548110	0.962167
168.404630	1.000000	6.042017	0.963351
158.640308	1.000000	6.138811	0.962273
169.846162	1.000000	7.062570	0.963550
166.141267	1.000000	6.706509	0.966519
154.979091	1.000000	6.249225	0.965967
160.257850	1.000000	6.502338	0.962483
161.575667	1.000000	6.861466	0.970643
163.650667	1.000000	7.000538	0.961950
163.903141	1.000000	6.550479	0.965754
163.390138	1.000000	7.016451	0.967132
153.517123	1.000000	6.939118	0.964153
163.341905	1.000000	6.181780	0.967710
163.202602	1.000000	6.498106	0.969162
157.279692	1.000000	6.649323	0.961826
157.167207	1.000000	6.459216	0.967796
164.519616	1.000000	6.337793	0.965765
172.830553	1.000000	6.946143	0.966555
157.045462	1.000000	6.116538	0.958633
158.939444	1.000000	6.169372	0.966601
166.080571	1.000000	6.873749	0.968836
155.330333	1.000000	6.276361	0.965957
157.490737	1.000000	6.500678	0.959955
169.363566	1.000000	6.415871	0.964399
162.087452	1.000000	6.448137	0.963823
166.146630	1.000000	6.409095	0.962624
152.824261	1.000000	6.379470	0.968502
146.829732	1.000000	6.456515	0.964139
168.877339	1.000000	6.621731	0.962982
168.070651	1.000000	6.735037	0.971279
160.946507	1.000000	6.945548	0.966535
167.129891	1.000000	6.698101	0.962313
169.302965	1.000000	6.386418	0.971382
162.711044	1.000000	6.820714	0.961286
171.184705	1.000000	6.479948	0.972740
163.758445	1.000000	6.786087	0.970029
157.228066	1.000000	6.897502	0.965146
168.130405	1.000000	7.153319	0.965005
166.130041	1.000000	6.869857	0.964518
175.154889	1.000000	7.173491	0.967136
160.073787	1.000000	6.545292	0.965389
152.161636	1.000000	6.392094	0.963388
164.625480	1.000000	6.573605	0.958175
159.881383	1.000000	6.868605	0.961161
156.692569	1.000000	6.437777	0.967282
158.534200	1.000000	6.013324	0.966350
179.911712	1.000000	7.174269	0.967939
158.838348	1.000000	5.587373	0.960218
162.954921	1.000000	7.203439	0.964688
161.096447	1.000000	6.860645	0.963565
156.183857	1.000000	6.461804	0.969288
169.187674	1.000000	6.765574	0.968920
151.841936	1.000000	5.958579	0.962246
163.061099	1.000000	6.349641	0.965186
150.880346	1.000000	6.106754	0.966595
161.731835	1.000000	6.297961	0.968711
174.998578	1.000000	6.804727	0.967600
168.526015	1.000000	6.916555	0.972225
162.954771	1.000000	6.502600	0.965421
169.277889	1.000000	6.708884	0.964870
168.228308	1.000000	6.492005	0.967914
159.667011	1.000000	7.356210	0.966465
150.853417	1.000000	6.376264	0.961893
156.269186	1.000000	6.799447	0.961121
173.503051	1.000000	6.649155	0.970533
168.339646	1.000000	6.439181	0.968147
155.678395	1.000000	6.491497	0.961954
159.265431	1.000000	6.530884	0.967143
168.691649	1.000000	6.532615	0.967979
161.585337	1.000000	6.650253	0.966058
165.321779	1.000000	6.535029	0.963632
155.662681	1.000000	6.734863	0.964913
176.757512	1.000000	6.260079	0.968214
177.075374	1.000000	6.768350	0.966107
150.342079	1.000000	5.996217	0.967129
167.786899	1.000000	6.814211	0.962453
158.571917	1.000000	6.793648	0.961036
172.169312	1.000000	7.102100	0.962579
155.700443	1.000000	6.887268	0.964365
155.115318	1.000000	6.317686	0.964678
159.181861	1.000000	6.494668	0.969242
160.698527	1.000000	6.389890	0.967635
165.051598	1.000000	6.528850	0.966169
160.228322	1.000000	6.611481	0.967602
141.442469	1.000000	6.018621	0.959190
161.683492	1.000000	6.363545	0.963881

eigenvalue	mag eigenvector	mag self attention	cosine similarity (without softmax)
9206.530596	1.000000	663563.299286	0.999997
153.035448	1.000000	986.932529	0.999315
146.985750	1.000000	921.390981	0.999603
160.615877	1.000000	1062.004238	0.999347
145.946463	1.000000	881.471524	0.998908
165.390938	1.000000	1060.110611	0.999368
168.939761	1.000000	1076.486190	0.999434
173.455692	1.000000	1217.497038	0.999531
164.364278	1.000000	1122.958085	0.999475
170.809366	1.000000	1117.169061	0.999353
168.725411	1.000000	1130.548081	0.999492
162.228956	1.000000	1004.520971	0.998330
152.398589	1.000000	942.872802	0.999132
152.788177	1.000000	884.139813	0.999482
159.292100	1.000000	1015.351799	0.999181
164.603275	1.000000	1078.569778	0.999247
165.112983	1.000000	1107.108619	0.999607
165.635110	1.000000	1034.053413	0.999077
157.807981	1.000000	989.866195	0.999169
177.225237	1.000000	1191.653017	0.999147
158.409995	1.000000	981.265974	0.998862
160.217627	1.000000	963.461947	0.999021
170.408037	1.000000	1161.864926	0.999457
168.460888	1.000000	1068.981161	0.999337
153.034498	1.000000	961.107531	0.999387
169.224380	1.000000	1055.491698	0.999409
174.409270	1.000000	1169.397747	0.999559
153.403013	1.000000	932.864167	0.999047
157.568193	1.000000	989.713896	0.999436
155.412338	1.000000	923.322647	0.999214
162.250681	1.000000	1028.272934	0.998982
162.858646	1.000000	1026.139767	0.999016
148.066191	1.000000	895.530530	0.998744
161.412856	1.000000	977.204049	0.998893
154.030834	1.000000	897.585165	0.998908
174.718474	1.000000	1166.419430	0.999318
165.243244	1.000000	1079.371510	0.999283
163.703212	1.000000	1019.041048	0.999333
168.063493	1.000000	1054.097491	0.999116
167.906161	1.000000	1042.329824	0.999138
167.534931	1.000000	1088.410514	0.999461
160.578051	1.000000	1054.150942	0.999504
164.235436	1.000000	1032.632653	0.999083
155.008998	1.000000	973.848494	0.999164
168.404630	1.000000	969.162695	0.998880
158.640308	1.000000	932.434099	0.999005
169.846162	1.000000	1142.183379	0.999762
166.141267	1.000000	1068.792570	0.999324
154.979091	1.000000	930.404063	0.999272
160.257850	1.000000	993.183997	0.999284
161.575667	1.000000	1073.839519	0.998952
163.650667	1.000000	1098.034117	0.999714
163.903141	1.000000	1033.038479	0.998705
163.390138	1.000000	1104.663242	0.999437
153.517123	1.000000	1029.581357	0.999417
163.341905	1.000000	972.653563	0.998229
163.202602	1.000000	1023.257487	0.999532
157.279692	1.000000	999.641199	0.999220
157.167207	1.000000	980.416235	0.999180
164.519616	1.000000	1000.514799	0.999327
172.830553	1.000000	1151.895329	0.999575
157.045462	1.000000	915.874324	0.998832
158.939444	1.000000	944.702202	0.998937
166.080571	1.000000	1096.793760	0.999513
155.330333	1.000000	940.219388	0.999416
157.490737	1.000000	979.557709	0.999411
169.363566	1.000000	1040.782782	0.999386
162.087452	1.000000	1001.808777	0.998964
166.146630	1.000000	1021.310957	0.999389
152.824261	1.000000	936.832629	0.998848
146.829732	1.000000	911.813623	0.998692
168.877339	1.000000	1067.384958	0.999622
168.070651	1.000000	1091.169738	0.999083
160.946507	1.000000	1076.632457	0.999656
167.129891	1.000000	1068.878228	0.999293
169.302965	1.000000	1043.349807	0.999226
162.711044	1.000000	1064.869687	0.999423
171.184705	1.000000	1070.166797	0.999216
163.758445	1.000000	1072.393726	0.999408
157.228066	1.000000	1046.396507	0.999302
168.130405	1.000000	1147.423876	0.999691
166.130041	1.000000	1098.639801	0.999291
175.154889	1.000000	1198.024532	0.999477
160.073787	1.000000	1002.511192	0.999387
152.161636	1.000000	940.430782	0.999528
164.625480	1.000000	1032.317496	0.999344
159.881383	1.000000	1054.191665	0.998984
156.692569	1.000000	975.302779	0.999110
158.534200	1.000000	915.931027	0.998167
179.911712	1.000000	1232.841465	0.999751
158.838348	1.000000	848.501567	0.998799
162.954921	1.000000	1126.246139	0.999698
161.096447	1.000000	1062.143340	0.999456
156.183857	1.000000	974.546894	0.999037
169.187674	1.000000	1100.147076	0.999688
151.841936	1.000000	869.090949	0.998766
163.061099	1.000000	995.329331	0.999024
150.880346	1.000000	888.211331	0.999080
161.731835	1.000000	984.603202	0.999145
174.998578	1.000000	1135.187239	0.999652
168.526015	1.000000	1126.843098	0.999610
162.954771	1.000000	1017.808123	0.999334
169.277889	1.000000	1086.866835	0.999141
168.228308	1.000000	1049.004683	0.999157
159.667011	1.000000	1132.786460	0.999613
150.853417	1.000000	924.619091	0.999382
156.269186	1.000000	1018.898690	0.998538
173.503051	1.000000	1105.225452	0.999182
168.339646	1.000000	1042.579505	0.999230
155.678395	1.000000	969.937637	0.999090
159.265431	1.000000	999.033539	0.999304
168.691649	1.000000	1056.922474	0.998963
161.585337	1.000000	1032.916585	0.999591
165.321779	1.000000	1030.084105	0.999210
155.662681	1.000000	1006.992628	0.999525
176.757512	1.000000	1060.100038	0.999358
177.075374	1.000000	1140.121328	0.999639
150.342079	1.000000	870.263814	0.998531
167.786899	1.000000	1089.167103	0.999548
158.571917	1.000000	1027.084912	0.999306
172.169312	1.000000	1164.404668	0.999624
155.700443	1.000000	1032.449923	0.999476
155.115318	1.000000	946.423578	0.999128
159.181861	1.000000	996.326855	0.999385
160.698527	1.000000	987.887217	0.999378
165.051598	1.000000	1034.203451	0.999606
160.228322	1.000000	1025.394188	0.999493
141.442469	1.000000	823.310567	0.999125
161.683492	1.000000	992.011656	0.999337

0/129 outside of cosine similarity of .95 (with softmax)
0/129 outside of cosine similarity of .99 (without softmax)

Principal eigenvector vs self attention 2nd vector

eigenvalue	mag eigenvector	mag self attention	cosine similarity (with softmax)
9206.530596	1.000000	45.431212	0.979031
153.035448	1.000000	6.643875	0.963647
146.985750	1.000000	6.292347	0.964310
160.615877	1.000000	6.846133	0.968209
145.946463	1.000000	5.898011	0.966736
165.390938	1.000000	6.625122	0.969893
168.939761	1.000000	7.179233	0.967478
173.455692	1.000000	6.605861	0.964463
164.364278	1.000000	6.458686	0.961065
170.809366	1.000000	7.232203	0.968815
168.725411	1.000000	6.270382	0.971171
162.228956	1.000000	6.937579	0.963713
152.398589	1.000000	6.186679	0.963130
152.788177	1.000000	6.830993	0.959128
159.292100	1.000000	6.668894	0.962029
164.603275	1.000000	6.983264	0.974073
165.112983	1.000000	6.419021	0.959302
165.635110	1.000000	6.784116	0.968903
157.807981	1.000000	6.684434	0.965812
177.225237	1.000000	6.934166	0.975708
158.409995	1.000000	6.819193	0.965502
160.217627	1.000000	6.761914	0.967685
170.408037	1.000000	6.200721	0.965801
168.460888	1.000000	7.008840	0.969861
153.034498	1.000000	6.745557	0.968670
169.224380	1.000000	6.490785	0.967896
174.409270	1.000000	6.672823	0.962002
153.403013	1.000000	6.425096	0.969844
157.568193	1.000000	6.536778	0.972215
155.412338	1.000000	6.933658	0.968430
162.250681	1.000000	6.753185	0.966531
162.858646	1.000000	6.396478	0.972689
148.066191	1.000000	6.407398	0.966365
161.412856	1.000000	6.865629	0.966142
154.030834	1.000000	6.810135	0.961072
174.718474	1.000000	6.832167	0.969851
165.243244	1.000000	6.556012	0.965164
163.703212	1.000000	6.384995	0.967731
168.063493	1.000000	7.314305	0.975042
167.906161	1.000000	6.461110	0.964899
167.534931	1.000000	7.169000	0.964686
160.578051	1.000000	7.116794	0.965054
164.235436	1.000000	7.070761	0.967734
155.008998	1.000000	6.605111	0.962232
168.404630	1.000000	7.240202	0.968440
158.640308	1.000000	6.097162	0.963838
169.846162	1.000000	6.663648	0.958705
166.141267	1.000000	6.715065	0.967781
154.979091	1.000000	6.587455	0.964150
160.257850	1.000000	6.832932	0.960118
161.575667	1.000000	6.082574	0.970461
163.650667	1.000000	6.827709	0.960974
163.903141	1.000000	6.552121	0.968834
163.390138	1.000000	6.458401	0.966191
153.517123	1.000000	6.355234	0.961995
163.341905	1.000000	6.895637	0.973309
163.202602	1.000000	6.311523	0.967069
157.279692	1.000000	7.232584	0.961731
157.167207	1.000000	6.791197	0.968367
164.519616	1.000000	7.036681	0.966046
172.830553	1.000000	6.814535	0.967312
157.045462	1.000000	6.336946	0.961892
158.939444	1.000000	7.169263	0.966185
166.080571	1.000000	6.496220	0.965227
155.330333	1.000000	6.867713	0.964519
157.490737	1.000000	6.576886	0.962501
169.363566	1.000000	6.950175	0.962902
162.087452	1.000000	6.625318	0.965546
166.146630	1.000000	6.804640	0.959658
152.824261	1.000000	6.207275	0.965771
146.829732	1.000000	6.504319	0.968329
168.877339	1.000000	7.058063	0.961673
168.070651	1.000000	6.978154	0.972926
160.946507	1.000000	6.660465	0.966669
167.129891	1.000000	6.631728	0.963092
169.302965	1.000000	6.802213	0.969989
162.711044	1.000000	6.748184	0.958732
171.184705	1.000000	7.095008	0.972894
163.758445	1.000000	6.422240	0.968471
157.228066	1.000000	5.967757	0.961971
168.130405	1.000000	6.724928	0.963659
166.130041	1.000000	6.266466	0.967390
175.154889	1.000000	7.265977	0.964555
160.073787	1.000000	6.353818	0.963119
152.161636	1.000000	6.705373	0.965946
164.625480	1.000000	7.109392	0.961951
159.881383	1.000000	6.996777	0.962087
156.692569	1.000000	6.410235	0.967659
158.534200	1.000000	6.525644	0.970323
179.911712	1.000000	6.939200	0.965738
158.838348	1.000000	6.907806	0.960267
162.954921	1.000000	6.042676	0.962306
161.096447	1.000000	6.356647	0.967446
156.183857	1.000000	6.673931	0.968644
169.187674	1.000000	6.589015	0.969718
151.841936	1.000000	6.451470	0.965447
163.061099	1.000000	6.504542	0.966156
150.880346	1.000000	6.735311	0.965319
161.731835	1.000000	6.782007	0.968079
174.998578	1.000000	6.914712	0.967057
168.526015	1.000000	6.809653	0.970070
162.954771	1.000000	6.811249	0.961744
169.277889	1.000000	6.571980	0.964043
168.228308	1.000000	7.093452	0.966385
159.667011	1.000000	6.153428	0.962081
150.853417	1.000000	6.358823	0.960819
156.269186	1.000000	6.316291	0.967301
173.503051	1.000000	7.213451	0.969577
168.339646	1.000000	6.544971	0.965318
155.678395	1.000000	6.178977	0.962322
159.265431	1.000000	6.440922	0.966130
168.691649	1.000000	6.662432	0.964060
161.585337	1.000000	6.753415	0.964149
165.321779	1.000000	6.847173	0.960586
155.662681	1.000000	6.400734	0.964781
176.757512	1.000000	6.983299	0.969690
177.075374	1.000000	7.042213	0.961593
150.342079	1.000000	6.357507	0.969735
167.786899	1.000000	6.487679	0.962533
158.571917	1.000000	6.595534	0.961135
172.169312	1.000000	6.960459	0.961366
155.700443	1.000000	6.375258	0.961260
155.115318	1.000000	6.917587	0.965207
159.181861	1.000000	6.612921	0.963192
160.698527	1.000000	7.253825	0.968740
165.051598	1.000000	6.631770	0.968471
160.228322	1.000000	6.754440	0.965282
141.442469	1.000000	6.215643	0.961827
161.683492	1.000000	6.431265	0.967939

eigenvalue	mag eigenvector	mag self attention	cosine similarity (without softmax)
9206.530596	1.000000	335508.409704	0.999959
153.035448	1.000000	980.576950	0.999218
146.985750	1.000000	893.156364	0.999037
160.615877	1.000000	1059.140536	0.999684
145.946463	1.000000	835.240866	0.999063
165.390938	1.000000	1056.013472	0.999341
168.939761	1.000000	1167.695788	0.999439
173.455692	1.000000	1100.162241	0.999209
164.364278	1.000000	1013.285135	0.999220
170.809366	1.000000	1184.056074	0.999533
168.725411	1.000000	1017.714913	0.999324
162.228956	1.000000	1077.815806	0.999588
152.398589	1.000000	905.883228	0.998999
152.788177	1.000000	1000.714900	0.999299
159.292100	1.000000	1016.551882	0.999486
164.603275	1.000000	1110.016882	0.999268
165.112983	1.000000	1007.651938	0.999207
165.635110	1.000000	1087.665259	0.999417
157.807981	1.000000	1015.792448	0.999045
177.225237	1.000000	1183.069916	0.999440
158.409995	1.000000	1041.341783	0.999103
160.217627	1.000000	1045.492795	0.999375
170.408037	1.000000	1006.962115	0.999552
168.460888	1.000000	1139.796276	0.999467
153.034498	1.000000	998.866526	0.999187
169.224380	1.000000	1056.073283	0.999463
174.409270	1.000000	1113.155996	0.999422
153.403013	1.000000	959.709497	0.999186
157.568193	1.000000	994.261380	0.999348
155.412338	1.000000	1039.196926	0.999620
162.250681	1.000000	1052.313715	0.999064
162.858646	1.000000	1009.010381	0.999345
148.066191	1.000000	918.977593	0.999327
161.412856	1.000000	1065.250962	0.999232
154.030834	1.000000	1006.422552	0.999484
174.718474	1.000000	1144.160700	0.999451
165.243244	1.000000	1036.244052	0.999196
163.703212	1.000000	1002.230678	0.999131
168.063493	1.000000	1194.027092	0.999732
167.906161	1.000000	1037.117655	0.999291
167.534931	1.000000	1153.219993	0.999348
160.578051	1.000000	1099.115470	0.999620
164.235436	1.000000	1119.036819	0.999459
155.008998	1.000000	983.492939	0.999347
168.404630	1.000000	1173.490961	0.999621
158.640308	1.000000	929.069986	0.998900
169.846162	1.000000	1073.715782	0.999079
166.141267	1.000000	1073.629802	0.999427
154.979091	1.000000	982.535438	0.999386
160.257850	1.000000	1044.637846	0.999435
161.575667	1.000000	950.606726	0.999074
163.650667	1.000000	1071.597437	0.999612
163.903141	1.000000	1036.558550	0.999079
163.390138	1.000000	1015.912389	0.999246
153.517123	1.000000	938.737242	0.999245
163.341905	1.000000	1087.966424	0.999131
163.202602	1.000000	990.607248	0.999282
157.279692	1.000000	1092.031681	0.999304
157.167207	1.000000	1031.914173	0.999375
164.519616	1.000000	1116.878004	0.999525
172.830553	1.000000	1127.607220	0.999212
157.045462	1.000000	949.857215	0.999393
158.939444	1.000000	1097.207891	0.999530
166.080571	1.000000	1032.473234	0.999307
155.330333	1.000000	1026.918002	0.999654
157.490737	1.000000	995.761941	0.999186
169.363566	1.000000	1126.070143	0.999389
162.087452	1.000000	1031.539309	0.999257
166.146630	1.000000	1082.673834	0.999461
152.824261	1.000000	910.017183	0.999206
146.829732	1.000000	924.393371	0.999365
168.877339	1.000000	1137.035860	0.999606
168.070651	1.000000	1129.742025	0.999584
160.946507	1.000000	1032.181823	0.999479
167.129891	1.000000	1065.179861	0.999144
169.302965	1.000000	1112.927707	0.999332
162.711044	1.000000	1054.900093	0.999504
171.184705	1.000000	1172.545820	0.999545
163.758445	1.000000	1014.453929	0.998668
157.228066	1.000000	901.574449	0.998980
168.130405	1.000000	1078.706157	0.999472
166.130041	1.000000	1002.529343	0.999341
175.154889	1.000000	1212.304183	0.999446
160.073787	1.000000	978.252126	0.999221
152.161636	1.000000	986.410238	0.999273
164.625480	1.000000	1116.785044	0.999591
159.881383	1.000000	1075.237116	0.999590
156.692569	1.000000	972.616152	0.999177
158.534200	1.000000	1000.061725	0.999020
179.911712	1.000000	1196.074154	0.999441
158.838348	1.000000	1050.462071	0.999292
162.954921	1.000000	937.901760	0.999124
161.096447	1.000000	980.331768	0.999430
156.183857	1.000000	1007.957108	0.999193
169.187674	1.000000	1069.793460	0.999277
151.841936	1.000000	946.482385	0.999365
163.061099	1.000000	1024.474030	0.999079
150.880346	1.000000	979.292387	0.999025
161.731835	1.000000	1060.704492	0.999267
174.998578	1.000000	1156.713788	0.999556
168.526015	1.000000	1104.741326	0.999258
162.954771	1.000000	1065.073759	0.999186
169.277889	1.000000	1069.862912	0.999285
168.228308	1.000000	1151.591753	0.998837
159.667011	1.000000	942.982663	0.999597
150.853417	1.000000	924.862940	0.998897
156.269186	1.000000	950.494281	0.998943
173.503051	1.000000	1202.286503	0.999509
168.339646	1.000000	1060.308303	0.998967
155.678395	1.000000	920.965998	0.999031
159.265431	1.000000	986.311731	0.999203
168.691649	1.000000	1080.569822	0.999205
161.585337	1.000000	1048.289312	0.999266
165.321779	1.000000	1077.226498	0.999056
155.662681	1.000000	959.719982	0.998696
176.757512	1.000000	1186.465938	0.999492
177.075374	1.000000	1190.868471	0.999425
150.342079	1.000000	926.105312	0.999111
167.786899	1.000000	1038.944430	0.999644
158.571917	1.000000	997.085097	0.999270
172.169312	1.000000	1147.495995	0.999657
155.700443	1.000000	951.025125	0.999222
155.115318	1.000000	1039.340709	0.999662
159.181861	1.000000	1010.550430	0.998771
160.698527	1.000000	1126.487105	0.999554
165.051598	1.000000	1051.438811	0.999469
160.228322	1.000000	1046.138799	0.999189
141.442469	1.000000	851.192952	0.999132
161.683492	1.000000	1005.032512	0.999332

0/129 outside of cosine similarity of .95 (with softmax)
0/129 outside of cosine similarity of .99 (without softmax)

Non-Principal eigenvector vs self attention 1st vector

eigenvalue	mag eigenvector	mag self attention	cosine similarity (with softmax)
9206.530596	1.000000	95.267962	0.831076
153.035448	1.000000	6.677844	0.827429
146.985750	1.000000	6.469487	0.854084
160.615877	1.000000	6.880530	0.818504
145.946463	1.000000	6.241726	0.809585
165.390938	1.000000	6.646201	0.846254
168.939761	1.000000	6.637303	0.821917
173.455692	1.000000	7.324602	0.838477
164.364278	1.000000	7.093303	0.817140
170.809366	1.000000	6.815885	0.817282
168.725411	1.000000	6.963146	0.855412
162.228956	1.000000	6.471354	0.849791
152.398589	1.000000	6.420290	0.806038
152.788177	1.000000	6.011698	0.803389
159.292100	1.000000	6.641088	0.828570
164.603275	1.000000	6.781595	0.805229
165.112983	1.000000	7.025318	0.817439
165.635110	1.000000	6.455346	0.817941
157.807981	1.000000	6.499539	0.841227
177.225237	1.000000	6.971489	0.845225
158.409995	1.000000	6.420343	0.849690
160.217627	1.000000	6.233390	0.817334
170.408037	1.000000	7.133776	0.799513
168.460888	1.000000	6.594924	0.832881
153.034498	1.000000	6.524873	0.841504
169.224380	1.000000	6.500595	0.806773
174.409270	1.000000	7.013595	0.815065
153.403013	1.000000	6.277223	0.819660
157.568193	1.000000	6.493295	0.794931
155.412338	1.000000	6.163024	0.808923
162.250681	1.000000	6.594340	0.796342
162.858646	1.000000	6.511601	0.843460
148.066191	1.000000	6.250871	0.812702
161.412856	1.000000	6.308377	0.815296
154.030834	1.000000	6.039407	0.810947
174.718474	1.000000	6.969898	0.796877
165.243244	1.000000	6.815741	0.820110
163.703212	1.000000	6.490107	0.806950
168.063493	1.000000	6.475724	0.845176
167.906161	1.000000	6.497066	0.821268
167.534931	1.000000	6.784336	0.816985
160.578051	1.000000	6.815965	0.830033
164.235436	1.000000	6.536982	0.815864
155.008998	1.000000	6.548110	0.821649
168.404630	1.000000	6.042017	0.801990
158.640308	1.000000	6.138811	0.837103
169.846162	1.000000	7.062570	0.802700
166.141267	1.000000	6.706509	0.827752
154.979091	1.000000	6.249225	0.847668
160.257850	1.000000	6.502338	0.803048
161.575667	1.000000	6.861466	0.808260
163.650667	1.000000	7.000538	0.826630
163.903141	1.000000	6.550479	0.843383
163.390138	1.000000	7.016451	0.805900
153.517123	1.000000	6.939118	0.806179
163.341905	1.000000	6.181780	0.853177
163.202602	1.000000	6.498106	0.795990
157.279692	1.000000	6.649323	0.815621
157.167207	1.000000	6.459216	0.801528
164.519616	1.000000	6.337793	0.802012
172.830553	1.000000	6.946143	0.840569
157.045462	1.000000	6.116538	0.792016
158.939444	1.000000	6.169372	0.809588
166.080571	1.000000	6.873749	0.832122
155.330333	1.000000	6.276361	0.816875
157.490737	1.000000	6.500678	0.819748
169.363566	1.000000	6.415871	0.831047
162.087452	1.000000	6.448137	0.839985
166.146630	1.000000	6.409095	0.827629
152.824261	1.000000	6.379470	0.826678
146.829732	1.000000	6.456515	0.860507
168.877339	1.000000	6.621731	0.829650
168.070651	1.000000	6.735037	0.824828
160.946507	1.000000	6.945548	0.795881
167.129891	1.000000	6.698101	0.828953
169.302965	1.000000	6.386418	0.833584
162.711044	1.000000	6.820714	0.841658
171.184705	1.000000	6.479948	0.814795
163.758445	1.000000	6.786087	0.876946
157.228066	1.000000	6.897502	0.834229
168.130405	1.000000	7.153319	0.829107
166.130041	1.000000	6.869857	0.824784
175.154889	1.000000	7.173491	0.828407
160.073787	1.000000	6.545292	0.800310
152.161636	1.000000	6.392094	0.808618
164.625480	1.000000	6.573605	0.824282
159.881383	1.000000	6.868605	0.837351
156.692569	1.000000	6.437777	0.792224
158.534200	1.000000	6.013324	0.836775
179.911712	1.000000	7.174269	0.814937
158.838348	1.000000	5.587373	0.798442
162.954921	1.000000	7.203439	0.826036
161.096447	1.000000	6.860645	0.796685
156.183857	1.000000	6.461804	0.829516
169.187674	1.000000	6.765574	0.822824
151.841936	1.000000	5.958579	0.817141
163.061099	1.000000	6.349641	0.806047
150.880346	1.000000	6.106754	0.803671
161.731835	1.000000	6.297961	0.844227
174.998578	1.000000	6.804727	0.820807
168.526015	1.000000	6.916555	0.853660
162.954771	1.000000	6.502600	0.790466
169.277889	1.000000	6.708884	0.829987
168.228308	1.000000	6.492005	0.843894
159.667011	1.000000	7.356210	0.835014
150.853417	1.000000	6.376264	0.837785
156.269186	1.000000	6.799447	0.818207
173.503051	1.000000	6.649155	0.864477
168.339646	1.000000	6.439181	0.849884
155.678395	1.000000	6.491497	0.837756
159.265431	1.000000	6.530884	0.807703
168.691649	1.000000	6.532615	0.827122
161.585337	1.000000	6.650253	0.817530
165.321779	1.000000	6.535029	0.785597
155.662681	1.000000	6.734863	0.818867
176.757512	1.000000	6.260079	0.825161
177.075374	1.000000	6.768350	0.823006
150.342079	1.000000	5.996217	0.804438
167.786899	1.000000	6.814211	0.839086
158.571917	1.000000	6.793648	0.809067
172.169312	1.000000	7.102100	0.806284
155.700443	1.000000	6.887268	0.847649
155.115318	1.000000	6.317686	0.818406
159.181861	1.000000	6.494668	0.826172
160.698527	1.000000	6.389890	0.823149
165.051598	1.000000	6.528850	0.809876
160.228322	1.000000	6.611481	0.781293
141.442469	1.000000	6.018621	0.803200
161.683492	1.000000	6.363545	0.832482

eigenvalue	mag eigenvector	mag self attention	cosine similarity (without softmax)
9206.530596	1.000000	663563.299286	0.759353
153.035448	1.000000	986.932529	0.794412
146.985750	1.000000	921.390981	0.844428
160.615877	1.000000	1062.004238	0.786020
145.946463	1.000000	881.471524	0.792560
165.390938	1.000000	1060.110611	0.824138
168.939761	1.000000	1076.486190	0.786310
173.455692	1.000000	1217.497038	0.806871
164.364278	1.000000	1122.958085	0.778544
170.809366	1.000000	1117.169061	0.792614
168.725411	1.000000	1130.548081	0.817988
162.228956	1.000000	1004.520971	0.822705
152.398589	1.000000	942.872802	0.778624
152.788177	1.000000	884.139813	0.763514
159.292100	1.000000	1015.351799	0.790285
164.603275	1.000000	1078.569778	0.768503
165.112983	1.000000	1107.108619	0.780043
165.635110	1.000000	1034.053413	0.788457
157.807981	1.000000	989.866195	0.811710
177.225237	1.000000	1191.653017	0.809651
158.409995	1.000000	981.265974	0.824366
160.217627	1.000000	963.461947	0.796600
170.408037	1.000000	1161.864926	0.780410
168.460888	1.000000	1068.981161	0.812708
153.034498	1.000000	961.107531	0.795096
169.224380	1.000000	1055.491698	0.770199
174.409270	1.000000	1169.397747	0.787765
153.403013	1.000000	932.864167	0.802928
157.568193	1.000000	989.713896	0.775089
155.412338	1.000000	923.322647	0.768451
162.250681	1.000000	1028.272934	0.752371
162.858646	1.000000	1026.139767	0.829964
148.066191	1.000000	895.530530	0.780310
161.412856	1.000000	977.204049	0.779351
154.030834	1.000000	897.585165	0.788568
174.718474	1.000000	1166.419430	0.759370
165.243244	1.000000	1079.371510	0.781315
163.703212	1.000000	1019.041048	0.773263
168.063493	1.000000	1054.097491	0.822969
167.906161	1.000000	1042.329824	0.790147
167.534931	1.000000	1088.410514	0.795205
160.578051	1.000000	1054.150942	0.803976
164.235436	1.000000	1032.632653	0.786649
155.008998	1.000000	973.848494	0.790484
168.404630	1.000000	969.162695	0.755415
158.640308	1.000000	932.434099	0.817776
169.846162	1.000000	1142.183379	0.751276
166.141267	1.000000	1068.792570	0.810726
154.979091	1.000000	930.404063	0.810392
160.257850	1.000000	993.183997	0.770809
161.575667	1.000000	1073.839519	0.767971
163.650667	1.000000	1098.034117	0.786050
163.903141	1.000000	1033.038479	0.800069
163.390138	1.000000	1104.663242	0.791498
153.517123	1.000000	1029.581357	0.782122
163.341905	1.000000	972.653563	0.818319
163.202602	1.000000	1023.257487	0.767478
157.279692	1.000000	999.641199	0.776580
157.167207	1.000000	980.416235	0.776756
164.519616	1.000000	1000.514799	0.775064
172.830553	1.000000	1151.895329	0.812888
157.045462	1.000000	915.874324	0.750592
158.939444	1.000000	944.702202	0.774818
166.080571	1.000000	1096.793760	0.801077
155.330333	1.000000	940.219388	0.772656
157.490737	1.000000	979.557709	0.767341
169.363566	1.000000	1040.782782	0.808411
162.087452	1.000000	1001.808777	0.808881
166.146630	1.000000	1021.310957	0.808150
152.824261	1.000000	936.832629	0.784438
146.829732	1.000000	911.813623	0.810947
168.877339	1.000000	1067.384958	0.793722
168.070651	1.000000	1091.169738	0.790385
160.946507	1.000000	1076.632457	0.760864
167.129891	1.000000	1068.878228	0.816008
169.302965	1.000000	1043.349807	0.816336
162.711044	1.000000	1064.869687	0.811908
171.184705	1.000000	1070.166797	0.789451
163.758445	1.000000	1072.393726	0.854383
157.228066	1.000000	1046.396507	0.816342
168.130405	1.000000	1147.423876	0.779455
166.130041	1.000000	1098.639801	0.803197
175.154889	1.000000	1198.024532	0.767754
160.073787	1.000000	1002.511192	0.757024
152.161636	1.000000	940.430782	0.755833
164.625480	1.000000	1032.317496	0.791767
159.881383	1.000000	1054.191665	0.803722
156.692569	1.000000	975.302779	0.766659
158.534200	1.000000	915.931027	0.806051
179.911712	1.000000	1232.841465	0.777079
158.838348	1.000000	848.501567	0.772948
162.954921	1.000000	1126.246139	0.790454
161.096447	1.000000	1062.143340	0.752874
156.183857	1.000000	974.546894	0.806801
169.187674	1.000000	1100.147076	0.788766
151.841936	1.000000	869.090949	0.804821
163.061099	1.000000	995.329331	0.770385
150.880346	1.000000	888.211331	0.765935
161.731835	1.000000	984.603202	0.808891
174.998578	1.000000	1135.187239	0.794548
168.526015	1.000000	1126.843098	0.839097
162.954771	1.000000	1017.808123	0.745439
169.277889	1.000000	1086.866835	0.815043
168.228308	1.000000	1049.004683	0.803221
159.667011	1.000000	1132.786460	0.796088
150.853417	1.000000	924.619091	0.830806
156.269186	1.000000	1018.898690	0.781596
173.503051	1.000000	1105.225452	0.817194
168.339646	1.000000	1042.579505	0.824729
155.678395	1.000000	969.937637	0.797230
159.265431	1.000000	999.033539	0.787179
168.691649	1.000000	1056.922474	0.770239
161.585337	1.000000	1032.916585	0.765049
165.321779	1.000000	1030.084105	0.736276
155.662681	1.000000	1006.992628	0.791940
176.757512	1.000000	1060.100038	0.799826
177.075374	1.000000	1140.121328	0.808242
150.342079	1.000000	870.263814	0.770739
167.786899	1.000000	1089.167103	0.789884
158.571917	1.000000	1027.084912	0.771537
172.169312	1.000000	1164.404668	0.780515
155.700443	1.000000	1032.449923	0.823544
155.115318	1.000000	946.423578	0.799834
159.181861	1.000000	996.326855	0.791866
160.698527	1.000000	987.887217	0.804006
165.051598	1.000000	1034.203451	0.759536
160.228322	1.000000	1025.394188	0.765167
141.442469	1.000000	823.310567	0.786806
161.683492	1.000000	992.011656	0.812388

129/129 outside of cosine similarity of .95 (with softmax)
129/129 outside of cosine similarity of .99 (without softmax)