Full Result Table for DIGI dataset

You sort the results by clicking on the table headers.

Metrics MRR@3 MRR@5 MRR@10 MRR@20 HitRate@3 HitRate@5 HitRate@10 HitRate@20 Cov@20 Pop@20 T-time(s) P-time (s)
SFSKNN 0.322 0.337 0.348 0.351 0.413 0.480 0.559 0.604 0.072 0.071 4.895 0.006
STAN 0.321 0.338 0.347 0.351 0.389 0.465 0.529 0.600 0.087 0.091 2.307 0.015
VSTAN 0.317 0.332 0.342 0.346 0.383 0.449 0.520 0.581 0.086 0.084 2.709 0.014
SR 0.304 0.324 0.333 0.337 0.413 0.497 0.568 0.617 0.086 0.075 7.273 0.004
VSKNN 0.300 0.312 0.325 0.330 0.353 0.406 0.499 0.576 0.082 0.084 9.030 0.037
COTREC 0.298 0.311 0.330 0.335 0.366 0.424 0.555 0.637 0.078 0.079 28842.562 1.992
MGS 0.292 0.309 0.322 0.329 0.383 0.458 0.559 0.656 0.073 0.089 23453.641 0.016
SR-GNN 0.286 0.305 0.321 0.327 0.385 0.469 0.591 0.688 0.079 0.078 3307.174 0.027
GNRRW 0.282 0.303 0.318 0.324 0.378 0.469 0.578 0.667 0.073 0.078 9279.445 0.174
CM-HGNN 0.274 0.296 0.310 0.316 0.361 0.458 0.561 0.649 0.072 0.083 56683.551 0.013
TAGNN 0.243 0.264 0.279 0.287 0.333 0.428 0.544 0.645 0.073 0.102 7189.390 0.007
GCE-GNN 0.202 0.216 0.227 0.236 0.269 0.333 0.419 0.553 0.065 0.097 69231.684 0.503
FLCSP 0.146 0.159 0.175 0.184 0.219 0.280 0.398 0.525 0.073 0.075 2449.509 0.010

Statistical analysis based on MRR@20

We perform a significance test by calculating session-individual metric values e.g., obtain the mean MRR values of all sessions in the test data and then use the ANOVA test to analyze whether the difference between the compared models is significant.

One-way ANOVA

Null hypothesis (Ho): There is no difference in the model means
Alternative hypothesis (H1): At least one model's mean differs from the others
Significance level = 0.05
F-value = 2.509
P-value = 0.004

The P-value is less than the value of of significance level. So, we have enough evidence to reject the Ho. We now know that at least the mean of a model is different from the others. Therefore, the post hoc analysis was performed using the Tukey test to determine the mean of which model differs from the others. Our analysis reveals that the following models have different means while others have the same mean. Moreover, in the first table, we can see that the MRR@20 values of the best performing models e.g., SFSKNN, STAN, VTSAN, SR, MGS and COREC are nearly equal. Therefore, the ANOVA test reveals that top-performing models do not have significant difference among their mean MRR@20 values.

Note: we report only those cases of the Tukey test where the Ho is rejected.


group1 group2 meandiff p-adj lower upper reject
FLCSP GGNN 0.0966 0.0277 0.0052 0.188 TRUE
FLCSP GNRWW 0.0941 0.0369 0.0027 0.1855 TRUE
FLCSP MGS 0.0927 0.0432 0.0013 0.1841 TRUE
FLCSP sfcknn 0.1047 0.01 0.0133 0.1961 TRUE
FLCSP sr 0.0921 0.0461 0.0007 0.1835 TRUE
FLCSP stan 0.0953 0.0321 0.0039 0.1867 TRUE
FLCSP vstan 0.0939 0.0379 0.0025 0.1853 TRUE

Statistical values based on MRR@20

  • Average: 0.335
  • Standard deviation: 0.010
  • Minimum value: 0.309
  • Maximum value: 0.343
  • Difference(%): 3.4
Metrics MRR@3: MRR@5: MRR@10: MRR@20: HitRate@3: HitRate@5: HitRate@10: HitRate@20:
embedding_size=16 0.276 0.289 0.302 0.310 0.335 0.396 0.495 0.596
embedding_size=32 0.292 0.308 0.322 0.329 0.357 0.428 0.538 0.624
embedding_size=64 0.304 0.322 0.338 0.343 0.366 0.441 0.566 0.632
embedding_size=80 0.303 0.319 0.335 0.340 0.374 0.447 0.557 0.632
embedding_size=100 0.306 0.325 0.337 0.343 0.381 0.465 0.557 0.639
embedding_size=120 0.301 0.318 0.335 0.340 0.368 0.441 0.566 0.645
embedding_size=150 0.300 0.314 0.329 0.337 0.372 0.437 0.553 0.654
embedding_size=200 0.296 0.311 0.328 0.333 0.368 0.432 0.563 0.637
embedding_size=230 0.294 0.313 0.328 0.334 0.361 0.447 0.559 0.641
embedding_size=250 0.306 0.321 0.337 0.343 0.376 0.445 0.563 0.647

Statistical values based on MRR@20

  • Average: 0.336
  • Standard deviation: 0.005
  • Minimum value: 0.327
  • Maximum value: 0.346
  • Difference(%): 1.9
Metrics MRR@3: MRR@5: MRR@10: MRR@20: HitRate@3: HitRate@5: HitRate@10: HitRate@20:
embedding_size=16 0.286 0.305 0.320 0.327 0.370 0.456 0.566 0.667
embedding_size=32 0.303 0.320 0.334 0.342 0.402 0.475 0.581 0.684
embedding_size=64 0.302 0.318 0.330 0.338 0.404 0.475 0.566 0.682
embedding_size=80 0.300 0.318 0.333 0.339 0.415 0.492 0.598 0.688
embedding_size=100 0.295 0.313 0.328 0.335 0.391 0.467 0.583 0.686
embedding_size=120 0.294 0.312 0.326 0.333 0.385 0.467 0.572 0.677
embedding_size=150 0.306 0.321 0.336 0.342 0.406 0.473 0.583 0.684
embedding_size=200 0.296 0.314 0.328 0.335 0.402 0.477 0.585 0.682
embedding_size=230 0.291 0.311 0.325 0.330 0.385 0.475 0.574 0.654
embedding_size=250 0.309 0.328 0.341 0.347 0.411 0.495 0.600 0.677

Statistical values based on MRR@20

  • Average: 0.247
  • Standard deviation: 0.033
  • Minimum value: 0.164
  • Maximum value: 0.294
  • Difference(%): 13
Metrics MRR@3: MRR@5: MRR@10: MRR@15: MRR@20: HitRate@3: HitRate@5: HitRate@10: HitRate@20:
embedding_size=16 0.132 0.146 0.157 0.163 0.165 0.187 0.247 0.333 0.439
embedding_size=32 0.208 0.226 0.240 0.244 0.246 0.295 0.372 0.480 0.566
embedding_size=64 0.255 0.276 0.289 0.293 0.294 0.344 0.434 0.535 0.617
embedding_size=80 0.221 0.241 0.256 0.261 0.263 0.295 0.383 0.495 0.594
embedding_size=100 0.221 0.243 0.257 0.262 0.263 0.310 0.404 0.514 0.598
embedding_size=120 0.203 0.219 0.235 0.240 0.242 0.288 0.359 0.473 0.581
embedding_size=150 0.226 0.239 0.253 0.258 0.262 0.305 0.359 0.467 0.596
embedding_size=200 0.214 0.232 0.250 0.254 0.256 0.290 0.366 0.505 0.583
embedding_size=230 0.206 0.226 0.240 0.244 0.247 0.299 0.383 0.490 0.587
embedding_size=250 0.200 0.219 0.233 0.237 0.239 0.303 0.385 0.488 0.583

Statistical values based on MRR@20

  • Average: 0.247
  • Standard deviation: 0.011
  • Minimum value: 0.191
  • Maximum value: 0.224
  • Difference(%): 3.3
Metrics MRR@3: MRR@5: MRR@10: MRR@20: HitRate@3: HitRate@5: HitRate@10: HitRate@20:
embedding_size=16 0.175 0.188 0.200 0.207 0.245 0.303 0.396 0.492
embedding_size=32 0.176 0.189 0.204 0.211 0.252 0.310 0.428 0.540
embedding_size=64 0.158 0.168 0.183 0.191 0.228 0.273 0.381 0.503
embedding_size=80 0.174 0.186 0.202 0.209 0.258 0.312 0.432 0.527
embedding_size=100 0.159 0.177 0.191 0.197 0.222 0.301 0.402 0.490
embedding_size=120 0.187 0.203 0.215 0.222 0.262 0.333 0.422 0.525
embedding_size=150 0.165 0.182 0.195 0.203 0.234 0.310 0.398 0.516
embedding_size=200 0.155 0.170 0.186 0.192 0.234 0.303 0.419 0.514
embedding_size=230 0.190 0.205 0.218 0.225 0.265 0.329 0.430 0.529
embedding_size=250 0.163 0.183 0.198 0.205 0.228 0.316 0.415 0.523

Statistical values based on MRR@20

  • Average: 0.313
  • Standard deviation: 0.008
  • Minimum value: 0.296
  • Maximum value: 0.324
  • Difference(%): 2.8
Metrics MRR@3: MRR@5: MRR@10: MRR@20: HitRate@3: HitRate@5: HitRate@10: HitRate@20:
embedding_size=16 0.257 0.277 0.289 0.297 0.361 0.447 0.540 0.654
embedding_size=32 0.272 0.291 0.307 0.314 0.372 0.456 0.576 0.675
embedding_size=64 0.269 0.289 0.304 0.312 0.378 0.465 0.578 0.688
embedding_size=80 0.287 0.306 0.318 0.324 0.389 0.473 0.563 0.656
embedding_size=100 0.272 0.289 0.304 0.311 0.372 0.452 0.566 0.671
embedding_size=120 0.270 0.291 0.303 0.310 0.383 0.475 0.568 0.669
embedding_size=150 0.271 0.292 0.307 0.313 0.368 0.462 0.572 0.660
embedding_size=200 0.284 0.300 0.315 0.321 0.394 0.469 0.576 0.667
embedding_size=230 0.276 0.294 0.311 0.318 0.378 0.460 0.587 0.680
embedding_size=250 0.279 0.295 0.309 0.315 0.394 0.462 0.572 0.660

Statistical analysis based on MRR@20

COTREC GNRWW TAGNN FLCSP GGNN
0.310 0.327 0.165 0.207 0.297
0.329 0.342 0.246 0.211 0.314
0.343 0.338 0.294 0.191 0.312
0.340 0.339 0.263 0.209 0.324
0.343 0.335 0.263 0.197 0.311
0.340 0.333 0.242 0.222 0.310
0.337 0.342 0.262 0.203 0.313
0.333 0.335 0.256 0.192 0.321
0.334 0.330 0.247 0.225 0.318
0.343 0.347 0.239 0.205 0.315

One-way ANOVA

Null hypothesis (Ho): There is no difference in the model means UCOTREC = UGNRWW = UTAGNN = UFLCSP = UGGNN
Alternative hypothesis (H1): At least one model's mean differs from the others
Significance level = 0.05

Summary
Groups Count Sum Average Variance
COTREC 10 3.350865671 0.3350865671 0.0001032043935
GNRWW 10 3.368446379 0.3368446379 0.00003443092133
TAGNN 10 2.476281933 0.2476281933 0.0010992159
FLCSP 10 2.062314624 0.2062314624 0.0001282693332
GGNN 10 3.135340815 0.3135340815 0.00005413629124
ANOVA
Source of Variation SS df MS F-value P-value F(critical)
Between Groups 0.1357081815 4 0.03392704538 119.5239806 0 2.578739184
Within Groups 0.01277331155 45 0.0002838513678
Total 0.1484814931 49

The F-value and critical F- value are 119.523 and 2.578, respectively. The F-value is greater than the critical F-value, therefore, we reject the Ho. The main purpose of the ANOVA test is to observe the variability between the groups/compared models. However, it does not explain about the magnitude of variability. Therefore, we use the effect size (η) to know the magnitude of variability between the models. The effect size uses the following table to explain the level of variability.


η Size
0.01 small
0.09 medium
0.25 large
Note: if less than .01, no effect is reported

η = 0.135708181506113 / 0.148481493058201
η = 0.9139
Here we observe the large effect size between the GNN models.

References:

  1. https://open.maricopa.edu/psy230mm/chapter/chapter-14-analysis-of-variance/
  2. https://en.wikipedia.org/wiki/Analysis_of_variance

Statistical values based on MRR@20

  • Average: 0.337
  • Standard deviation: 0.003
  • Minimum value: 0.329
  • Maximum value: 0.343
  • Difference(%): 1.4

MRR@20 varies across of random seeds

MRR@20 vs random seeds

Metrics MRR@3: MRR@5: MRR@10: MRR@20: HitRate@3: HitRate@5: HitRate@10: HitRate@20:
seed=1996073 0.305 0.320 0.336 0.341 0.374 0.441 0.559 0.632
seed=854750 0.301 0.318 0.332 0.338 0.368 0.443 0.551 0.628
seed=888529 0.300 0.314 0.330 0.336 0.372 0.432 0.553 0.630
seed=1184622 0.301 0.312 0.330 0.336 0.374 0.424 0.548 0.645
seed=17829 0.303 0.318 0.333 0.340 0.376 0.441 0.548 0.634
seed=1830297 0.305 0.321 0.337 0.342 0.374 0.447 0.568 0.634
seed=2143189 0.297 0.315 0.330 0.336 0.366 0.439 0.551 0.637
seed=2286376 0.290 0.309 0.325 0.329 0.355 0.441 0.557 0.626
seed=174185 0.299 0.318 0.334 0.339 0.368 0.454 0.566 0.637
seed=2056912 0.302 0.317 0.332 0.338 0.376 0.443 0.551 0.637
seed=2634968 0.303 0.318 0.335 0.341 0.366 0.434 0.555 0.643
seed=804381 0.299 0.319 0.332 0.339 0.363 0.454 0.551 0.639
seed=1918225 0.300 0.317 0.333 0.339 0.366 0.439 0.559 0.637
seed=2054242 0.301 0.316 0.331 0.338 0.370 0.432 0.546 0.639
seed=2698057 0.301 0.316 0.334 0.339 0.368 0.430 0.566 0.634
seed=2973600 0.304 0.319 0.334 0.341 0.376 0.441 0.548 0.647
seed=1386487 0.311 0.323 0.338 0.344 0.387 0.443 0.555 0.632
seed=2765375 0.300 0.316 0.334 0.338 0.363 0.439 0.568 0.634
seed=2754545 0.291 0.313 0.327 0.333 0.353 0.452 0.553 0.634
seed=204245 0.304 0.319 0.333 0.339 0.381 0.447 0.553 0.639
seed=770362 0.299 0.313 0.330 0.336 0.368 0.426 0.555 0.639
seed=1274950 0.300 0.317 0.332 0.338 0.363 0.439 0.546 0.639
seed=2920651 0.300 0.317 0.332 0.338 0.366 0.439 0.553 0.634
seed=2874302 0.301 0.320 0.335 0.340 0.366 0.452 0.555 0.630
seed=1526224 0.296 0.314 0.330 0.336 0.363 0.443 0.557 0.637
seed=1287909 0.303 0.323 0.336 0.342 0.372 0.462 0.557 0.645
seed=1536514 0.297 0.315 0.330 0.336 0.357 0.434 0.553 0.632
seed=1202831 0.304 0.319 0.336 0.342 0.370 0.439 0.561 0.641
seed=424710 0.299 0.314 0.329 0.335 0.376 0.443 0.557 0.632
seed=2050972 0.295 0.311 0.325 0.331 0.368 0.443 0.548 0.634
seed=608005 0.301 0.319 0.332 0.339 0.366 0.445 0.540 0.641
seed=831980 0.300 0.317 0.333 0.339 0.361 0.434 0.555 0.634
seed=1798918 0.293 0.312 0.327 0.333 0.359 0.441 0.561 0.634
seed=384211 0.303 0.322 0.335 0.340 0.366 0.452 0.553 0.634

Statistical values based on MRR@20

  • Average: 0.340
  • Standard deviation: 0.006
  • Minimum value: 0.327
  • Maximum value: 0.352
  • Difference(%): 2.5

MRR@20 varies across of random seeds

MRR@20 vs random seeds

Metrics MRR@3: MRR@5: MRR@10: MRR@20: HitRate@3: HitRate@5: HitRate@10: HitRate@20:
seed=1447236 0.299 0.315 0.329 0.336 0.398 0.471 0.576 0.671
seed=1304289 0.298 0.320 0.333 0.340 0.383 0.480 0.581 0.684
seed=87127 0.292 0.312 0.328 0.334 0.378 0.469 0.581 0.665
seed=1881266 0.296 0.316 0.332 0.339 0.385 0.473 0.594 0.697
seed=2840823 0.291 0.312 0.325 0.332 0.387 0.475 0.574 0.673
seed=1851086 0.302 0.318 0.334 0.341 0.389 0.456 0.581 0.675
seed=1075394 0.304 0.319 0.334 0.342 0.396 0.462 0.570 0.673
seed=493588 0.297 0.314 0.331 0.337 0.391 0.462 0.594 0.680
seed=1592372 0.306 0.329 0.341 0.348 0.387 0.486 0.574 0.675
seed=167556 0.306 0.323 0.338 0.345 0.394 0.465 0.585 0.697
seed=2886392 0.296 0.315 0.327 0.334 0.394 0.475 0.568 0.675
seed=712613 0.307 0.322 0.336 0.344 0.400 0.462 0.570 0.673
seed=803331 0.310 0.328 0.342 0.348 0.398 0.475 0.583 0.671
seed=738772 0.296 0.314 0.328 0.335 0.394 0.475 0.572 0.675
seed=628465 0.296 0.314 0.328 0.335 0.387 0.462 0.572 0.667
seed=2958120 0.303 0.319 0.333 0.342 0.396 0.465 0.563 0.690
seed=207155 0.301 0.317 0.331 0.338 0.391 0.462 0.561 0.669
seed=1096544 0.297 0.318 0.332 0.339 0.381 0.471 0.578 0.680
seed=1746569 0.317 0.332 0.346 0.353 0.415 0.482 0.589 0.686
seed=420360 0.308 0.326 0.337 0.345 0.402 0.480 0.568 0.673
seed=48698 0.306 0.321 0.335 0.343 0.400 0.462 0.570 0.684
seed=2866263 0.292 0.310 0.325 0.333 0.381 0.458 0.561 0.673
seed=2588109 0.307 0.328 0.337 0.346 0.391 0.484 0.557 0.677
seed=2442223 0.307 0.323 0.337 0.343 0.396 0.465 0.572 0.662
seed=2795284 0.289 0.308 0.320 0.328 0.387 0.467 0.563 0.673
seed=2083911 0.306 0.322 0.337 0.343 0.394 0.467 0.572 0.660
seed=1226771 0.306 0.323 0.335 0.342 0.404 0.482 0.574 0.667
seed=2477232 0.312 0.329 0.344 0.349 0.400 0.475 0.585 0.667
seed=127267 0.310 0.330 0.345 0.351 0.383 0.473 0.591 0.675
seed=895429 0.287 0.308 0.323 0.330 0.378 0.473 0.578 0.684
seed=1299549 0.306 0.325 0.341 0.346 0.400 0.482 0.598 0.673
seed=477658 0.309 0.326 0.342 0.350 0.387 0.460 0.583 0.688
seed=700523 0.299 0.314 0.330 0.336 0.389 0.454 0.578 0.671
seed=1974593 0.300 0.321 0.333 0.340 0.400 0.492 0.583 0.684
seed=851690 0.297 0.317 0.328 0.336 0.394 0.480 0.563 0.677
seed=1957974 0.299 0.318 0.331 0.338 0.389 0.473 0.572 0.665
seed=2553520 0.296 0.314 0.329 0.338 0.387 0.465 0.574 0.690
seed=2047942 0.292 0.310 0.326 0.333 0.376 0.456 0.570 0.677
seed=1995953 0.303 0.319 0.332 0.340 0.402 0.473 0.572 0.680
seed=1785178 0.314 0.332 0.343 0.351 0.398 0.477 0.561 0.673
seed=123577 0.305 0.324 0.338 0.346 0.387 0.471 0.581 0.684
seed=402210 0.302 0.324 0.338 0.346 0.374 0.471 0.570 0.682
seed=709883 0.293 0.312 0.327 0.334 0.378 0.458 0.570 0.669
seed=534417 0.295 0.315 0.329 0.336 0.389 0.480 0.585 0.675
seed=1919455 0.296 0.315 0.329 0.336 0.381 0.467 0.568 0.665
seed=138216 0.299 0.317 0.332 0.338 0.389 0.471 0.583 0.675
seed=1682370 0.312 0.328 0.341 0.348 0.411 0.482 0.578 0.680
seed=252274 0.314 0.331 0.346 0.353 0.404 0.477 0.583 0.686
seed=917448 0.304 0.321 0.337 0.344 0.394 0.465 0.587 0.675
seed=2813644 0.296 0.319 0.331 0.338 0.394 0.495 0.581 0.688

Statistical values based on MRR@20

  • Average: 0.256
  • Standard deviation: 0.015
  • Minimum value: 0.228
  • Maximum value: 0.281
  • Difference(%): 5.2

MRR@20 varies across of random seeds

MRR@20 vs random seeds

Metrics MRR@3: MRR@5: MRR@10: MRR@20: HitRate@3: HitRate@5: HitRate@10: HitRate@20:
seed=2156700 0.199 0.218 0.235 0.241 0.292 0.378 0.505 0.589
seed=2625800 0.219 0.246 0.261 0.267 0.310 0.426 0.533 0.626
seed=1019900 0.233 0.250 0.264 0.272 0.314 0.389 0.497 0.606
seed=1169800 0.201 0.220 0.240 0.247 0.284 0.366 0.505 0.604
seed=8510500 0.204 0.216 0.237 0.244 0.286 0.338 0.492 0.596
seed=3405000 0.197 0.220 0.234 0.241 0.301 0.400 0.503 0.606
seed=7621100 0.192 0.212 0.227 0.234 0.282 0.370 0.480 0.581
seed=6717900 0.228 0.244 0.259 0.266 0.308 0.378 0.495 0.594
seed=4359300 0.207 0.224 0.243 0.249 0.284 0.361 0.503 0.596
seed=6402500 0.225 0.242 0.258 0.266 0.310 0.385 0.508 0.619
seed=374300 0.232 0.246 0.263 0.268 0.331 0.391 0.514 0.585
seed=1765400 0.202 0.221 0.233 0.240 0.288 0.374 0.462 0.572
seed=7477300 0.232 0.254 0.267 0.275 0.323 0.417 0.516 0.626
seed=2982000 0.225 0.245 0.260 0.267 0.318 0.406 0.516 0.632
seed=7066500 0.229 0.242 0.261 0.270 0.318 0.374 0.520 0.641
seed=7864000 0.216 0.235 0.250 0.257 0.292 0.376 0.488 0.589
seed=1964600 0.203 0.221 0.235 0.242 0.303 0.383 0.490 0.594
seed=9114700 0.233 0.247 0.261 0.268 0.303 0.366 0.475 0.570
seed=9102300 0.200 0.216 0.231 0.240 0.280 0.348 0.465 0.587
seed=5097000 0.200 0.218 0.235 0.243 0.282 0.361 0.477 0.596
seed=9151700 0.233 0.255 0.269 0.277 0.318 0.415 0.523 0.630
seed=5503200 0.220 0.235 0.251 0.258 0.297 0.361 0.482 0.583
seed=5008900 0.225 0.241 0.258 0.265 0.320 0.391 0.512 0.615
seed=9707200 0.186 0.201 0.218 0.228 0.275 0.342 0.471 0.615
seed=5080000 0.242 0.263 0.273 0.281 0.333 0.426 0.503 0.606

Statistical values based on MRR@20

  • Average: 0.203
  • Standard deviation: 0.009
  • Minimum value: 0.172
  • Maximum value: 0.217
  • Difference(%): 4.5

MRR@20 varies across of random seeds

MRR@20 vs random seeds

Metrics MRR@3: MRR@5: MRR@10: MRR@20: HitRate@3: HitRate@5: HitRate@10: HitRate@20:
seed=851200 0.163 0.185 0.200 0.207 0.222 0.316 0.437 0.533
seed=3238900 0.176 0.193 0.207 0.215 0.245 0.316 0.424 0.531
seed=2971600 0.165 0.180 0.195 0.203 0.226 0.290 0.406 0.525
seed=1863000 0.166 0.181 0.195 0.204 0.239 0.303 0.406 0.531
seed=6541600 0.163 0.182 0.198 0.207 0.215 0.299 0.415 0.544
seed=3411300 0.162 0.180 0.195 0.203 0.237 0.316 0.426 0.538
seed=4766000 0.165 0.182 0.195 0.202 0.237 0.312 0.411 0.520
seed=3108300 0.160 0.180 0.193 0.202 0.228 0.318 0.417 0.542
seed=3605600 0.160 0.173 0.186 0.194 0.222 0.277 0.376 0.501
seed=7563200 0.177 0.193 0.206 0.216 0.254 0.323 0.419 0.553
seed=7407300 0.175 0.192 0.207 0.214 0.239 0.316 0.428 0.525
seed=5491200 0.162 0.179 0.190 0.200 0.243 0.316 0.400 0.555
seed=7271100 0.166 0.180 0.196 0.203 0.237 0.299 0.415 0.527
seed=9514600 0.151 0.171 0.184 0.191 0.228 0.312 0.411 0.518
seed=5767800 0.166 0.179 0.191 0.199 0.243 0.303 0.391 0.505
seed=6856200 0.153 0.173 0.184 0.194 0.222 0.308 0.394 0.533
seed=2725700 0.155 0.168 0.181 0.189 0.222 0.282 0.381 0.505
seed=4395900 0.179 0.197 0.210 0.217 0.249 0.331 0.428 0.525
seed=8766000 0.162 0.178 0.192 0.200 0.224 0.295 0.406 0.518
seed=3461600 0.167 0.189 0.201 0.208 0.234 0.329 0.422 0.531
seed=3184400 0.172 0.185 0.200 0.210 0.234 0.288 0.404 0.548
seed=4451000 0.155 0.172 0.186 0.192 0.217 0.288 0.394 0.482
seed=2861600 0.177 0.191 0.207 0.214 0.241 0.303 0.424 0.527
seed=3847600 0.173 0.188 0.203 0.212 0.243 0.308 0.426 0.546
seed=9242000 0.153 0.167 0.183 0.191 0.224 0.286 0.404 0.512
seed=5152200 0.176 0.189 0.205 0.214 0.245 0.299 0.426 0.553
seed=8868300 0.129 0.147 0.163 0.172 0.181 0.262 0.383 0.520
seed=6399100 0.175 0.191 0.204 0.213 0.254 0.323 0.424 0.544
seed=602300 0.164 0.177 0.192 0.197 0.254 0.312 0.424 0.508
seed=8556600 0.168 0.182 0.195 0.204 0.243 0.303 0.409 0.531
seed=900000 0.158 0.177 0.189 0.198 0.243 0.323 0.417 0.538
seed=724600 0.154 0.169 0.184 0.192 0.217 0.280 0.396 0.505
seed=8223300 0.155 0.171 0.185 0.193 0.217 0.288 0.396 0.510
seed=2641400 0.165 0.181 0.195 0.205 0.230 0.303 0.409 0.557
seed=8817900 0.174 0.193 0.208 0.217 0.237 0.318 0.437 0.557
seed=4217200 0.168 0.182 0.197 0.206 0.239 0.301 0.413 0.538
seed=9522300 0.159 0.177 0.191 0.199 0.215 0.290 0.394 0.523
seed=3571000 0.167 0.183 0.195 0.203 0.241 0.310 0.400 0.512
seed=6604100 0.157 0.174 0.187 0.196 0.239 0.316 0.413 0.555
seed=2226700 0.174 0.189 0.201 0.209 0.254 0.320 0.406 0.527
seed=740400 0.173 0.190 0.203 0.211 0.247 0.320 0.426 0.535
seed=7336000 0.169 0.187 0.202 0.209 0.237 0.314 0.426 0.527
seed=2197700 0.160 0.183 0.196 0.205 0.230 0.329 0.426 0.563
seed=5396900 0.175 0.189 0.200 0.209 0.265 0.331 0.415 0.538
seed=3416000 0.160 0.180 0.189 0.198 0.232 0.318 0.389 0.518
seed=1039800 0.162 0.179 0.193 0.199 0.228 0.301 0.411 0.501
seed=4015400 0.154 0.176 0.188 0.196 0.219 0.312 0.402 0.516
seed=5080900 0.154 0.174 0.188 0.196 0.222 0.305 0.419 0.525
seed=9209300 0.179 0.198 0.210 0.216 0.247 0.329 0.424 0.512
seed=3671400 0.171 0.190 0.203 0.211 0.234 0.316 0.417 0.523

Statistical values based on MRR@20

  • Average: 0.322
  • Standard deviation: 0.009
  • Minimum value: 0.307
  • Maximum value: 0.342
  • Difference(%):3.5

MRR@20 varies across of random seeds

MRR@20 vs random seeds

Metrics MRR@3: MRR@5: MRR@10: MRR@20: HitRate@3: HitRate@5: HitRate@10: HitRate@20:
seed=4116600 0.278 0.299 0.314 0.322 0.381 0.469 0.585 0.690
seed=1448500 0.275 0.297 0.313 0.320 0.359 0.458 0.578 0.677
seed=5464300 0.274 0.292 0.307 0.315 0.383 0.462 0.578 0.692
seed=8419700 0.281 0.302 0.320 0.327 0.376 0.465 0.600 0.708
seed=5415500 0.261 0.285 0.302 0.309 0.359 0.465 0.585 0.686
seed=5744100 0.279 0.303 0.316 0.322 0.394 0.497 0.596 0.680
seed=5835200 0.277 0.301 0.314 0.322 0.376 0.480 0.581 0.692
seed=7885800 0.287 0.303 0.319 0.326 0.391 0.465 0.583 0.677
seed=1751600 0.283 0.303 0.319 0.327 0.378 0.465 0.589 0.690
seed=393300 0.290 0.308 0.319 0.328 0.398 0.480 0.566 0.690
seed=9353400 0.289 0.306 0.322 0.331 0.385 0.462 0.585 0.705
seed=8454600 0.296 0.314 0.330 0.337 0.391 0.469 0.587 0.690
seed=8112600 0.265 0.285 0.303 0.310 0.353 0.441 0.578 0.669
seed=2880500 0.267 0.284 0.300 0.308 0.383 0.458 0.574 0.690
seed=4272900 0.288 0.307 0.322 0.331 0.378 0.462 0.570 0.692
seed=5626300 0.277 0.294 0.309 0.316 0.387 0.462 0.570 0.673
seed=5732000 0.291 0.311 0.325 0.332 0.387 0.480 0.589 0.686
seed=7404800 0.300 0.321 0.334 0.341 0.389 0.484 0.587 0.677
seed=1919800 0.277 0.294 0.312 0.319 0.378 0.454 0.581 0.682
seed=3354400 0.276 0.296 0.315 0.320 0.372 0.458 0.602 0.682
seed=1685800 0.303 0.319 0.333 0.340 0.400 0.469 0.576 0.669
seed=8395700 0.279 0.299 0.314 0.321 0.387 0.473 0.587 0.686
seed=1208800 0.270 0.287 0.303 0.311 0.381 0.458 0.578 0.695
seed=3102900 0.283 0.303 0.320 0.326 0.372 0.460 0.585 0.667
seed=3163100 0.269 0.290 0.306 0.313 0.368 0.460 0.574 0.673
seed=5601800 0.290 0.310 0.323 0.329 0.394 0.477 0.578 0.667
seed=151700 0.263 0.284 0.301 0.308 0.368 0.456 0.578 0.682
seed=5621500 0.259 0.281 0.300 0.307 0.348 0.447 0.587 0.686
seed=2264600 0.287 0.304 0.319 0.325 0.398 0.473 0.591 0.677
seed=3172900 0.280 0.298 0.315 0.322 0.385 0.465 0.591 0.697
seed=5694900 0.282 0.299 0.315 0.322 0.389 0.467 0.578 0.686
seed=3813500 0.281 0.305 0.323 0.329 0.374 0.482 0.609 0.699
seed=8295000 0.270 0.297 0.311 0.318 0.357 0.473 0.581 0.671
seed=5495900 0.277 0.295 0.314 0.321 0.368 0.445 0.589 0.692
seed=7106200 0.302 0.318 0.336 0.342 0.404 0.477 0.613 0.705

Statistical analysis based on MRR@20

COTREC GNRRW TAGNN FLCSP GGNN
0.3409937 0.33599206 0.2409126 0.2065565 0.3218514
0.3376615 0.34049176 0.2671731 0.2150780 0.3196387
0.3360568 0.33384089 0.2719202 0.2032149 0.3150520
0.3364039 0.33920997 0.2466212 0.2038967 0.3270256
0.3395593 0.33225052 0.2441588 0.2071149 0.3085969
0.3417518 0.34050359 0.2408133 0.2030578 0.3219477
0.3358070 0.34157847 0.2343214 0.2023181 0.3219050
0.3294316 0.33690436 0.2658386 0.2021730 0.3257824
0.3385516 0.34753455 0.2494611 0.1944837 0.3266497
0.3379797 0.34518535 0.2662642 0.2156291 0.3278797
0.3405640 0.33449492 0.2681523 0.2141539 0.3308277
0.3385499 0.34387576 0.2400835 0.2003993 0.3369646
0.3385337 0.34791349 0.2747149 0.2031463 0.3096330
0.3376391 0.33509712 0.2674166 0.1910997 0.3084844
0.3392099 0.33452941 0.2699714 0.1991495 0.3305054
0.3409725 0.34168398 0.2573615 0.1941384 0.3159991
0.3439723 0.33834337 0.2423687 0.1894486 0.3318970
0.3382212 0.33945972 0.2679075 0.2170486 0.3408552
0.3327706 0.35296907 0.2401311 0.1999889 0.3188616
0.3386084 0.34456512 0.2427355 0.2084436 0.3202280
0.3359206 0.34302696 0.2766572 0.2101228 0.3398760
0.3381904 0.33284777 0.2577398 0.1916497 0.3208971
0.3378326 0.34612621 0.2649578 0.2138115 0.3110524
0.3400954 0.34324776 0.2280838 0.2120328 0.3260680
0.3355072 0.32798605 0.2805328 0.1907867 0.3131729
0.3420006 0.34320468 0.2141117 0.3291776
0.3361578 0.34163871 0.1721716 0.3079964
0.3419555 0.34923896 0.2126700 0.3072804
0.3349865 0.35120298 0.1974497 0.3251047
0.3313467 0.32971577 0.2041034 0.3220460
0.3392978 0.34585229 0.1975199 0.3219706
0.3388039 0.34994560 0.1921189 0.3290076
0.3327864 0.33646468 0.1927468 0.3177662
0.3403953 0.33990110 0.2049453 0.3208899
0.33623396 0.2167602 0.3423190
0.33762308 0.2058433
0.33762454 0.1994011
0.33330213 0.2027731
0.33991445 0.1963051
0.35101249 0.2091475
0.34575456 0.2106636
0.34615514 0.2093573
0.33388054 0.2053261
0.33553872 0.2088231
0.33578366 0.1983147
0.33838848 0.1990353
0.34806194 0.1960416
0.35296958 0.1956724
0.34350593 0.2163352
0.33812976 0.2106004

One-way ANOVA

Null hypothesis (Ho): There is no difference in the model means UCOTREC = UGNRWW = UTAGNN = UFLCSP = UGGNN
Alternative hypothesis (H1): At least one model's mean differs from the others
Significance level = 0.05

Summary
Groups Count Sum Average Variance
COTREC 34 11.48851541 0.3378975119 0.000009992151219
GNRRW 50 17.04070195 0.340814039 0.00003779439459
TAGNN 25 6.406298825 0.256251953 0.000229300123
FLCSP 50 10.15718027 0.2031436054 0.00008132955373
GGNN 35 11.29520972 0.3227202778 0.00008863380708
ANOVA
Source of Variation SS df MS F-value P-value F(critical)
Between Groups 0.650167345 4 0.1625418362 2092.16244 0 2.419440031
Within Groups 0.01468356685 189 0.0000776908299
Total 0.6648509118 193

The F-value and critical F- value are 2092.162 and 2.4194, respectively. The F-value is greater than the critical F-value, therefore, we reject the Ho. The main purpose of the ANOVA test is to observe the variability between the groups/compared models. However, it does not explain about the magnitude of variability. Therefore, we use the effect size (η) to know the magnitude of variability between the models. The effect size uses the following table to explain the level of variability.


η Size
0.01 small
0.09 medium
0.25 large
Note: if less than .01, no effect is reported

η = 0.650167344986137 / 0.664850911836957
η = 0.9779
Here we observe the large effect size between the GNN models.

References:

  1. https://open.maricopa.edu/psy230mm/chapter/chapter-14-analysis-of-variance/
  2. https://en.wikipedia.org/wiki/Analysis_of_variance