QSFP+/ QDR connectx 2/3 pcie gen2/3:
Ez meg nem hasznal FEC -et, a FEC noveli a latencyt.
Hany meteres a DAC kabeled ? FEC mar kotelezo -e rajta ?
2m jofele kabelnel FEC nem kell.
Egyesek allitjak, hogy 3m -nel is ki lehet kapcsolni:
https://blog.mellanox.com/2018/04/mellanox-ofc-2018-mellanox-live-demo-…
ill. hogy 2x250ns lehet az RS-FEC latency.
Mellanox cuccok tipikusan tudnak mas FEC -et pl. Fire code ami kicsit gyorsabb.
QSFP28 -nal 3m no-fec (26AWG) -et tobben is allitjak hogy OK.
#ib_write_lat
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 1000 0.81 5.94 0.88 0.91 0.16 1.10 5.94
4 1000 0.81 6.51 0.88 0.90 0.17 1.11 6.51
8 1000 0.83 7.21 0.88 0.91 0.20 1.10 7.21
16 1000 0.82 8.42 0.89 0.91 0.18 1.14 8.42
32 1000 0.85 6.67 0.94 0.96 0.21 1.15 6.67
64 1000 0.90 7.93 0.97 0.99 0.24 1.18 7.93
128 1000 1.03 8.39 1.09 1.11 0.25 1.30 8.39
256 1000 1.96 8.65 2.03 2.08 0.26 2.35 8.65
512 1000 2.19 7.94 2.28 2.33 0.31 2.84 7.94
1024 1000 2.58 8.75 2.67 2.71 0.30 3.67 8.75
2048 1000 3.40 8.41 3.49 3.54 0.29 5.37 8.41
4096 1000 4.04 10.03 4.17 4.25 0.37 5.94 10.03
8192 1000 5.13 10.09 5.46 5.55 0.30 6.84 10.09
16384 1000 7.58 14.16 7.91 7.99 0.49 11.23 14.16
32768 1000 12.57 18.12 12.96 13.00 0.38 15.29 18.12
65536 1000 22.87 32.33 23.05 23.09 0.20 23.63 32.33
131072 1000 43.04 48.95 43.28 43.45 0.66 47.03 48.95
262144 1000 83.42 87.29 83.64 83.69 0.26 84.20 87.29
524288 1000 164.14 169.87 164.45 164.49 0.27 165.15 169.87
1048576 1000 325.65 329.35 326.03 326.08 0.30 327.62 329.35
2097152 1000 648.70 656.20 649.54 649.67 0.72 653.58 656.20
4194304 1000 1294.62 1304.04 1296.07 1296.17 0.71 1298.89 1304.04
8388608 1000 2613.64 2642.17 2631.10 2631.24 1.44 2636.03 2642.17
# ib_read_lat
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 1000 1.70 5.88 1.75 1.79 0.16 2.23 5.88
4 1000 1.71 5.95 1.76 1.81 0.14 2.24 5.95
8 1000 1.72 4.62 1.78 1.81 0.10 2.23 4.62
16 1000 1.70 7.95 1.73 1.77 0.12 2.19 7.95
32 1000 1.71 5.31 1.74 1.78 0.12 2.15 5.31
64 1000 1.67 7.11 1.76 1.80 0.12 2.19 7.11
128 1000 1.78 9.94 1.82 1.86 0.13 2.34 9.94
256 1000 1.88 6.63 1.93 1.97 0.11 2.37 6.63
512 1000 2.13 6.20 2.18 2.23 0.14 2.64 6.20
1024 1000 2.53 9.66 2.57 2.62 0.22 3.01 9.66
2048 1000 3.33 10.38 3.38 3.42 0.18 3.86 10.38
4096 1000 3.94 11.33 4.01 4.05 0.33 4.49 11.33
8192 1000 5.16 12.39 5.23 5.33 0.35 6.05 12.39
16384 1000 7.43 14.33 7.68 7.81 0.25 8.56 14.33
32768 1000 12.30 18.21 12.79 12.74 0.37 13.57 18.21
65536 1000 22.47 27.71 22.55 22.61 0.17 23.06 27.71
131072 1000 42.44 50.41 42.58 42.67 0.40 43.32 50.41
262144 1000 82.30 88.90 82.45 82.52 0.37 83.19 88.90
524288 1000 162.13 168.88 162.28 162.36 0.39 163.20 168.88
1048576 1000 321.65 328.88 321.93 321.99 0.38 322.74 328.88
2097152 1000 640.69 646.62 641.29 641.33 0.36 643.05 646.62
4194304 1000 1278.99 1307.72 1279.90 1279.99 0.92 1282.17 1307.72
8388608 1000 2555.37 2567.06 2557.46 2557.62 0.95 2560.52 2567.06
A valasz idok jobbnak tunnek a kisebb csomagoknal, de nagyobb csomagoknal a bandwith szamit.
ib_read_bw
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
2 1000 6.04 5.91 3.099655
4 1000 12.39 12.23 3.206630
8 1000 24.56 24.11 3.160711
16 1000 49.51 48.64 3.187653
32 1000 97.76 95.79 3.138737
64 1000 208.36 194.84 3.192182
128 1000 379.81 375.05 3.072390
256 1000 776.82 766.43 3.139281
512 1000 1529.26 1500.69 3.073417
1024 1000 2907.27 2857.69 2.926272
2048 1000 3099.51 3092.07 1.583142
4096 1000 3118.43 3116.47 0.797816
8192 1000 3124.32 3122.52 0.399682
16384 1000 3125.64 3125.57 0.200036
32768 1000 3131.76 3131.70 0.100214
65536 1000 3131.81 3131.81 0.050109
131072 1000 3132.74 3132.74 0.025062
262144 1000 3132.77 3132.66 0.012531
524288 1000 3132.43 3132.43 0.006265
1048576 1000 3130.04 3130.01 0.003130
2097152 1000 3130.06 3129.72 0.001565
4194304 1000 3133.08 3132.71 0.000783
8388608 1000 3132.58 3132.58 0.000392
ib_write_bw
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
2 5000 6.44 6.39 3.350616
4 5000 12.84 12.75 3.341600
8 5000 25.91 25.81 3.382843
16 5000 52.09 51.45 3.371902
32 5000 103.72 102.88 3.371066
64 5000 207.24 204.10 3.343947
128 5000 412.64 408.28 3.344640
256 5000 831.96 823.11 3.371450
512 5000 1627.38 1610.81 3.298936
1024 5000 3048.47 3041.68 3.114685
2048 5000 3059.80 3057.74 1.565565
4096 5000 3062.97 3062.34 0.783960
8192 5000 3064.21 3064.10 0.392205
16384 5000 3064.67 3064.62 0.196136
32768 5000 3064.99 3064.96 0.098079
65536 5000 3065.13 3065.09 0.049042
131072 5000 3065.12 3065.11 0.024521
262144 5000 3065.24 3065.20 0.012261
524288 5000 3065.25 3065.22 0.006130
1048576 5000 3065.12 3064.82 0.003065
2097152 5000 3065.22 3065.21 0.001533
4194304 5000 3065.28 3065.24 0.000766
8388608 5000 3065.25 3065.25 0.000383
hometech from ebay ;-)
gep1:
Mellanox Technologies MT25408A0-FCC-QI ConnectX, Dual Port 40Gb/s InfiniBand / 10GigE Adapter IC with PCIe 2.0 x8 5.0GT/s In... (rev b0)
Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
gep2:
Mellanox Technologies MT27500 Family [ConnectX-3]
AMD Ryzen 9 3900X 12-Core Processor
NVMeoF, nfs+rdma kozel az itt lathato BW -el megy, meg az osregi Q9300 -as procival is.
Dual port kartyak, 3 gep directbe (DAC).
(A test allatt cpu intiseve boinc/rosetta futott parhuzamosan)