NVMe over Fabrics - RDMA

raw fio output:

filename1: (groupid=0, jobs=8): err= 0: pid=8545: Thu May 16 14:09:01 2019
  read: IOPS=1872k, BW=7312MiB/s (7667MB/s)(428GiB/60001msec)
    slat (nsec): min=242, max=168565, avg=281.83, stdev=51.02
    clat (usec): min=22, max=135616, avg=136.22, stdev=1508.77
     lat (usec): min=23, max=135617, avg=136.50, stdev=1508.77
    clat percentiles (usec):
     |  1.00th=[    75],  5.00th=[    80], 10.00th=[    83], 20.00th=[   113],
     | 30.00th=[   119], 40.00th=[   122], 50.00th=[   124], 60.00th=[   126],
     | 70.00th=[   128], 80.00th=[   130], 90.00th=[   135], 95.00th=[   157],
     | 99.00th=[   178], 99.50th=[   182], 99.90th=[   192], 99.95th=[   208],
     | 99.99th=[133694]
   bw (  MiB/s): min= 7274, max= 7324, per=99.99%, avg=7311.34, stdev= 1.18, samples=954
   iops        : min=1862245, max=1874976, avg=1871701.48, stdev=301.79, samples=954
  lat (usec)   : 50=0.01%, 100=18.27%, 250=81.71%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=200, majf=0, minf=284
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=2.0%, 32=98.0%, >=64=0.0%
[global]
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=112315747,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

filename1: (groupid=0, jobs=8): err= 0: pid=11254: Thu May 16 14:14:20 2019
  write: IOPS=2313k, BW=9034MiB/s (9473MB/s)(529GiB/60001msec); 0 zone resets
    slat (nsec): min=221, max=170666, avg=315.19, stdev=56.51
    clat (usec): min=20, max=111830, avg=110.15, stdev=1220.54
     lat (usec): min=21, max=111830, avg=110.46, stdev=1220.54
    clat percentiles (usec):
     |  1.00th=[    69],  5.00th=[    71], 10.00th=[    72], 20.00th=[    93],
     | 30.00th=[    98], 40.00th=[    99], 50.00th=[   101], 60.00th=[   102],
     | 70.00th=[   104], 80.00th=[   105], 90.00th=[   108], 95.00th=[   109],
     | 99.00th=[   113], 99.50th=[   114], 99.90th=[   119], 99.95th=[   122],
     | 99.99th=[108528]
   bw (  MiB/s): min= 8978, max= 9070, per=99.96%, avg=9031.24, stdev= 2.22, samples=952
   iops        : min=2298590, max=2322080, avg=2311997.15, stdev=567.50, samples=952
  lat (usec)   : 50=0.01%, 100=45.48%, 250=54.51%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
  cpu          : usr=100.00%, sys=0.00%, ctx=161, majf=0, minf=284
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.5%, 32=99.5%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,138771317,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32