ceph tapasztalatok #3

sikerult osszeloni, hogy a nova, a cinder es a glance is rbd backenddel dolgozzon. fellottem egy virtualis gepet 12.04-el, nezzunk egy fio teszt eredmenyt: 


root@testubi:~# fio fio
seq-read: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=4
rand-read: (g=1): rw=randread, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=4
seq-write: (g=2): rw=write, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=4
rand-write: (g=3): rw=randwrite, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=4
fio 1.59
Starting 4 processes
Jobs: 1 (f=1): [___w] [100.0% done] [0K/203.6M /s] [0 /198  iops] [eta 00m:00s]
seq-read: (groupid=0, jobs=1): err= 0: pid=3736
  read : io=8192.0MB, bw=2066.7MB/s, iops=2066 , runt=  3964msec
    slat (usec): min=50 , max=527 , avg=70.49, stdev=19.65
    clat (usec): min=1019 , max=8543 , avg=1860.89, stdev=524.81
     lat (usec): min=1077 , max=8652 , avg=1932.05, stdev=523.12
    bw (KB/s) : min=1882112, max=2572288, per=101.96%, avg=2157714.29, stdev=268610.53
  cpu          : usr=0.81%, sys=17.46%, ctx=7687, majf=0, minf=1047
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=8192/0/0, short=0/0/0

     lat (msec): 2=67.46%, 4=31.59%, 10=0.95%
rand-read: (groupid=1, jobs=1): err= 0: pid=3737
  read : io=8192.0MB, bw=1824.6MB/s, iops=1824 , runt=  4490msec
    slat (usec): min=51 , max=438 , avg=68.23, stdev=14.27
    clat (usec): min=1346 , max=4898 , avg=2119.71, stdev=348.83
     lat (usec): min=1410 , max=5005 , avg=2188.56, stdev=348.99
    bw (KB/s) : min=1722368, max=2007040, per=100.62%, avg=1879808.00, stdev=99614.50
  cpu          : usr=1.34%, sys=14.52%, ctx=8186, majf=0, minf=1046
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=8192/0/0, short=0/0/0

     lat (msec): 2=48.74%, 4=50.74%, 10=0.51%
seq-write: (groupid=2, jobs=1): err= 0: pid=3738
  write: io=8192.0MB, bw=180114KB/s, iops=175 , runt= 46574msec
    slat (usec): min=76 , max=1375 , avg=147.83, stdev=36.58
    clat (msec): min=8 , max=1298 , avg=22.58, stdev=42.12
     lat (msec): min=8 , max=1298 , avg=22.73, stdev=42.12
    bw (KB/s) : min=24232, max=216966, per=104.78%, avg=188721.33, stdev=31671.36
  cpu          : usr=1.74%, sys=1.39%, ctx=5098, majf=0, minf=21
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/8192/0, short=0/0/0

     lat (msec): 10=1.53%, 20=50.59%, 50=46.48%, 100=1.06%, 250=0.15%
     lat (msec): 500=0.10%, 2000=0.10%
rand-write: (groupid=3, jobs=1): err= 0: pid=3750
  write: io=8192.0MB, bw=192342KB/s, iops=187 , runt= 43613msec
    slat (usec): min=76 , max=1064 , avg=151.50, stdev=45.75
    clat (msec): min=8 , max=1565 , avg=21.13, stdev=38.75
     lat (msec): min=8 , max=1565 , avg=21.29, stdev=38.74
    bw (KB/s) : min= 9351, max=224632, per=102.92%, avg=197955.64, stdev=37704.16
  cpu          : usr=1.93%, sys=1.48%, ctx=5555, majf=0, minf=19
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/8192/0, short=0/0/0

     lat (msec): 10=1.16%, 20=54.10%, 50=43.53%, 100=0.95%, 250=0.09%
     lat (msec): 500=0.07%, 750=0.05%, 2000=0.05%

Run status group 0 (all jobs):
   READ: io=8192.0MB, aggrb=2066.7MB/s, minb=2116.3MB/s, maxb=2116.3MB/s, mint=3964msec, maxt=3964msec

Run status group 1 (all jobs):
   READ: io=8192.0MB, aggrb=1824.6MB/s, minb=1868.3MB/s, maxb=1868.3MB/s, mint=4490msec, maxt=4490msec

Run status group 2 (all jobs):
  WRITE: io=8192.0MB, aggrb=180113KB/s, minb=184436KB/s, maxb=184436KB/s, mint=46574msec, maxt=46574msec

Run status group 3 (all jobs):
  WRITE: io=8192.0MB, aggrb=192341KB/s, minb=196958KB/s, maxb=196958KB/s, mint=43613msec, maxt=43613msec

Disk stats (read/write):
  vdb: ios=32929/32708, merge=0/0, ticks=62516/711472, in_queue=773968, util=99.29%

nem rossz, nem rossz. nyilvan a 8G befer a journalokba (10G-s journalt hasznalok), szoval nezzuk elosszor egy timeout=60-al, hogy mennyi adat megy ki egy perc alatt: 


root@testubi:~# fio fio
seq-read: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=4
rand-read: (g=1): rw=randread, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=4
seq-write: (g=2): rw=write, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=4
rand-write: (g=3): rw=randwrite, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=4
fio 1.59
Starting 4 processes
Jobs: 1 (f=1): [___w] [61.8% done] [0K/230.2M /s] [0 /224  iops] [eta 01m:40s]     
seq-read: (groupid=0, jobs=1): err= 0: pid=3786
  read : io=32768MB, bw=1642.2MB/s, iops=1642 , runt= 19954msec
    slat (usec): min=49 , max=1668 , avg=72.86, stdev=26.53
    clat (usec): min=987 , max=11126 , avg=2358.89, stdev=1449.12
     lat (msec): min=1 , max=11 , avg= 2.43, stdev= 1.45
    bw (KB/s) : min=833916, max=2740224, per=99.74%, avg=1677183.46, stdev=791508.54
  cpu          : usr=0.90%, sys=14.23%, ctx=30038, majf=0, minf=1047
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=32768/0/0, short=0/0/0
     lat (usec): 1000=0.01%
     lat (msec): 2=64.84%, 4=19.69%, 10=15.44%, 20=0.02%
rand-read: (groupid=1, jobs=1): err= 0: pid=3787
  read : io=32768MB, bw=1609.4MB/s, iops=1609 , runt= 20365msec
    slat (usec): min=49 , max=1295 , avg=73.41, stdev=30.20
    clat (usec): min=912 , max=7423 , avg=2407.90, stdev=968.42
     lat (usec): min=974 , max=7486 , avg=2482.08, stdev=968.57
    bw (KB/s) : min=1400832, max=1888574, per=100.05%, avg=1648545.13, stdev=114633.20
  cpu          : usr=1.36%, sys=13.87%, ctx=31387, majf=0, minf=1046
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=32768/0/0, short=0/0/0
     lat (usec): 1000=0.03%
     lat (msec): 2=47.79%, 4=44.10%, 10=8.08%
seq-write: (groupid=2, jobs=1): err= 0: pid=3798
  write: io=11446MB, bw=195273KB/s, iops=190 , runt= 60022msec
    slat (usec): min=76 , max=962 , avg=146.29, stdev=36.02
    clat (msec): min=8 , max=865 , avg=20.82, stdev=21.59
     lat (msec): min=8 , max=865 , avg=20.97, stdev=21.59
    bw (KB/s) : min=44433, max=225972, per=101.44%, avg=198085.69, stdev=25975.54
  cpu          : usr=1.87%, sys=1.53%, ctx=7510, majf=0, minf=21
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/11446/0, short=0/0/0

     lat (msec): 10=1.78%, 20=54.81%, 50=42.31%, 100=0.95%, 500=0.10%
     lat (msec): 1000=0.03%
rand-write: (groupid=3, jobs=1): err= 0: pid=3810
  write: io=11778MB, bw=200974KB/s, iops=196 , runt= 60011msec
    slat (usec): min=78 , max=1041 , avg=151.55, stdev=42.61
    clat (msec): min=7 , max=1732 , avg=20.22, stdev=50.51
     lat (msec): min=8 , max=1732 , avg=20.37, stdev=50.51
    bw (KB/s) : min= 3052, max=253952, per=105.60%, avg=212231.16, stdev=51460.14
  cpu          : usr=1.97%, sys=1.63%, ctx=9746, majf=0, minf=19
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/11778/0, short=0/0/0

     lat (msec): 10=1.90%, 20=71.88%, 50=24.22%, 100=1.47%, 250=0.25%
     lat (msec): 500=0.07%, 750=0.08%, 1000=0.06%, 2000=0.07%

Run status group 0 (all jobs):
   READ: io=32768MB, aggrb=1642.2MB/s, minb=1681.7MB/s, maxb=1681.7MB/s, mint=19954msec, maxt=19954msec

Run status group 1 (all jobs):
   READ: io=32768MB, aggrb=1609.4MB/s, minb=1647.7MB/s, maxb=1647.7MB/s, mint=20365msec, maxt=20365msec

Run status group 2 (all jobs):
  WRITE: io=11446MB, aggrb=195273KB/s, minb=199960KB/s, maxb=199960KB/s, mint=60022msec, maxt=60022msec

Run status group 3 (all jobs):
  WRITE: io=11778MB, aggrb=200974KB/s, minb=205797KB/s, maxb=205797KB/s, mint=60011msec, maxt=60011msec

Disk stats (read/write):
  vdb: ios=131233/46418, merge=0/0, ticks=288636/948700, in_queue=1237136, util=99.61%
root@testubi:~#

majd nezzuk meg timeout nelkul, 32GB adattal: 


root@testubi:~# fio fio
seq-read: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=4
rand-read: (g=1): rw=randread, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=4
seq-write: (g=2): rw=write, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=4
rand-write: (g=3): rw=randwrite, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=4
fio 1.59
Starting 4 processes
Jobs: 1 (f=1): [___w] [100.0% done] [0K/255.8M /s] [0 /249  iops] [eta 00m:00s]   
seq-read: (groupid=0, jobs=1): err= 0: pid=3946
  read : io=32768MB, bw=974400KB/s, iops=951 , runt= 34436msec
    slat (usec): min=51 , max=2002 , avg=74.29, stdev=28.65
    clat (msec): min=1 , max=11 , avg= 4.12, stdev= 1.40
     lat (msec): min=1 , max=11 , avg= 4.20, stdev= 1.40
    bw (KB/s) : min=817152, max=1077248, per=99.98%, avg=974235.57, stdev=73128.49
  cpu          : usr=0.85%, sys=8.34%, ctx=32202, majf=0, minf=1047
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=32768/0/0, short=0/0/0

     lat (msec): 2=7.72%, 4=42.17%, 10=50.08%, 20=0.02%
rand-read: (groupid=1, jobs=1): err= 0: pid=3957
  read : io=32768MB, bw=1138.6MB/s, iops=1138 , runt= 28782msec
    slat (usec): min=49 , max=3387 , avg=79.32, stdev=39.72
    clat (usec): min=227 , max=11197 , avg=3428.76, stdev=1002.90
     lat (msec): min=1 , max=11 , avg= 3.51, stdev= 1.00
    bw (KB/s) : min=1067008, max=1220608, per=100.05%, avg=1166453.95, stdev=38853.47
  cpu          : usr=1.17%, sys=10.67%, ctx=31564, majf=0, minf=1046
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=32768/0/0, short=0/0/0
     lat (usec): 250=0.01%, 750=0.01%
     lat (msec): 2=13.41%, 4=59.25%, 10=27.33%, 20=0.01%
seq-write: (groupid=2, jobs=1): err= 0: pid=3968
  write: io=32768MB, bw=205487KB/s, iops=200 , runt=163292msec
    slat (usec): min=75 , max=1062 , avg=146.54, stdev=38.05
    clat (msec): min=8 , max=1353 , avg=19.78, stdev=22.10
     lat (msec): min=8 , max=1353 , avg=19.93, stdev=22.10
    bw (KB/s) : min= 5247, max=233472, per=101.25%, avg=208056.46, stdev=26633.32
  cpu          : usr=1.90%, sys=1.71%, ctx=23351, majf=0, minf=21
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/32768/0, short=0/0/0

     lat (msec): 10=2.60%, 20=64.85%, 50=31.51%, 100=0.81%, 250=0.09%
     lat (msec): 500=0.11%, 750=0.01%, 1000=0.01%, 2000=0.01%
rand-write: (groupid=3, jobs=1): err= 0: pid=4000
  write: io=32768MB, bw=196701KB/s, iops=192 , runt=170586msec
    slat (usec): min=78 , max=1194 , avg=148.41, stdev=46.38
    clat (msec): min=8 , max=3138 , avg=20.67, stdev=55.44
     lat (msec): min=8 , max=3138 , avg=20.82, stdev=55.44
    bw (KB/s) : min= 1324, max=263664, per=104.89%, avg=206323.46, stdev=64765.63
  cpu          : usr=1.81%, sys=1.71%, ctx=31522, majf=0, minf=19
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/32768/0, short=0/0/0

     lat (msec): 10=1.91%, 20=84.31%, 50=10.46%, 100=1.97%, 250=1.02%
     lat (msec): 500=0.20%, 750=0.05%, 1000=0.01%, 2000=0.07%, >=2000=0.01%

Run status group 0 (all jobs):
   READ: io=32768MB, aggrb=974399KB/s, minb=997785KB/s, maxb=997785KB/s, mint=34436msec, maxt=34436msec

Run status group 1 (all jobs):
   READ: io=32768MB, aggrb=1138.6MB/s, minb=1165.9MB/s, maxb=1165.9MB/s, mint=28782msec, maxt=28782msec

Run status group 2 (all jobs):
  WRITE: io=32768MB, aggrb=205487KB/s, minb=210418KB/s, maxb=210418KB/s, mint=163292msec, maxt=163292msec

Run status group 3 (all jobs):
  WRITE: io=32768MB, aggrb=196700KB/s, minb=201421KB/s, maxb=201421KB/s, mint=170586msec, maxt=170586msec

Disk stats (read/write):
  vdb: ios=131233/131002, merge=0/0, ticks=452192/2640972, in_queue=3092892, util=99.94%
root@testubi:~#

a 200MB/s-et azert nem ertem picit, mert az SSD-k terhelese az iostat alapjan sosem ment ~85% fole.

kar, hogy nem tudok latency-grafikont rajzolni (fioval lehet ilyet?), kivancsi lennek, hogy valtozik, amikor elerem a journal meretet.

egyelore ennyi, ha minden jol megy, jovohettol tudok patcheket bekuldeni openstackhez, van par otletem, amit a dashboardon lehetne valtoztatni.

( petya | 2013. 08. 18., v – 22:38 )

Sysbench fileio tud second granularity throughputot es latencyt mutatni neked, launchpadrol hasznalj trunk sysbenchet. Ha gondolod segitek szivesen.