ceph tapasztalatok #2

nezzunk egy rados benchmarkot, mielott megnezzuk KVM-en belulrol is.

a felallas ez:


root@signina:~# ceph osd tree
# id	weight	type name	up/down	reweight
-1	7.16	root default
-2	4.55		host zc2store
0	0.91			osd.0	up	1	
1	0.91			osd.1	up	1	
2	0.91			osd.2	up	1	
3	0.91			osd.3	up	1	
4	0.91			osd.4	up	1	
-3	2.61		host signina
5	0.45			osd.5	up	1	
6	0.45			osd.6	up	1	
7	0.27			osd.7	up	1	
8	0.27			osd.8	up	1	
9	0.27			osd.9	up	1	
10	0.45			osd.10	up	1	
11	0.45			osd.11	up	1	
root@signina:~# 

mindket nodeon van egy-egy Intel 520-as SSD a journalnak.

nezzuk a rados benchet:


root@signina:~#  rados -p volumes bench 60 write -t 16
 Maintaining 16 concurrent writes of 4194304 bytes for up to 60 seconds or 0 objects
 Object prefix: benchmark_data_signina_23003
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
     0       0         0         0         0         0         -         0
     1      16        75        59    235.91       236  0.236961  0.244561
     2      16       138       122    243.94       252  0.330535  0.252054
     3      16       196       180    239.95       232  0.196866  0.252951
     4      16       263       247   246.954       268  0.161687   0.25204
     5      16       324       308   246.358       244  0.135385  0.251702
     6      16       389       373   248.626       260  0.288089  0.253287
     7      16       451       435   248.532       248  0.126835  0.253278
     8      16       516       500   249.962       260   0.16171  0.252724
     9      16       577       561   249.297       244  0.316671  0.254008
    10      16       637       621   248.365       240  0.184376  0.253623
    11      15       704       689    250.51       272  0.178265   0.25353
    12      16       762       746   248.632       228  0.316239  0.253596
    13      16       825       809   248.888       252  0.181267  0.254503
    14      16       886       870   248.537       244  0.169095  0.254109
    15      16       951       935     249.3       260  0.289912  0.254225
    16      16      1014       998   249.467       252   0.15858  0.254069
    17      16      1075      1059   249.143       244  0.300017   0.25452
    18      16      1142      1126   250.189       268  0.251801  0.254021
    19      16      1205      1189   250.283       252  0.178848  0.254244
2013-08-17 14:35:17.468771min lat: 0.064823 max lat: 0.547082 avg lat: 0.254134
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    20      16      1265      1249   249.767       240  0.327125  0.254134
    21      16      1326      1310   249.465       244   0.12045  0.254108
    22      16      1395      1379   250.669       276  0.285549   0.25436
    23      16      1456      1440   250.378       244  0.243521  0.254197
    24      16      1517      1501   250.111       244  0.158143  0.254331
    25      16      1581      1565   250.345       256   0.26384  0.254318
    26      16      1644      1628   250.408       252  0.219004   0.25407
    27      16      1712      1696   251.206       272  0.288466  0.253976
    28      16      1770      1754   250.519       232   0.30551  0.254079
    29      16      1831      1815   250.293       244  0.131759   0.25416
    30      16      1893      1877   250.215       248  0.160367  0.254161
    31      16      1962      1946   251.045       276  0.306111  0.254152
    32      16      2020      2004   250.449       232   0.20502  0.254466
    33      16      2082      2066   250.374       248  0.347549  0.254303
    34      16      2146      2130   250.539       256  0.320295  0.254419
    35      15      2205      2190   250.237       240   0.29376   0.25415
    36      16      2271      2255   250.507       260  0.298762  0.254514
    37      16      2335      2319   250.655       256  0.308033   0.25451
    38      16      2399      2383   250.795       256  0.190646  0.254352
    39      16      2458      2442   250.415       236  0.177154  0.254671
2013-08-17 14:35:37.473525min lat: 0.064823 max lat: 0.547082 avg lat: 0.254501
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    40      16      2524      2508   250.754       264  0.142427  0.254501
    41      16      2584      2568   250.491       240  0.360782  0.254454
    42      16      2650      2634   250.812       264   0.30188   0.25438
    43      16      2713      2697   250.838       252  0.336351  0.254456
    44      16      2774      2758   250.682       244  0.242157  0.254297
    45      16      2841      2825   251.067       268  0.279732  0.254369
    46      16      2904      2888   251.086       252  0.274434  0.254282
    47      16      2964      2948    250.85       240  0.231653  0.254426
    48      16      3032      3016    251.29       272  0.301616  0.254398
    49      16      3093      3077    251.14       244  0.311436  0.254347
    50      16      3157      3141   251.237       256  0.239624   0.25431
    51      16      3215      3199   250.859       232  0.307918  0.254335
    52      16      3280      3264   251.034       260  0.226089  0.254313
    53      16      3343      3327   251.052       252  0.293137  0.254339
    54      16      3404      3388    250.92       244  0.258462  0.254436
    55      16      3467      3451    250.94       252  0.136508  0.254342
    56      16      3533      3517   251.172       264  0.204906  0.254422
    57      16      3592      3576   250.906       236  0.262631   0.25435
    58      15      3655      3640   250.993       256   0.22725   0.25436
    59      16      3718      3702   250.942       248   0.28189  0.254354
2013-08-17 14:35:57.476035min lat: 0.064823 max lat: 0.547082 avg lat: 0.25456
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    60      16      3781      3765   250.959       252  0.331387   0.25456
 Total time run:         60.196061
Total writes made:      3782
Write size:             4194304
Bandwidth (MB/sec):     251.312 

Stddev Bandwidth:       34.2316
Max bandwidth (MB/sec): 276
Min bandwidth (MB/sec): 0
Average Latency:        0.254477
Stddev Latency:         0.0699765
Max latency:            0.547082
Min latency:            0.064823
root@signina:~#

neztem kozben a nodeokon, hogy sajnos azert ennyi a write, mert a 7 diszkes gepen elfogy a journal terhelhetosege, es 100% terhelest mutat. lehet, hogy kiprobalom, hogy kiveszem az egyik diszket, es berakok megegy SSD-t a helyere, majd a maradek 6 diszket 3-3 alapon szetosztom kozottuk.

ha megmaradok az egy SSD + 3 OSD kombonal, a write akkor is felmegy ~400MB korulire, de akkor meg bukok eleg sok helyet. jelenleg a 250MB/s eleg lenne, szoval nem akkora problema.