На блочномом устройстве, подключенном по multipath создан lvm и отдан под drbd. Поднимается ha + heartbeat кластер, после чего имитируется сбой multipath устройста. После чего кластер продолжает работать в обычном режиме. Причем, на главной ноде исправно работает доступ к примонтрованной файловой системе drbd диска.
В чем подвох? Я ожидал , что сбой multipath диска приведет к сбою работы drbd.
LinuxForum.Fans
drbd + multipath
Автор: Macumazan
6 сообщений
2724 просмотров
В чем подвох? Я ожидал , что сбой multipath диска приведет к сбою работы drbd.
Здесь надо смотреть журнал системы(логов во время теста) на наличие ошибок и тд, так трудно что либо сказать. Да и конфигурацю не мешало бы увидеть. Что у вас там :)
[quote="Macumazan⇓"]В чем подвох? Я ожидал , что сбой multipath диска приведет к сбою работы drbd.
Здесь надо смотреть журнал системы(логов во время теста) на наличие ошибок и тд, так трудно что либо сказать. Да и конфигурацю не мешало бы увидеть. Что у вас там :)[/quote]
Логи так логи)
Имитация сбоя устройства
text
Mar 22 10:10:17 g8-3 kernel: [42434.766145] usb 1-1.5: USB disconnect, device number 5
Mar 22 10:10:18 g8-3 kernel: [42435.144030] mpt2sas0: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
Mar 22 10:10:18 g8-3 kernel: [42435.155471] mpt2sas0: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
Mar 22 10:10:18 g8-3 kernel: [42435.166962] mpt2sas0: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
Mar 22 10:10:18 g8-3 kernel: [42435.178476] mpt2sas0: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
Mar 22 10:10:18 g8-3 kernel: [42435.189946] mpt2sas0: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
Mar 22 10:10:18 g8-3 kernel: [42435.201424] mpt2sas0: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
Mar 22 10:10:18 g8-3 kernel: [42435.212901] mpt2sas0: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
Mar 22 10:10:18 g8-3 kernel: [42435.224389] mpt2sas0: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
Mar 22 10:10:18 g8-3 kernel: [42435.235863] mpt2sas0: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
Mar 22 10:10:18 g8-3 kernel: [42435.247335] mpt2sas0: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
Mar 22 10:10:48 g8-3 multipathd: sdf: remove path (uevent)
Mar 22 10:10:48 g8-3 multipathd: 3600c0ff00014fc48204b405101000000: load table [0 1170637568 multipath 0 0 3 1 round-robin 0 1 1 8:112 1000 round-robin 0 1 1 8:
Mar 22 10:10:48 g8-3 kernel: [42465.550898] sd 5:0:0:1: [sdf] Synchronizing SCSI cache
Mar 22 10:10:48 g8-3 kernel: [42465.551030] sd 5:0:0:1: [sdf] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:48 g8-3 kernel: [42465.552497] sd 5:0:0:2: [sdg] Synchronizing SCSI cache
Mar 22 10:10:48 g8-3 kernel: [42465.552666] sd 5:0:0:2: [sdg] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:48 g8-3 kernel: [42465.552894] mpt2sas1: removing handle(0x0009), sas_addr(0x500c0ff139468404)
Mar 22 10:10:48 g8-3 multipathd: sdf: path removed from map 3600c0ff00014fc48204b405101000000
Mar 22 10:10:48 g8-3 multipathd: sdg: remove path (uevent)
Mar 22 10:10:48 g8-3 multipathd: 3600c0ff00014fd09f95e405101000000: load table [0 1170637568 multipath 0 0 3 1 round-robin 0 1 1 8:32 1000 round-robin 0 1 1 8:6
Mar 22 10:10:48 g8-3 multipathd: sdg: path removed from map 3600c0ff00014fd09f95e405101000000
Mar 22 10:10:48 g8-3 multipathd: 8:32: mark as failed
Mar 22 10:10:48 g8-3 multipathd: 3600c0ff00014fd09f95e405101000000: remaining active paths: 2
Mar 22 10:10:49 g8-3 multipathd: sdb: unusable path
Mar 22 10:10:49 g8-3 multipathd: 3600c0ff00014fd09f95e405101000000: sdc - directio checker reports path is down
Mar 22 10:10:49 g8-3 multipathd: 3600c0ff00014fd09f95e405101000000: sde - directio checker reports path is down
Mar 22 10:10:49 g8-3 multipathd: checker failed path 8:64 in map 3600c0ff00014fd09f95e405101000000
Mar 22 10:10:49 g8-3 multipathd: 3600c0ff00014fd09f95e405101000000: remaining active paths: 1
Mar 22 10:10:49 g8-3 multipathd: sdh: unusable path
Mar 22 10:10:49 g8-3 multipathd: 3600c0ff00014fd09f95e405101000000: sdi - directio checker reports path is down
Mar 22 10:10:49 g8-3 multipathd: checker failed path 8:128 in map 3600c0ff00014fd09f95e405101000000
Mar 22 10:10:49 g8-3 multipathd: 3600c0ff00014fd09f95e405101000000: remaining active paths: 0
Mar 22 10:10:52 g8-3 kernel: [42465.555899] sd 5:0:1:1: [sdh] Unhandled error code
Mar 22 10:10:52 g8-3 kernel: [42465.555902] sd 5:0:1:1: [sdh] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:52 g8-3 kernel: [42465.555905] sd 5:0:1:1: [sdh] CDB: Read(10): 28 00 45 c6 82 80 00 00 08 00
Mar 22 10:10:52 g8-3 kernel: [42465.555912] end_request: I/O error, dev sdh, sector 1170637440
Mar 22 10:10:52 g8-3 kernel: [42465.556118] sd 3:0:0:2: [sdc] Unhandled error code
Mar 22 10:10:52 g8-3 kernel: [42465.556121] sd 3:0:0:2: [sdc] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:52 g8-3 kernel: [42465.556123] sd 3:0:0:2: [sdc] CDB: Read(10): 28 00 45 c6 82 80 00 00 08 00
Mar 22 10:10:52 g8-3 kernel: [42465.556128] end_request: I/O error, dev sdc, sector 1170637440
Mar 22 10:10:52 g8-3 kernel: [42465.556132] device-mapper: multipath: Failing path 8:32.
Mar 22 10:10:52 g8-3 kernel: [42465.581155] sd 3:0:1:2: [sde] Unhandled error code
Mar 22 10:10:52 g8-3 kernel: [42465.581157] sd 3:0:1:2: [sde] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:52 g8-3 kernel: [42465.581159] sd 3:0:1:2: [sde] CDB: Read(10): 28 00 45 c6 82 80 00 00 08 00
Mar 22 10:10:52 g8-3 kernel: [42465.581164] end_request: I/O error, dev sde, sector 1170637440
Mar 22 10:10:52 g8-3 kernel: [42465.581168] device-mapper: multipath: Failing path 8:64.
Mar 22 10:10:52 g8-3 kernel: [42465.609141] sd 5:0:1:2: [sdi] Unhandled error code
Mar 22 10:10:52 g8-3 kernel: [42465.609143] sd 5:0:1:2: [sdi] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:52 g8-3 kernel: [42465.609145] sd 5:0:1:2: [sdi] CDB: Read(10): 28 00 45 c6 82 80 00 00 08 00
Mar 22 10:10:52 g8-3 kernel: [42465.609150] end_request: I/O error, dev sdi, sector 1170637440
Mar 22 10:10:52 g8-3 kernel: [42465.609154] device-mapper: multipath: Failing path 8:128.
Mar 22 10:10:52 g8-3 kernel: [42465.609158] end_request: I/O error, dev dm-3, sector 1170637440
Mar 22 10:10:52 g8-3 kernel: [42465.609160] Buffer I/O error on device dm-3, logical block 146329680
Mar 22 10:10:52 g8-3 kernel: [42465.609229] end_request: I/O error, dev dm-3, sector 1170637440
Mar 22 10:10:52 g8-3 kernel: [42465.609231] Buffer I/O error on device dm-3, logical block 146329680
Mar 22 10:10:52 g8-3 kernel: [42465.609300] end_request: I/O error, dev dm-3, sector 1170637552
Mar 22 10:10:52 g8-3 kernel: [42465.609302] Buffer I/O error on device dm-3, logical block 146329694
Mar 22 10:10:52 g8-3 kernel: [42465.609351] end_request: I/O error, dev dm-3, sector 1170637552
Mar 22 10:10:52 g8-3 kernel: [42465.609353] Buffer I/O error on device dm-3, logical block 146329694
Mar 22 10:10:52 g8-3 kernel: [42465.609411] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609414] Buffer I/O error on device dm-3, logical block 0
Mar 22 10:10:52 g8-3 kernel: [42465.609464] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609467] Buffer I/O error on device dm-3, logical block 0
Mar 22 10:10:52 g8-3 kernel: [42465.609516] end_request: I/O error, dev dm-3, sector 8
Mar 22 10:10:52 g8-3 kernel: [42465.609519] Buffer I/O error on device dm-3, logical block 1
Mar 22 10:10:52 g8-3 kernel: [42465.609541] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:52 g8-3 kernel: [42465.609542] Buffer I/O error on device dm-3, logical block 146329695
Mar 22 10:10:52 g8-3 kernel: [42465.609627] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:52 g8-3 kernel: [42465.609628] Buffer I/O error on device dm-3, logical block 146329695
Mar 22 10:10:52 g8-3 kernel: [42465.609640] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:52 g8-3 kernel: [42465.609642] Buffer I/O error on device dm-3, logical block 146329695
Mar 22 10:10:52 g8-3 kernel: [42465.609652] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:52 g8-3 kernel: [42465.609662] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:52 g8-3 kernel: [42465.609673] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:52 g8-3 kernel: [42465.609685] end_request: I/O error, dev dm-3, sector 1170637504
Mar 22 10:10:52 g8-3 kernel: [42465.609696] end_request: I/O error, dev dm-3, sector 1170637552
Mar 22 10:10:52 g8-3 kernel: [42465.609709] end_request: I/O error, dev dm-3, sector 8
Mar 22 10:10:52 g8-3 kernel: [42465.609719] end_request: I/O error, dev dm-3, sector 8
Mar 22 10:10:52 g8-3 kernel: [42465.609729] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:52 g8-3 kernel: [42465.609740] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:52 g8-3 kernel: [42465.609752] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:52 g8-3 kernel: [42465.609762] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609772] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609782] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609792] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609802] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609812] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609822] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609834] end_request: I/O error, dev dm-3, sector 2048
Mar 22 10:10:52 g8-3 kernel: [42465.609844] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609854] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609864] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609874] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609883] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609893] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609903] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609913] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609923] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609933] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609943] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609952] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609962] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609972] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.609982] end_request: I/O error, dev dm-3, sector 8
Mar 22 10:10:52 g8-3 kernel: [42465.609992] end_request: I/O error, dev dm-3, sector 8
Mar 22 10:10:52 g8-3 kernel: [42465.610001] end_request: I/O error, dev dm-3, sector 8
Mar 22 10:10:52 g8-3 kernel: [42465.610011] end_request: I/O error, dev dm-3, sector 8
Mar 22 10:10:52 g8-3 kernel: [42465.610022] end_request: I/O error, dev dm-3, sector 24
Mar 22 10:10:52 g8-3 kernel: [42465.610033] end_request: I/O error, dev dm-3, sector 24
Mar 22 10:10:52 g8-3 kernel: [42465.610042] end_request: I/O error, dev dm-3, sector 24
Mar 22 10:10:52 g8-3 multipathd: 8:112: mark as failed
Mar 22 10:10:52 g8-3 kernel: [42465.610052] end_request: I/O error, dev dm-3, sector 24
Mar 22 10:10:52 g8-3 kernel: [42465.610063] end_request: I/O error, dev dm-3, sector 56
Mar 22 10:10:52 g8-3 multipathd: 3600c0ff00014fc48204b405101000000: remaining active paths: 2
Mar 22 10:10:52 g8-3 kernel: [42465.610073] end_request: I/O error, dev dm-3, sector 56
Mar 22 10:10:52 g8-3 kernel: [42465.610083] end_request: I/O error, dev dm-3, sector 56
Mar 22 10:10:52 g8-3 kernel: [42465.610093] end_request: I/O error, dev dm-3, sector 56
Mar 22 10:10:52 g8-3 kernel: [42465.610104] end_request: I/O error, dev dm-3, sector 120
Mar 22 10:10:52 g8-3 kernel: [42465.610114] end_request: I/O error, dev dm-3, sector 120
Mar 22 10:10:52 g8-3 kernel: [42465.610124] end_request: I/O error, dev dm-3, sector 120
Mar 22 10:10:52 g8-3 kernel: [42465.610134] end_request: I/O error, dev dm-3, sector 120
Mar 22 10:10:52 g8-3 kernel: [42465.610144] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610154] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610164] end_request: I/O error, dev dm-3, sector 8
Mar 22 10:10:52 g8-3 kernel: [42465.610174] end_request: I/O error, dev dm-3, sector 8
Mar 22 10:10:52 g8-3 kernel: [42465.610184] end_request: I/O error, dev dm-3, sector 24
Mar 22 10:10:52 g8-3 kernel: [42465.610194] end_request: I/O error, dev dm-3, sector 24
Mar 22 10:10:52 g8-3 kernel: [42465.610204] end_request: I/O error, dev dm-3, sector 56
Mar 22 10:10:52 g8-3 kernel: [42465.610213] end_request: I/O error, dev dm-3, sector 56
Mar 22 10:10:52 g8-3 kernel: [42465.610223] end_request: I/O error, dev dm-3, sector 120
Mar 22 10:10:52 g8-3 kernel: [42465.610233] end_request: I/O error, dev dm-3, sector 120
Mar 22 10:10:52 g8-3 kernel: [42465.610243] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610253] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610263] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610273] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610282] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610292] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610303] end_request: I/O error, dev dm-3, sector 16
Mar 22 10:10:52 g8-3 kernel: [42465.610314] end_request: I/O error, dev dm-3, sector 128
Mar 22 10:10:52 g8-3 kernel: [42465.610324] end_request: I/O error, dev dm-3, sector 128
Mar 22 10:10:52 g8-3 kernel: [42465.610334] end_request: I/O error, dev dm-3, sector 128
Mar 22 10:10:52 g8-3 kernel: [42465.610344] end_request: I/O error, dev dm-3, sector 16
Mar 22 10:10:52 g8-3 kernel: [42465.610354] end_request: I/O error, dev dm-3, sector 128
Mar 22 10:10:52 g8-3 kernel: [42465.610365] end_request: I/O error, dev dm-3, sector 64
Mar 22 10:10:52 g8-3 kernel: [42465.610375] end_request: I/O error, dev dm-3, sector 64
Mar 22 10:10:52 g8-3 kernel: [42465.610385] end_request: I/O error, dev dm-3, sector 64
Mar 22 10:10:52 g8-3 kernel: [42465.610395] end_request: I/O error, dev dm-3, sector 64
Mar 22 10:10:52 g8-3 kernel: [42465.610404] end_request: I/O error, dev dm-3, sector 64
Mar 22 10:10:52 g8-3 kernel: [42465.610414] end_request: I/O error, dev dm-3, sector 64
Mar 22 10:10:52 g8-3 kernel: [42465.610424] end_request: I/O error, dev dm-3, sector 64
Mar 22 10:10:52 g8-3 kernel: [42465.610434] end_request: I/O error, dev dm-3, sector 64
Mar 22 10:10:52 g8-3 kernel: [42465.610444] end_request: I/O error, dev dm-3, sector 64
Mar 22 10:10:52 g8-3 kernel: [42465.610454] end_request: I/O error, dev dm-3, sector 64
Mar 22 10:10:52 g8-3 kernel: [42465.610465] end_request: I/O error, dev dm-3, sector 256
Mar 22 10:10:52 g8-3 kernel: [42465.610476] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610485] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610495] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610505] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610515] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610525] end_request: I/O error, dev dm-3, sector 16
Mar 22 10:10:52 g8-3 kernel: [42465.610536] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610546] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610556] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610565] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610575] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610585] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610595] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610605] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610615] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610625] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610634] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610644] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610654] end_request: I/O error, dev dm-3, sector 128
Mar 22 10:10:52 g8-3 kernel: [42465.610664] end_request: I/O error, dev dm-3, sector 128
Mar 22 10:10:52 g8-3 kernel: [42465.610674] end_request: I/O error, dev dm-3, sector 16
Mar 22 10:10:52 g8-3 kernel: [42465.610684] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610694] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610704] end_request: I/O error, dev dm-3, sector 8
Mar 22 10:10:52 g8-3 kernel: [42465.610714] end_request: I/O error, dev dm-3, sector 16
Mar 22 10:10:52 g8-3 kernel: [42465.610724] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610734] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610744] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610753] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610763] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610773] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610783] end_request: I/O error, dev dm-3, sector 8
Mar 22 10:10:52 g8-3 kernel: [42465.610793] end_request: I/O error, dev dm-3, sector 128
Mar 22 10:10:52 g8-3 kernel: [42465.610803] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610813] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610824] end_request: I/O error, dev dm-3, sector 4096
Mar 22 10:10:52 g8-3 kernel: [42465.610834] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610844] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610854] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610864] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610873] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610883] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610896] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610906] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610916] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610926] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610937] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610948] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610958] end_request: I/O error, dev dm-3, sector 24
Mar 22 10:10:52 g8-3 kernel: [42465.610968] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610978] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610988] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.610998] end_request: I/O error, dev dm-3, sector 56
Mar 22 10:10:52 g8-3 kernel: [42465.611008] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42465.611018] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:52 g8-3 kernel: [42466.138226] sd 3:0:0:2: [sdc] Unhandled error code
Mar 22 10:10:52 g8-3 kernel: [42466.138229] sd 3:0:0:2: [sdc] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:52 g8-3 kernel: [42466.138231] sd 3:0:0:2: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Mar 22 10:10:52 g8-3 kernel: [42466.138235] end_request: I/O error, dev sdc, sector 0
Mar 22 10:10:52 g8-3 kernel: [42466.140915] sd 3:0:1:2: [sde] Unhandled error code
Mar 22 10:10:52 g8-3 kernel: [42466.140918] sd 3:0:1:2: [sde] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:52 g8-3 kernel: [42466.140920] sd 3:0:1:2: [sde] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Mar 22 10:10:52 g8-3 kernel: [42466.140924] end_request: I/O error, dev sde, sector 0
Mar 22 10:10:52 g8-3 kernel: [42466.141105] sd 5:0:1:2: [sdi] Unhandled error code
Mar 22 10:10:52 g8-3 kernel: [42466.141107] sd 5:0:1:2: [sdi] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:52 g8-3 kernel: [42466.141109] sd 5:0:1:2: [sdi] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Mar 22 10:10:52 g8-3 kernel: [42466.141112] end_request: I/O error, dev sdi, sector 0
Mar 22 10:10:52 g8-3 kernel: [42469.956024] device-mapper: multipath: Failing path 8:112.
Mar 22 10:10:52 g8-3 kernel: [42469.956600] sd 3:0:0:1: [sdb] Synchronizing SCSI cache
Mar 22 10:10:52 g8-3 kernel: [42469.956654] sd 3:0:0:1: [sdb] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:52 g8-3 kernel: [42469.956685] sd 5:0:1:1: [sdh] Synchronizing SCSI cache
Mar 22 10:10:52 g8-3 kernel: [42469.956717] sd 5:0:1:1: [sdh] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:52 g8-3 multipathd: sdb: remove path (uevent)
Mar 22 10:10:52 g8-3 kernel: [42469.958824] sd 3:0:0:2: [sdc] Synchronizing SCSI cache
Mar 22 10:10:52 g8-3 kernel: [42469.958847] sd 3:0:0:2: [sdc] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:52 g8-3 kernel: [42469.958960] sd 5:0:1:2: [sdi] Synchronizing SCSI cache
Mar 22 10:10:52 g8-3 kernel: [42469.958994] sd 5:0:1:2: [sdi] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:52 g8-3 kernel: [42469.959104] mpt2sas0: removing handle(0x0009), sas_addr(0x500c0ff139468400)
Mar 22 10:10:57 g8-3 kernel: [42469.959155] sd 3:0:1:1: [sdd] Unhandled error code
Mar 22 10:10:57 g8-3 kernel: [42469.959157] sd 3:0:1:1: [sdd] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:57 g8-3 kernel: [42469.959160] sd 3:0:1:1: [sdd] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Mar 22 10:10:57 g8-3 kernel: [42469.959166] end_request: I/O error, dev sdd, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.959242] mpt2sas1: removing handle(0x000a), sas_addr(0x500c0ff139468004)
Mar 22 10:10:57 g8-3 kernel: [42469.959992] device-mapper: table: 252:4: multipath: error getting device
Mar 22 10:10:57 g8-3 kernel: [42469.959995] device-mapper: ioctl: error adding target to table
Mar 22 10:10:57 g8-3 kernel: [42469.966808] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.966829] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.966847] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42469.966857] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42469.966871] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.966891] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.966907] end_request: I/O error, dev dm-3, sector 24
Mar 22 10:10:57 g8-3 kernel: [42469.971196] device-mapper: multipath: Failing path 8:16.
Mar 22 10:10:57 g8-3 kernel: [42469.999105] device-mapper: multipath: Failing path 8:48.
Mar 22 10:10:57 g8-3 kernel: [42469.999121] end_request: I/O error, dev dm-4, sector 1170637440
Mar 22 10:10:57 g8-3 kernel: [42469.999208] end_request: I/O error, dev dm-4, sector 1170637440
Mar 22 10:10:57 g8-3 kernel: [42469.999240] end_request: I/O error, dev dm-4, sector 1170637552
Mar 22 10:10:57 g8-3 kernel: [42469.999252] end_request: I/O error, dev dm-4, sector 1170637552
Mar 22 10:10:57 g8-3 kernel: [42469.999272] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999282] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999306] end_request: I/O error, dev dm-4, sector 8
Mar 22 10:10:57 g8-3 kernel: [42469.999325] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42469.999418] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42469.999430] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42469.999440] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42469.999450] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42469.999461] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42469.999473] end_request: I/O error, dev dm-4, sector 1170637504
Mar 22 10:10:57 g8-3 kernel: [42469.999484] end_request: I/O error, dev dm-4, sector 1170637552
Mar 22 10:10:57 g8-3 kernel: [42469.999496] end_request: I/O error, dev dm-4, sector 8
Mar 22 10:10:57 g8-3 kernel: [42469.999506] end_request: I/O error, dev dm-4, sector 8
Mar 22 10:10:57 g8-3 kernel: [42469.999516] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42469.999526] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42469.999538] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42469.999548] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999558] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999568] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999578] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999588] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999597] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999607] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999619] end_request: I/O error, dev dm-4, sector 2048
Mar 22 10:10:57 g8-3 kernel: [42469.999628] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999638] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999648] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999658] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999667] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999677] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999687] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999696] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999706] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999716] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999725] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999735] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999745] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999754] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999764] end_request: I/O error, dev dm-4, sector 8
Mar 22 10:10:57 g8-3 kernel: [42469.999774] end_request: I/O error, dev dm-4, sector 8
Mar 22 10:10:57 g8-3 kernel: [42469.999784] end_request: I/O error, dev dm-4, sector 8
Mar 22 10:10:57 g8-3 kernel: [42469.999794] end_request: I/O error, dev dm-4, sector 8
Mar 22 10:10:57 g8-3 kernel: [42469.999805] end_request: I/O error, dev dm-4, sector 24
Mar 22 10:10:57 g8-3 kernel: [42469.999815] end_request: I/O error, dev dm-4, sector 24
Mar 22 10:10:57 g8-3 kernel: [42469.999825] end_request: I/O error, dev dm-4, sector 24
Mar 22 10:10:57 g8-3 kernel: [42469.999835] end_request: I/O error, dev dm-4, sector 24
Mar 22 10:10:57 g8-3 kernel: [42469.999846] end_request: I/O error, dev dm-4, sector 56
Mar 22 10:10:57 g8-3 kernel: [42469.999856] end_request: I/O error, dev dm-4, sector 56
Mar 22 10:10:57 g8-3 kernel: [42469.999866] end_request: I/O error, dev dm-4, sector 56
Mar 22 10:10:57 g8-3 kernel: [42469.999876] end_request: I/O error, dev dm-4, sector 56
Mar 22 10:10:57 g8-3 kernel: [42469.999886] end_request: I/O error, dev dm-4, sector 120
Mar 22 10:10:57 g8-3 kernel: [42469.999896] end_request: I/O error, dev dm-4, sector 120
Mar 22 10:10:57 g8-3 kernel: [42469.999906] end_request: I/O error, dev dm-4, sector 120
Mar 22 10:10:57 g8-3 kernel: [42469.999916] end_request: I/O error, dev dm-4, sector 120
Mar 22 10:10:57 g8-3 kernel: [42469.999926] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999935] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42469.999945] end_request: I/O error, dev dm-4, sector 8
Mar 22 10:10:57 g8-3 kernel: [42469.999955] end_request: I/O error, dev dm-4, sector 8
Mar 22 10:10:57 g8-3 kernel: [42469.999964] end_request: I/O error, dev dm-4, sector 24
Mar 22 10:10:57 g8-3 kernel: [42469.999974] end_request: I/O error, dev dm-4, sector 24
Mar 22 10:10:57 g8-3 kernel: [42469.999984] end_request: I/O error, dev dm-4, sector 56
Mar 22 10:10:57 g8-3 kernel: [42469.999994] end_request: I/O error, dev dm-4, sector 56
Mar 22 10:10:57 g8-3 kernel: [42470.000003] end_request: I/O error, dev dm-4, sector 120
Mar 22 10:10:57 g8-3 kernel: [42470.000013] end_request: I/O error, dev dm-4, sector 120
Mar 22 10:10:57 g8-3 kernel: [42470.000023] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000033] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000043] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000052] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000062] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000072] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000083] end_request: I/O error, dev dm-4, sector 16
Mar 22 10:10:57 g8-3 kernel: [42470.000094] end_request: I/O error, dev dm-4, sector 128
Mar 22 10:10:57 g8-3 kernel: [42470.000104] end_request: I/O error, dev dm-4, sector 128
Mar 22 10:10:57 g8-3 kernel: [42470.000114] end_request: I/O error, dev dm-4, sector 128
Mar 22 10:10:57 g8-3 kernel: [42470.000123] end_request: I/O error, dev dm-4, sector 16
Mar 22 10:10:57 g8-3 kernel: [42470.000133] end_request: I/O error, dev dm-4, sector 128
Mar 22 10:10:57 g8-3 kernel: [42470.000144] end_request: I/O error, dev dm-4, sector 64
Mar 22 10:10:57 g8-3 kernel: [42470.000155] end_request: I/O error, dev dm-4, sector 64
Mar 22 10:10:57 g8-3 kernel: [42470.000164] end_request: I/O error, dev dm-4, sector 64
Mar 22 10:10:57 g8-3 kernel: [42470.000174] end_request: I/O error, dev dm-4, sector 64
Mar 22 10:10:57 g8-3 kernel: [42470.000183] end_request: I/O error, dev dm-4, sector 64
Mar 22 10:10:57 g8-3 kernel: [42470.000193] end_request: I/O error, dev dm-4, sector 64
Mar 22 10:10:57 g8-3 kernel: [42470.000203] end_request: I/O error, dev dm-4, sector 64
Mar 22 10:10:57 g8-3 kernel: [42470.000212] end_request: I/O error, dev dm-4, sector 64
Mar 22 10:10:57 g8-3 kernel: [42470.000222] end_request: I/O error, dev dm-4, sector 64
Mar 22 10:10:57 g8-3 kernel: [42470.000232] end_request: I/O error, dev dm-4, sector 64
Mar 22 10:10:57 g8-3 kernel: [42470.000246] end_request: I/O error, dev dm-4, sector 256
Mar 22 10:10:57 g8-3 kernel: [42470.000256] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000266] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000276] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000285] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000295] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000305] end_request: I/O error, dev dm-4, sector 16
Mar 22 10:10:57 g8-3 kernel: [42470.000315] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000325] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000334] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000344] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000354] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000363] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000373] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000383] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000392] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000402] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000412] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000421] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000431] end_request: I/O error, dev dm-4, sector 128
Mar 22 10:10:57 g8-3 kernel: [42470.000441] end_request: I/O error, dev dm-4, sector 128
Mar 22 10:10:57 g8-3 kernel: [42470.000451] end_request: I/O error, dev dm-4, sector 16
Mar 22 10:10:57 g8-3 kernel: [42470.000460] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000470] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000480] end_request: I/O error, dev dm-4, sector 8
Mar 22 10:10:57 g8-3 kernel: [42470.000490] end_request: I/O error, dev dm-4, sector 16
Mar 22 10:10:57 g8-3 kernel: [42470.000499] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000509] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000519] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000529] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000538] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000548] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000558] end_request: I/O error, dev dm-4, sector 8
Mar 22 10:10:57 g8-3 kernel: [42470.000567] end_request: I/O error, dev dm-4, sector 128
Mar 22 10:10:57 g8-3 kernel: [42470.000577] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000587] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000601] end_request: I/O error, dev dm-4, sector 4096
Mar 22 10:10:57 g8-3 kernel: [42470.000612] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000621] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000631] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000641] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000650] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000660] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000672] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000682] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000691] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000701] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000712] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000722] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000732] end_request: I/O error, dev dm-4, sector 24
Mar 22 10:10:57 g8-3 kernel: [42470.000742] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000751] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000761] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000771] end_request: I/O error, dev dm-4, sector 56
Mar 22 10:10:57 g8-3 kernel: [42470.000780] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42470.000790] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42474.100776] sd 3:0:1:1: [sdd] Synchronizing SCSI cache
Mar 22 10:10:57 g8-3 kernel: [42474.100800] sd 3:0:1:1: [sdd] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:57 g8-3 kernel: [42474.101047] device-mapper: table: 252:4: multipath: error getting device
Mar 22 10:10:57 g8-3 kernel: [42474.110746] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42474.110748] quiet_error: 292 callbacks suppressed
Mar 22 10:10:57 g8-3 kernel: [42474.110750] Buffer I/O error on device dm-4, logical block 0
Mar 22 10:10:57 g8-3 kernel: [42474.110752] Buffer I/O error on device dm-4, logical block 1
Mar 22 10:10:57 g8-3 kernel: [42474.110753] Buffer I/O error on device dm-4, logical block 2
Mar 22 10:10:57 g8-3 kernel: [42474.110754] Buffer I/O error on device dm-4, logical block 3
Mar 22 10:10:57 g8-3 kernel: [42474.110770] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42474.110771] Buffer I/O error on device dm-4, logical block 0
Mar 22 10:10:57 g8-3 kernel: [42474.110788] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42474.110789] Buffer I/O error on device dm-4, logical block 146329695
Mar 22 10:10:57 g8-3 kernel: [42474.110798] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42474.110800] Buffer I/O error on device dm-4, logical block 146329695
Mar 22 10:10:57 g8-3 kernel: [42474.110811] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42474.110813] Buffer I/O error on device dm-4, logical block 0
Mar 22 10:10:57 g8-3 kernel: [42474.110830] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:57 g8-3 kernel: [42474.110835] Buffer I/O error on device dm-4, logical block 0
Mar 22 10:10:57 g8-3 kernel: [42474.110848] end_request: I/O error, dev dm-4, sector 24
Mar 22 10:10:57 g8-3 kernel: [42474.110849] Buffer I/O error on device dm-4, logical block 3
Mar 22 10:10:57 g8-3 kernel: [42474.111821] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:57 g8-3 kernel: [42474.111839] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:57 g8-3 kernel: [42474.111855] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42474.111864] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:57 g8-3 kernel: [42474.111876] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:57 g8-3 kernel: [42474.111893] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:57 g8-3 kernel: [42474.111907] end_request: I/O error, dev dm-3, sector 24
Mar 22 10:10:57 g8-3 kernel: [42474.811829] device-mapper: ioctl: error adding target to table
Mar 22 10:10:57 g8-3 multipathd: 3600c0ff00014fc48204b405101000000: failed in domap for removal of path sdb
Mar 22 10:10:57 g8-3 multipathd: uevent trigger error
Mar 22 10:10:57 g8-3 multipathd: sdh: remove path (uevent)
Mar 22 10:10:57 g8-3 kernel: [42474.812356] sd 3:0:1:2: [sde] Synchronizing SCSI cache
Mar 22 10:10:57 g8-3 kernel: [42474.812392] sd 3:0:1:2: [sde] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 22 10:10:57 g8-3 kernel: [42474.812537] mpt2sas0: removing handle(0x000a), sas_addr(0x500c0ff139468000)
Mar 22 10:10:58 g8-3 kernel: [42474.813995] device-mapper: table: 252:4: multipath: error getting device
Mar 22 10:10:58 g8-3 kernel: [42474.823243] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:58 g8-3 kernel: [42474.823261] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:58 g8-3 kernel: [42474.823278] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:58 g8-3 kernel: [42474.823288] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:58 g8-3 kernel: [42474.823299] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:58 g8-3 kernel: [42474.823318] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:58 g8-3 kernel: [42474.823332] end_request: I/O error, dev dm-3, sector 24
Mar 22 10:10:58 g8-3 kernel: [42474.829940] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:58 g8-3 kernel: [42474.829957] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:58 g8-3 kernel: [42474.829973] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:58 g8-3 kernel: [42474.829983] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:58 g8-3 kernel: [42474.829995] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:58 g8-3 kernel: [42474.830012] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:58 g8-3 kernel: [42474.830026] end_request: I/O error, dev dm-4, sector 24
Mar 22 10:10:58 g8-3 kernel: [42475.227592] sched: RT throttling activated
Mar 22 10:10:58 g8-3 kernel: [42475.238943] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:58 g8-3 kernel: [42475.239072] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:58 g8-3 kernel: [42475.239091] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:58 g8-3 kernel: [42475.239108] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:58 g8-3 kernel: [42475.239118] end_request: I/O error, dev dm-3, sector 1170637560
Mar 22 10:10:58 g8-3 kernel: [42475.239129] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:58 g8-3 kernel: [42475.239146] end_request: I/O error, dev dm-3, sector 0
Mar 22 10:10:58 g8-3 kernel: [42475.239160] end_request: I/O error, dev dm-3, sector 24
Mar 22 10:10:58 g8-3 kernel: [42475.454916] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:58 g8-3 kernel: [42475.480585] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:58 g8-3 kernel: [42475.510552] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:58 g8-3 kernel: [42475.540039] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:58 g8-3 kernel: [42475.566615] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:58 g8-3 kernel: [42475.593303] end_request: I/O error, dev dm-4, sector 24
Mar 22 10:10:58 g8-3 kernel: [42475.633985] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:58 g8-3 kernel: [42475.661731] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:58 g8-3 kernel: [42475.687935] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:58 g8-3 kernel: [42475.717359] end_request: I/O error, dev dm-4, sector 1170637560
Mar 22 10:10:58 g8-3 kernel: [42475.747445] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:58 g8-3 multipathd: 3600c0ff00014fc48204b405101000000: failed in domap for removal of path sdh
Mar 22 10:10:58 g8-3 multipathd: uevent trigger error
Mar 22 10:10:58 g8-3 multipathd: sdc: remove path (uevent)
Mar 22 10:10:58 g8-3 multipathd: 3600c0ff00014fd09f95e405101000000: failed in domap for removal of path sdc
Mar 22 10:10:58 g8-3 multipathd: uevent trigger error
Mar 22 10:10:58 g8-3 multipathd: sdi: remove path (uevent)
Mar 22 10:10:58 g8-3 multipathd: 3600c0ff00014fd09f95e405101000000: failed in domap for removal of path sdi
Mar 22 10:10:58 g8-3 multipathd: uevent trigger error
Mar 22 10:10:58 g8-3 multipathd: dm-3: add map (uevent)
Mar 22 10:10:58 g8-3 multipathd: dm-3: devmap already registered
Mar 22 10:10:58 g8-3 multipathd: sdd: remove path (uevent)
Mar 22 10:10:58 g8-3 multipathd: 3600c0ff00014fc48204b405101000000: map in use
Mar 22 10:10:58 g8-3 multipathd: 3600c0ff00014fc48204b405101000000: can't flush
Mar 22 10:10:58 g8-3 multipathd: 3600c0ff00014fc48204b405101000000: failed in domap for removal of path sdd
Mar 22 10:10:58 g8-3 multipathd: uevent trigger error
Mar 22 10:10:58 g8-3 multipathd: dm-4: add map (uevent)
Mar 22 10:10:58 g8-3 multipathd: dm-4: devmap already registered
Mar 22 10:10:58 g8-3 multipathd: dm-3: add map (uevent)
Mar 22 10:10:58 g8-3 multipathd: dm-3: devmap already registered
Mar 22 10:10:58 g8-3 multipathd: sde: remove path (uevent)
Mar 22 10:10:58 g8-3 kernel: [42475.748503] device-mapper: ioctl: error adding target to table
Mar 22 10:10:58 g8-3 kernel: [42475.750116] device-mapper: table: 252:4: multipath: error getting device
Mar 22 10:10:58 g8-3 kernel: [42475.750118] device-mapper: ioctl: error adding target to table
Mar 22 10:10:58 g8-3 kernel: [42475.751901] device-mapper: table: 252:3: multipath: error getting device
Mar 22 10:10:58 g8-3 kernel: [42475.751903] device-mapper: ioctl: error adding target to table
Mar 22 10:10:58 g8-3 kernel: [42475.753603] device-mapper: table: 252:3: multipath: error getting device
Mar 22 10:10:58 g8-3 kernel: [42475.753605] device-mapper: ioctl: error adding target to table
Mar 22 10:10:58 g8-3 kernel: [42475.755428] device-mapper: table: 252:3: multipath: error getting device
Mar 22 10:10:58 g8-3 kernel: [42475.755430] device-mapper: ioctl: error adding target to table
Mar 22 10:10:58 g8-3 kernel: [42475.757096] device-mapper: table: 252:3: multipath: error getting device
Mar 22 10:10:58 g8-3 kernel: [42475.757098] device-mapper: ioctl: error adding target to table
Mar 22 10:10:58 g8-3 kernel: [42475.757517] device-mapper: table: 252:4: multipath: invalid initial priority group
Mar 22 10:10:58 g8-3 kernel: [42475.757519] device-mapper: ioctl: error adding target to table
Mar 22 10:10:58 g8-3 kernel: [42475.757642] device-mapper: table: 252:4: multipath: invalid initial priority group
Mar 22 10:10:58 g8-3 kernel: [42475.757645] device-mapper: ioctl: error adding target to table
Mar 22 10:10:58 g8-3 kernel: [42476.018495] end_request: I/O error, dev dm-4, sector 0
Mar 22 10:10:58 g8-3 multipathd: 3600c0ff00014fd09f95e405101000000: devmap removed
Mar 22 10:10:58 g8-3 multipathd: 3600c0ff00014fd09f95e405101000000: stop event checker thread (140496825583360)
Mar 22 10:10:58 g8-3 multipathd: 3600c0ff00014fd09f95e405101000000: removed map after removing all paths
Mar 22 10:10:58 g8-3 multipathd: dm-3: add map (uevent)
Mar 22 10:10:58 g8-3 multipathd: uevent trigger error
Mar 22 10:10:58 g8-3 multipathd: dm-4: add map (uevent)
Mar 22 10:10:58 g8-3 multipathd: dm-4: devmap already registered
Mar 22 10:10:58 g8-3 multipathd: dm-3: add map (uevent)
Mar 22 10:10:58 g8-3 multipathd: uevent trigger error
Mar 22 10:10:58 g8-3 multipathd: dm-4: add map (uevent)
Mar 22 10:10:58 g8-3 multipathd: dm-4: devmap already registered
Mar 22 10:10:58 g8-3 multipathd: dm-3: remove map (uevent)
Mar 22 10:10:58 g8-3 multipathd: dm-3: devmap not registered, can't remove
Mar 22 10:10:58 g8-3 multipathd: dm-3: remove map (uevent)
Mar 22 10:10:58 g8-3 multipathd: dm-3: devmap not registered, can't remove
Mar 22 10:10:58 g8-3 multipathd: 8:112: mark as failed
Mar 22 10:10:58 g8-3 multipathd: 8:16: mark as failed
Mar 22 10:10:58 g8-3 multipathd: 8:48: mark as failed
Mar 22 10:10:58 g8-3 multipathd: 8:112: mark as failed
Mar 22 10:10:58 g8-3 multipathd: 8:16: mark as failed
Mar 22 10:10:58 g8-3 multipathd: 8:48: mark as failed
Mar 22 10:10:58 g8-3 multipathd: 3600c0ff00014fd09f95e405101000000: event checker exit
Mar 22 10:10:58 g8-3 kernel: [42476.044440] end_request: I/O error, dev dm-4, sector 24
Mar 22 10:10:59 g8-3 multipathd: dm-4: add map (uevent)
Mar 22 10:10:59 g8-3 multipathd: dm-4: devmap already registered
Видно, что lvm отвалился
text
# lvdisplay g8-3
/dev/dm-1: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-4: read failed after 0 of 4096 at 0: Input/output error
--- Logical volume ---
LV Name /dev/CtrlSlot0-raid5/root
VG Name CtrlSlot0-raid5
LV UUID 78nUJu-dMVO-HFuT-SsCw-1lwr-X9r6-MdG1wX
LV Write Access read/write
LV Status available
# open 1
LV Size 15.00 GiB
Current LE 3840
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:0
mount не отвалился и доступен
text
# mount | grep drbd g8-3
/dev/drbd1 on /mnt type ext4 (rw)
# touch /mnt/2 g8-3
g8-3 [touch /mnt/2] ~ 13-03-22 10:18AM
g8-3#
drbd
text
g8-3# cd /etc/drbd.d g8-3
g8-3 [cd /etc/drbd.d] /etc/drbd.d 13-03-22 10:25AM
g8-3# find . -print0 -exec cat {} \; g8-3
.cat: .: Is a directory
./global_common.conf#BCFG2
global {
usage-count yes;
# minor-count dialog-refresh disable-ip-verification
}
common {
protocol C;
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup {
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
}
disk {
# on-io-error fencing use-bmbv no-disk-barrier no-disk-flushes
# no-disk-drain no-md-flushes max-bio-bvecs
#no-disk-barrier;
#no-disk-flushes;
}
net {
# sndbuf-size rcvbuf-size timeout connect-int ping-int ping-timeout max-buffers
# max-epoch-size ko-count allow-two-primaries cram-hmac-alg shared-secret
# after-sb-0pri after-sb-1pri after-sb-2pri data-integrity-alg no-tcp-cork
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 0;
}
syncer {
# rate after al-extents use-rle cpu-mask verify-alg csums-alg
rate 1000M;
}
}
./1.res#BCFG2
resource r0 {
on g8-3 {
device /dev/drbd1;
disk /dev/mapper/drbd-drbd;
address 172.24.24.1:7789;
meta-disk internal;
}
on g8-4 {
device /dev/drbd1;
disk /dev/mapper/CtrlSlot0--raid5-drbd;
address 172.24.24.2:7789;
meta-disk internal;
}
}
text
g8-3# service drbd status g8-3
drbd driver loaded OK; device status:
version: 8.3.11 (api:88/proto:86-96)
srcversion: 71955441799F513ACA6DA60
m:res cs ro ds p mounted fstype
1:r0 Connected Primary/Secondary Diskless/UpToDate C /mnt ext4Все же похоже, что это такая фича.
primary ноду перезагрузи, если это тестовый сервер. :)
А ничего интересного). Пока 1я в буте, 2я все берёт на себя. После поднятия 1й, она становится главной.
Войдите или зарегистрируйтесь, чтобы ответить.
