Home

Recovering from a pvmove crash

Tue, 04/06/2010 - 08:57 — liquid

Hi,

We had a server reboot in the middle of a pvmove operation, it wasn't and important FS and recovery this time was easy. Because pvmove does a extent by extent mirror as it copies the data.you allways have a valid copy of the extents on either disk(you hope!), this is is a dangerous subject and each case has to be treated individually, so this is just a little, useless example:

We were moving lvol /dev/vg00/lvcrash from /dev/dsk/c0t2d0s2 to /dev/dsk/c0t4d0s2

After the crash we have this ouput from lvdisplay:

# lvdisplay -v /dev/vg00/lvcrash | more (04-06 10:27)
--- Logical volumes ---
LV Name /dev/vg00/lvcrash
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 5008
Current LE 313
Allocated PE 314
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation non-strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t4d0s2 313 322
/dev/dsk/c0t2d0s2 304 304

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c0t4d0s2 02313 current PX_NOPV 00000 current
00001 /dev/dsk/c0t4d0s2 02314 current PX_NOPV 00000 current
00002 /dev/dsk/c0t4d0s2 02315 current PX_NOPV 00000 current
00003 /dev/dsk/c0t4d0s2 02316 current PX_NOPV 00000 current
00004 /dev/dsk/c0t4d0s2 02317 current PX_NOPV 00000 current
00005 /dev/dsk/c0t4d0s2 02318 current PX_NOPV 00000 current
00006 /dev/dsk/c0t4d0s2 02319 current PX_NOPV 00000 current
00007 /dev/dsk/c0t4d0s2 02320 current PX_NOPV 00000 current
00008 /dev/dsk/c0t4d0s2 02321 current PX_NOPV 00000 current --> up to here they moved ok
00009 /dev/dsk/c0t2d0s2 00009 current /dev/dsk/c0t4d0s2 02322 current -> when it crashed
00010 /dev/dsk/c0t2d0s2 00010 current PX_NOPV 00000 current ->from hereon the source disk
00011 /dev/dsk/c0t2d0s2 00011 current PX_NOPV 00000 current
00012 /dev/dsk/c0t2d0s2 00012 current PX_NOPV 00000 current
00013 /dev/dsk/c0t2d0s2 00013 current PX_NOPV 00000 current
00014 /dev/dsk/c0t2d0s2 00014 current PX_NOPV 00000 current
00015 /dev/dsk/c0t2d0s2 00015 current PX_NOPV 00000 current
00016 /dev/dsk/c0t2d0s2 00016 current PX_NOPV 00000 current

the first thing we need to do is get rid of the peusdo mirror pvmove creates, because if try to pvmove it will give us io errors, and we will get a third pseudo mirror(things getting worse!)

# pvmove -n /dev/vg00/lvcrash /dev/dsk/c0t2d0s2 /dev/dsk/c0t4d0s2
Transferring logical extents of logical volume "/dev/vg00/lvcrash"...
pvmove: Couldn't update the contents of logical extent:
I/O error
# lvdisplay -v /dev/vg00/lvcrash | more
--- Logical volumes ---
LV Name /dev/vg00/lvcrash
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 2
...................................
--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t4d0s2 313 636
/dev/dsk/c0t2d0s2 303 303

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2 PV3 PE3 Status 3
00000 /dev/dsk/c0t4d0s2 02313 current PX_NOPV 00000 current PX_NOPV 00000 current
00001 /dev/dsk/c0t4d0s2 02314 current PX_NOPV 00000 current PX_NOPV 00000 current
00002 /dev/dsk/c0t4d0s2 02315 current PX_NOPV 00000 current PX_NOPV 00000 current
00003 /dev/dsk/c0t4d0s2 02316 current PX_NOPV 00000 current PX_NOPV 00000 current

beacuse we have one current copy of all the extents on pv1, lets get rid of the other mirror copies:

# lvreduce -m 1 /dev/vg00/lvcrash
Logical volume "/dev/vg00/lvcrash" has been successfully reduced.

check with lvdisplay

# lvreduce -m 0 /dev/vg00/lvcrash
Logical volume "/dev/vg00/lvcrash" has been successfully reduced.

# lvdisplay -v /dev/vg00/lvcrash | more (04-06 10:41)
--- Logical volumes ---
LV Name /dev/vg00/lvcrash
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 0

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t4d0s2 10 10
/dev/dsk/c0t2d0s2 303 303

--- Logical extents ---
LE PV1 PE1 Status 1

00000 /dev/dsk/c0t4d0s2 02313 current
00001 /dev/dsk/c0t4d0s2 02314 current
00002 /dev/dsk/c0t4d0s2 02315 current
00003 /dev/dsk/c0t4d0s2 02316 current
00004 /dev/dsk/c0t4d0s2 02317 current
00005 /dev/dsk/c0t4d0s2 02318 current
00006 /dev/dsk/c0t4d0s2 02319 current
00007 /dev/dsk/c0t4d0s2 02320 current
00008 /dev/dsk/c0t4d0s2 02321 current
00009 /dev/dsk/c0t4d0s2 02322 current
00010 /dev/dsk/c0t2d0s2 00010 current
00011 /dev/dsk/c0t2d0s2 00011 current
00012 /dev/dsk/c0t2d0s2 00012 current
00013 /dev/dsk/c0t2d0s2 00013 current
00014 /dev/dsk/c0t2d0s2 00014 current
00015 /dev/dsk/c0t2d0s2 00015 current
00016 /dev/dsk/c0t2d0s2 00016 current
00017 /dev/dsk/c0t2d0s2 00017 current
00018 /dev/dsk/c0t2d0s2 00018 current
00019 /dev/dsk/c0t2d0s2 00019 current

No we are back on track, we have the lvol with extents on both disks, no we can launch again the pvmove command(if you still feel like it ;d) and finish the migration:

# pvmove -n /dev/vg00/lvcrash /dev/dsk/c0t2d0s2 /dev/dsk/c0t4d0s2
Transferring logical extents of logical volume "/dev/vg00/lvcrash"...
Physical volume "/dev/dsk/c0t2d0s2" has been successfully moved.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf

Finished.