Rescue HP-UX SAN booted SO with ignites recovery shell
Rescue HP-UX SAN booted SO after changes in the san enviroment, when you can't boot into -lm mode
As allways on a monday morning at 8 AM you get a nice present, in this ocasion it was a non booting hp-ux system, they had problems saturday night with a san switch, and they changed our server to another switch, changing the HDW path of the boot disk.It was just a preproduction box, but there is allways a hurry...
Don't tell me why it wouldn't boot in -lm mode so i couldn't vgexport/import vg00 and get going like I normally do.
I booted from ignite and loaded a recovery shell, I pick on from there:
first load all the tools:
# loadfile /sbin/ioscan /usr/sbin/diskinfo
#loadfile /sbin/lvchange /sbin/vgimport /sbin/vgdisplay /sbin/vgchange
now let's try and import the vg:
# mkdir /dev/vg00
# mknod /dev/vg00/group c 64 0x010000
I do a weekly backup of all vgmaps to a central server, got the vg00 map for this server, I couldn't get vi/scp/ftp to work so I could copy the map file, I ended using echo ....
# echo "VGID 4b495dba487c7a31" >> /tmp/mapa
# echo "9 lvcrash" >> /tmp/mapa
# echo "1 lvol1" >> /tmp/mapa
# echo "2 lvol2" >> /tmp/mapa
# echo "3 lvol3" >> /tmp/mapa
# echo "4 lvol4" >> /tmp/mapa
# echo "5 lvol5" >> /tmp/mapa
# echo "6 lvol6" >> /tmp/mapa
# echo "7 lvol7" >> /tmp/mapa
# echo "8 lvol8" >> /tmp/mapa
# echo "10 lvol10" >> /tmp/mapa
#vgimport -s -v -m -p /tmp/mapa /dev/vg00
......
#vgimport -s -v -m /tmp/mapa /dev/vg00
# vgimport -p -v -m /tmp/mapa /dev/vg00 /dev/dsk/c0t4d0s2
Creating "/etc/lvmtab".
Beginning the import process on Volume Group "/dev/vg00".
vgimport: Warning: Volume Group belongs to different CPU ID.
Can not determine if Volume Group is in use on another system. Continuing.
Logical volume "/dev/vg00/lvcrash" has been successfully created
with lv number 9.
Logical volume "/dev/vg00/lvol1" has been successfully created
with lv number 1.
Logical volume "/dev/vg00/lvol2" has been successfully created
with lv number 2.
Logical volume "/dev/vg00/lvol3" has been successfully created
with lv number 3.
Logical volume "/dev/vg00/lvol4" has been successfully created
with lv number 4.
Logical volume "/dev/vg00/lvol5" has been successfully created
with lv number 5.
Logical volume "/dev/vg00/lvol6" has been successfully created
with lv number 6.
Logical volume "/dev/vg00/lvol7" has been successfully created
with lv number 7.
Logical volume "/dev/vg00/lvol8" has been successfully created
with lv number 8.
Logical volume "/dev/vg00/lvol10" has been successfully created
with lv number 10.
Volume group "/dev/vg00" has been successfully created.
Warning: A backup of this volume group may not exist on this machine.
Please remember to take a backup using the vgcfgbackup command after activating the volume group.
The import worked fine, now time to fsck and mount the lvols:
#loadfile /sbin/fs/vxfs/fsck /sbin/mount /sbin/lib/mfsconfig.d
# for i in 1 2 3 4 5 6 7 8 10
> do
> /sbin/fs/vxfs/fsck -y /dev/vg00/rlvol$i
> done
log replay in progress
replay complete - marking super-block as CLEAN
log replay in progress
replay complete - marking super-block as CLEAN
log replay in progress
replay complete - marking super-block as CLEAN
log replay in progress
replay complete - marking super-block as CLEAN
log replay in progress
replay complete - marking super-block as CLEAN
log replay in progress
replay complete - marking super-block as CLEAN
log replay in progress
replay complete - marking super-block as CLEAN
lets mount them in /mnt
# mount /dev/vg00/lvol3 /mnt
# mount /dev/vg00/lvol1 /mnt/stand
# mount /dev/vg00/lvol8 /mnt/usr
# mount /dev/vg00/lvol7 /mnt/var
# mount /dev/vg00/lvol6 /mnt/tmp
Once they are ready we chroot into our server
# chroot /mnt /sbin/sh
the lvmtab is pointing to the old HDW path/device /dev/dsk/c20t2d0s2 :
# vgdisplay -v /dev/vg00
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c20t2d0s2":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 10
Open LV 10
Max PV 16
Cur PV 1
Act PV 0
Max PE per PV 4326
VGDA 0
PE Size (Mbytes) 16
Total PE 0
Alloc PE 0
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
let's change it, first rescan our new HDW/path, beacause at the moment it doesn't have device files created:
# ioscan -fnkC disk
............
disk 114 6/0/1/1/0/4/0.5.141.0.0.0.7 sdisk CLAIMED DEVICE IBM 2145 <----------- no disk device
.............
# insf -e -H /0/1/1/0/4/0.5.141.0.0.0.7
.......
insf: Installing special files for sdisk instance 114 address 6/0/1/1/0/4/0.5.141.0.0.0.7
.......
# ioscan -fnkC disk
...........
disk 114 6/0/1/1/0/4/0.5.141.0.0.0.7 sdisk CLAIMED DEVICE IBM 2145
/dev/dsk/c20t0d7 /dev/dsk/c20t0d7s2 /dev/rdsk/c20t0d7 /dev/rdsk/c20t0d7s2
/dev/dsk/c20t0d7s1 /dev/dsk/c20t0d7s3 /dev/rdsk/c20t0d7s1 /dev/rdsk/c20t0d7s3
...........
#mv /etc/lvmtab /etc/lvmtab.old
# vgscan -av
Physical Volume "/dev/dsk/c1t0d0" contains no LVM information
Following Physical Volumes belong to one Volume Group.
Unable to match these Physical Volumes to a Volume Group.
Use the vgimport command to complete the process.
/dev/dsk/c0t0d0
/dev/dsk/c0t1d0
/dev/dsk/c0t3d0
/dev/vg00
/dev/dsk/c20t0d7s2
<------- ok it looks better
Scan of Physical Volumes Complete.
*** LVMTAB has been updated successfully.
*** Do the following to resync the information on the disk.
*** #1. vgchange -a y
*** #2. lvlnboot -R
# vgchange -a y
Volume group "/dev/vg00" has been successfully changed.
# lvlnboot -R
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
# vgdisplay -v /dev/vg00
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 10
Open LV 10
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 4326
VGDA 2
PE Size (Mbytes) 16
Total PE 4316
Alloc PE 2678
Free PE 1638
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
..........................
--- Physical volumes ---
PV Name /dev/dsk/c20t0d7s2
PV Status available
Total PE 4316
Free PE 1638
Autoswitch On
Proactive Polling On
All look's ok, finally setboot:
# setboot -p 6/0/1/1/0/4/0.5.141.0.0.0.7
# setboot
Primary bootpath : 6/0/1/1/0/4/0.5.141.0.0.0.7
HA Alternate bootpath :
Alternate bootpath :
exit the chroot and reboot the server.
# exit
NOTE: Pushing a shell, you will have to use "reboot" to reboot the system
when done. Use the "loadfile" command to bring in more commands as
you need them.
# rebiit
/sbin/sh: rebiit: not found
# reboot
Shutdown at 09:19 (in 0 minutes)
System shutdown time has arrived
After that it booted fine.
2 important things I allways keep of all the servers I have is a weekly cfg2html and vgmap exports. This way if you loose a server you allways have info that can save you.
On the other hand is probably quicker to restore an ignite image, than doing all this stuff, but it's fun to give it a go.