Check current TL/SP and consistency :
lppchk -vm3 instfix -i | egrep "ML|SP" instfix -icqk 5200-09_AIX_ML | grep ":-:" shows filesets lower than TL09 oslevel -s lsvg -o lsvg -p rootvg lspv | grep alt alt_disk_install -X chpv -c hdisk1 lsvg rootvg; lsvg -l rootvg lsdev -Ccdisk
Check the cluster status :
lssrc -ls clstrmgrES lslpp -l | grep -i cluster lssrc -g cluster lssrc -ls clstrmgrES; clshowsrv -v; clRGinfo; clstat; cldump; cltopinfo; clshowres; cldisp cllsserv
4. Save network configurations and exported FS information :
# ifconfig -a # lsattr -EHl inet0; netstat -rn;odmget -q name=inet0 CuAt
- Check if NFS demon is running
# ps -ef | grep nfs # lssrc -s nfsd
- Find the exports file which is used
# find / -name exports
- Save the contents of the file
# cat </etc/exports>
- Print which FS are allowed to be exported to which clients
# showmount -e
- Print exported FS and which clients are using them
# showmount -a # showmount -a | wc -l
5. In case of LPAR, test the connection from HMC - Check the LPAR status
# lparstat -i | egrep "^Node Name|^Partition Name|^Partition Number|^Partition Group-ID"
- Find the machine serial number
# prtconf | grep "Machine Serial Number"
- Find HMC IP
# lsrsrc IBM.ManagementServer | grep Hostname | head -1
- Connect to HMC and then to the server
via console: # ssh hscroot / user@<HMC_IP> (passwd abc123 or abc1234 or serv1ce) => # vtmenu => choose the managed system => choose the partition => log on as root via citrix farm: => HMCWebSM_Power5 => <HMC_IP> (hscroot / abc123......) => Server and partition => server management => Server and partition > server management => choose Machine Serial Number => open terminal window
- Optionally save the current LPAR values
# lparstat -i
- If for some reason the connection to the server is lost and it is not possible to log on again directly, run this from the HMC
# chsysstate -m <managed_system> -r lpar -o shutdown -n <partition> --immed --restart
6. Create a new FS for the TL to download into - Check if there is enough free physical space on the system and the PP size # lsvg rootvg - Check if there is enough space on hdisk0 # lsvg -p rootvg ; lspv hdisk0 - Create a new LV on the first disk (approximately 2,6 GB needed, - 3,6GB for redundancy,
which is 52 or 26 LP, depends on their size - 64 or 128 MB) # mklv -y TL -t jfs2 rootvg 26 hdisk0 or
# smitty lvm ⇒ Add a Logical Volume - Create a new FS jfs on the previously defined LV # smitty fs ⇒ Add a Journaled File System on a Previously Defined LV - Mount this FS
# mount /TL
- Change its permissions to allow scp / ftp transfer # chmod 777 /TL or increase the existing one: # chfs -a size=+500M /TL if there is not enough space, you can also: - Use one of the present FS with enough free space, ensuring that this will not affect the availability of the applications using this FS, i.e. /appoggio or /opt/Tivoli/tmp - If there is a NFS mounted, you can store the TL there and extract it locally - You can also install TL from a remote NFS mounted locally, but you should consider this as the last possibility, because it is much slower. And if there is a problem with the connection, the installation could hang up in the middle. 7. Upload the latest TL and all available SPs to the server - Connect to the repository server and go to /aixsw for the latest TL or just SP # ssh 195.118.230.202 # cd /aixsw/<5.3> (for example) - Check if there is a newer version on http://www-933.ibm.com/eserver/support/fixes/fixcentral/main/pseries/aix (if so ⇒ upload it to the repository) - Before uploading check the Soft and Hard file size limits on the target server and change them to -1 if necessary # smitty user ⇒ Change / Show Characteristics of a User
or # lsuser -a fsize <username>=> # chuser fsize=-1 <username>
- Upload the TL or SP to the target server
# scp <5.3_TL8.tar.gz> czy44185@<target_IP>:/TL
- On target server go to /TL and extract the file
# cd /TL; gzip -cd <xxx.tar.gz> | tar -xvf -
zcat 6.0.2.7-WS-MQ-AIXPPC64-LAIC62450.tar.Z | tar -xvf -
(It is recommended to extract the SPs into different directories and install them separately.)
II. Phase two - Backup 1. Contact CC before starting the change 2. Check mounted FS and if they are going to be full (especially /var and /usr)
# mount # mount | wc -l # df
3. Perform the backup
* ALT_DISK_INSTALL a) Remove the secondary dump device - Check which physical disks are included in rootvg
# lsvg -p rootvg
- Check from which disk the system last booted
# bootinfo -b
- Check current dump devices
# sysdumpdev -l
- Check the attributes of the secondary dump device # lspv -l hdisk1 - Change the secondary dump device to /dev/sysdumpnull # smitty dump
or # sysdumpdev -P -s /dev/sysdumpnull
- Remove the secondary dump Logical Volume # smitty lvm
or # rmlv -f dump2lv
b) Unmirror the root VG - Remove the second disk and eventually the remote PV from the mirror
(perform these steps separately)
# unmirrorvg rootvg hdisk1 (hdisk2) - Check if there are any LVs left on the second disk and eventually on the remote PV
(perform these steps separately) # lspv -l hdisk1 (hdisk2)
- Migrate the remaining LVs to the first disk
(perform these steps separately)
# migratepv hdisk1 (hdisk2) hdisk0 c) Remove the second disk and eventually the remote PV from the root VG - Clear the boot record of the second PV
(perform these steps separately)
# chpv -c hdisk1 (hdisk2) - Add a new boot image to the first PV # bosboot -ad /dev/hdisk0 - Check if the bootlist is set to the first PV in the current rootvg
# bootlist -m normal -o if not => # bootlist -m normal hdisk0
- Remove the second disk and eventually the remote PV from the root VG
(perform these steps separately) # reducevg rootvg hdisk1 (hdisk2)
d) Create exclude list /etc/exclude.altdisk - Check the existing FS and exclude the unnecessary ones from the backup
including the FS created for the download of the TL # lsvg -l rootvg
- Check if there is already an exclude list made for the backup # cat /etc/exclude.altdisk The file may contain these FS:
if not ⇒ # vi /etc/exclude.altdisk e) Run the alternate disk install - Rename LVs with names longer than 11 characters (unmount before and then mount again)
# smitty lvm => Rename LV or # chlv -n <new_lv> <old_lv> (if problem to umount /opt/Tivoli => # cd /opt/Tivoli/lcf/dat/1 => # ./lcfd.sh stop => # fuser -xuck /opt/Tivoli)
- Create the alt_disk_install
# nohup alt_disk_install -BC -e /etc/exclude.altdisk hdisk1 &
- Check the job is done # tail -f nohup.out (the end of backup looks like “Fixing file system superblocks…”)
or # jobs
- Test the backup
# alt_disk_install -W hdisk1 (wake up the disk) # lspv -l hdisk1 # alt_disk_install -S (put the disk to sleep)
- Recreate the bootlist
# bootlist -m normal hdisk0 hdisk1
* MKSYSB
- Check for a tape device and for an inserted tape
# lsdev -Cc tape # tctl -f /dev/rmt0 rewind
- Send an email to this Notes address when you need to mount a tape:
ITALY TAPE LIBRARY OAO/Italy/IBM, Marco Bertuzzi/Italy/Contr/IBM
- Create an exclude list like the one for alt_disk_install
# vi /etc/exclude.rootvg
- Run the backup, create map files, exclude FS in exclude.rootvg and make new image.data file
# smitty mksysb # nohup mksysb -m -e -i /dev/rmt0 &
* MKSYSB on file
- If you cannot do backup on the tape or do alt_disk_install, you can do it on a local or NFS FS. # smitty mksysb # nohup /usr/bin/mksysb '-m' '-e' '-i' '-X' /backup/image_date & 4. Check and stop the services on the system - Stop the cluster on 2nd node
# smit clstop :
node => 2ND NODE !
mode => "graceful or Bring resource groups offline"
(on some hacmp versions option "graceful" = "Bring resource groups offline")
=> wait until one node is in ST_INIT state, then continue on the other node,
if some problems occur and cluster hangs, try # smitty hacmp => Problem Determination Tools
=> Recover From HACMP Script Failure and then again # smitty clstop
and/or
- Check / stop the services:
# ps -ef | grep http; ps -ef | grep maestro; ps -ef | grep pmon; ps -ef | grep lsnr; ps -ef | grep db2sysc; ps -ef | grep mqm; ps -ef | grep was; ps -ef | grep was | wc -l; ps -ef | grep sap; ps -ef | grep sap | wc -l; ps -ef | grep oserv; ps -ef | grep dsm; ps -ef | grep rembak; ps -ef; ps -ef | wc -l
or =>
# services=“http maestro pmon lsnr db2sysc mqm was sap oserv dsm rembak”; for service in $services; do running_processes=“$(ps -ef | grep $service | grep -v grep)”; if [ -n “$running_processes” ]; then echo; echo $service:; echo; echo “$running_processes”; echo; echo “number of lines: $(echo “$running_processes” | wc -l)”; echo; echo XXXXXXXXXX; fi; done; echo; echo “summary:”; echo; ps -ef; echo; echo “number of lines:$(ps -ef | wc -l)”
- Stop Tivoli # /etc/Tivoli/oserv.rc stop or /usr/local/Tivoli/etc/oserv.rc stop - Stop Netview
# ovstop
- Stop rembak (printing) before performing the backup and roll it back after the backup
# cd /etc # mv qconfig qconfig_backup_orig # cat qconfig_backup_orig | grep "^*" > qconfig # lssrc -a | grep spooler # stopsrc -g spooler
- Stop DB2 III. Phase three - Installation 1. Commit all software - Before the installation you should always commit all software installed on the server
(first do it in PREVIEW mode to ensure correct output) # smitty commit
- Check there is no applied software left
# lslpp -l | grep APPLIED
2. Installation - Before installation change to the directory containing the extracted packages - Install LPP Install Commands package (again first in PREVIEW mode, COMMIT software - NO) # smitty install ⇒ bos.rte.install (in current directory . ) - Install the new TL or SP (again first in PREVIEW mode to ensure correct upgrade)
# smitty update_all => set directory containing TL or SP (.) => COMMIT software - NO => Accept new licenses - YES
If the installation fails, clean the unsuccessful installation ⇒ # installp -C 3. Check the result of the installation and the system status
# lppchk -v; oslevel -s; instfix -i | egrep "ML|SP" (All filesets should be present.)
- Check again the bootlist # bootlist -m normal -o 4. For GV servers start script fixing routes affected when upgrading from AIX TL5 to a newer one
# nohup ./routes.sh TEST & # nohup ./routes.sh CONFIRM &
5. Reboot the server
# shutdown -Fr
- If the connection to GV servers fails after the reboot, use HMC (HMCWebSM_Power5 from MILSNISM1 farm)
=> hscroot / abc123, => Server and Partition => Server Management => choose the server with correct serial number => choose the correct hostname => Open terminal window => log on as root => add repository to the routing table: # chdev -l inet0 -a addroute="net,-hopcount,0,-netmask,255.255.255.0,,195.118.230.0,130.103.136.1"
6. Verify if the new TL / SP is installed, all VGs are varied on and all FS are mounted
# lppchk -v; oslevel -s; instfix -i | egrep "ML|SP" # lsvg -o (all VGs may be varied on after the cluster is running) if not => # varyonvg <vg> # lsvg -l rootvg # mount # mount | wc -l (all NFS may be mounted after the cluster is running) if not => # mount <host>:<remote_FS> <local_FS> # df
- Print exported FS and what clients are using them
# showmount -a
- Export all directories listed in /etc/exports file (if there were any and it was not done automatically after the reboot)
# exportfs -a if there is a different file used instead of /etc/exports, be sure to do it, as this is not done automatically => # exportfs -a -f <hacmp/etc/exports> (if some problems with unknown hosts occur, check /etc/hosts)
7. Start the cluster or services stopped before the backup # smitty clstart
=> wait until one node is in ST_STABLE state, then continue on the other node
- Check the cluster status
# lssrc -ls clstrmgrES
8. Eventually assign the remote PV back to rootvg and mirror the disks
# extendvg -f rootvg hdisk2 # nohup mirrorvg -S rootvg hdisk2 & # bosboot -ad /dev/hdisk2 # bootlist -m normal hdisk0 hdisk1 hdisk2
9. Update the change, Inventario DB and check if all apars are involved
in the update
- Set task to 100%, change code completed, change status completed - Check the apars
# instfix -ivk <apar_number>
- Update Inventario - enter the new TL, SP and set individual apars as applied IV. Phase four - Recreating the Mirror (3 working days after installation) 1. Remove the alternate install - Check if “alt_disk_install” still exists on hdisk1
# lspv | grep alt
- If so, remove it
# alt_disk_install -X (or # alt_disk_install -X old_rootvg if a rollback has been done before)
- Check if it was changed to “none”
# lspv
2. Add hdisk1 to root VG - Check which hdisk was previously included in rootvg and add it # extendvg -f rootvg hdisk1
# lsvg -p rootvg
3. Create the secondary dump device on hdisk1 - Extract the name and size of the primary dump device on hdisk0 (for example dumplv LPs 20 - should be the same number on both disks; you can also calculate the size of the necessary paging space with the command # sysdumpdev -e )
# lsvg -l rootvg
- Create sysdump LV similar to the first one # smitty lvm ⇒ Logical Volumes ⇒ Add a new LV in rootvg ⇒ LV name dump2lv, type sysdump, size 20 or # mklv -y dump2lv -t sysdump rootvg 20 hdisk1 - Change the secondary dump device # smitty dump ⇒ change 2nd dump device ⇒ /dev/dump2lv or # sysdumpdev -P -s /dev/dump2lv 4. Mirror the VG - Mirror the rootvg in background synchronization mode
# smitty lvm => Volume Groups => Mirror a Volume Group
or # nohup mirrorvg -S rootvg hdisk1 & if there were 3 disks in rootvg ⇒ - Remove the remote PV from the mirror
# unmirrorvg rootvg hdisk2 - Clear the boot record of the remote PV # chpv -c hdisk2 - Add a new boot image to the first PV # bosboot -ad /dev/hdisk0 - Set the bootlist to the first PV in the current rootvg # bootlist -m normal hdisk0 - Mirror the rootvg including all PVs # nohup mirrorvg -S -c 3 rootvg hdisk1 hdisk2 &
- Check the status of mirroring
# lsvg -l rootvg
- Create the boot image on hdisk1 and eventually on the remote PV
# bosboot -ad /dev/hdisk1 (hdisk2)
- Add hdisk1 and eventually the remote PV to the bootlist
# bootlist -m normal hdisk0 hdisk1 (hdisk2)
- Check the bootlist
# bootlist -m normal -o
- Synchronize LVs on both disks (only if syncd is not running - ps -ef | grep sync)
# nohup syncvg -v rootvg &