====== AIX Installation & Maintentance ====== ===== LPAR Installation ===== ===== TL/SP Update (APAR Installation) ===== ==== I. Phase one - Preparation ==== **Check current TL/SP and consistency :** lppchk -vm3 instfix -i | egrep "ML|SP" instfix -icqk 5200-09_AIX_ML | grep ":-:" shows filesets lower than TL09 oslevel -s lsvg -o lsvg -p rootvg lspv | grep alt alt_disk_install -X chpv -c hdisk1 lsvg rootvg; lsvg -l rootvg lsdev -Ccdisk **Check the cluster status :** lssrc -ls clstrmgrES lslpp -l | grep -i cluster lssrc -g cluster lssrc -ls clstrmgrES; clshowsrv -v; clRGinfo; clstat; cldump; cltopinfo; clshowres; cldisp cllsserv **4. Save network configurations and exported FS information :** # ifconfig -a # lsattr -EHl inet0; netstat -rn;odmget -q name=inet0 CuAt - Check if NFS demon is running # ps -ef | grep nfs # lssrc -s nfsd - Find the exports file which is used # find / -name exports - Save the contents of the file # cat - Print which FS are allowed to be exported to which clients # showmount -e - Print exported FS and which clients are using them # showmount -a # showmount -a | wc -l 5. In case of LPAR, test the connection from HMC - Check the LPAR status # lparstat -i | egrep "^Node Name|^Partition Name|^Partition Number|^Partition Group-ID" - Find the machine serial number # prtconf | grep "Machine Serial Number" - Find HMC IP # lsrsrc IBM.ManagementServer | grep Hostname | head -1 - Connect to HMC and then to the server via console: # ssh hscroot / user@ (passwd abc123 or abc1234 or serv1ce) => # vtmenu => choose the managed system => choose the partition => log on as root via citrix farm: => HMCWebSM_Power5 => (hscroot / abc123......) => Server and partition => server management => Server and partition > server management => choose Machine Serial Number => open terminal window - Optionally save the current LPAR values # lparstat -i - If for some reason the connection to the server is lost and it is not possible to log on again directly, run this from the HMC # chsysstate -m -r lpar -o shutdown -n --immed --restart 6. Create a new FS for the TL to download into - Check if there is enough free physical space on the system and the PP size # lsvg rootvg - Check if there is enough space on hdisk0 # lsvg -p rootvg ; lspv hdisk0 - Create a new LV on the first disk (approximately 2,6 GB needed, - 3,6GB for redundancy, which is 52 or 26 LP, depends on their size - 64 or 128 MB) # mklv -y TL -t jfs2 rootvg 26 hdisk0 or # smitty lvm => Add a Logical Volume - Create a new FS jfs on the previously defined LV # smitty fs => Add a Journaled File System on a Previously Defined LV - Mount this FS # mount /TL - Change its permissions to allow scp / ftp transfer # chmod 777 /TL or increase the existing one: # chfs -a size=+500M /TL if there is not enough space, you can also: - Use one of the present FS with enough free space, ensuring that this will not affect the availability of the applications using this FS, i.e. /appoggio or /opt/Tivoli/tmp - If there is a NFS mounted, you can store the TL there and extract it locally - You can also install TL from a remote NFS mounted locally, but you should consider this as the last possibility, because it is much slower. And if there is a problem with the connection, the installation could hang up in the middle. 7. Upload the latest TL and all available SPs to the server - Connect to the repository server and go to /aixsw for the latest TL or just SP # ssh 195.118.230.202 # cd /aixsw/<5.3> (for example) - Check if there is a newer version on http://www-933.ibm.com/eserver/support/fixes/fixcentral/main/pseries/aix (if so => upload it to the repository) - Before uploading check the Soft and Hard file size limits on the target server and change them to -1 if necessary # smitty user => Change / Show Characteristics of a User or # lsuser -a fsize => # chuser fsize=-1 - Upload the TL or SP to the target server # scp <5.3_TL8.tar.gz> czy44185@:/TL - On target server go to /TL and extract the file # cd /TL; gzip -cd | tar -xvf - zcat 6.0.2.7-WS-MQ-AIXPPC64-LAIC62450.tar.Z | tar -xvf - (It is recommended to extract the SPs into different directories and install them separately.) II. Phase two - Backup 1. Contact CC before starting the change 2. Check mounted FS and if they are going to be full (especially /var and /usr) # mount # mount | wc -l # df 3. Perform the backup * ALT_DISK_INSTALL a) Remove the secondary dump device - Check which physical disks are included in rootvg # lsvg -p rootvg - Check from which disk the system last booted # bootinfo -b - Check current dump devices # sysdumpdev -l - Check the attributes of the secondary dump device # lspv -l hdisk1 - Change the secondary dump device to /dev/sysdumpnull # smitty dump or # sysdumpdev -P -s /dev/sysdumpnull - Remove the secondary dump Logical Volume # smitty lvm or # rmlv -f dump2lv b) Unmirror the root VG - Remove the second disk and eventually the remote PV from the mirror (perform these steps separately) # unmirrorvg rootvg hdisk1 (hdisk2) - Check if there are any LVs left on the second disk and eventually on the remote PV (perform these steps separately) # lspv -l hdisk1 (hdisk2) - Migrate the remaining LVs to the first disk (perform these steps separately) # migratepv hdisk1 (hdisk2) hdisk0 c) Remove the second disk and eventually the remote PV from the root VG - Clear the boot record of the second PV (perform these steps separately) # chpv -c hdisk1 (hdisk2) - Add a new boot image to the first PV # bosboot -ad /dev/hdisk0 - Check if the bootlist is set to the first PV in the current rootvg # bootlist -m normal -o if not => # bootlist -m normal hdisk0 - Remove the second disk and eventually the remote PV from the root VG (perform these steps separately) # reducevg rootvg hdisk1 (hdisk2) d) Create exclude list /etc/exclude.altdisk - Check the existing FS and exclude the unnecessary ones from the backup including the FS created for the download of the TL # lsvg -l rootvg - Check if there is already an exclude list made for the backup # cat /etc/exclude.altdisk The file may contain these FS: ^./stgora/ ^./appoggio/ ^./scmfs/ ^./umi_img/ ^./ML/ ^./kit_aix/ ^./filesets/ ^./NMON/ ^./cibkp/ ^./opt/Tivoli/ ^./usr/local/Tivoli/ ^./var/spool/Tivoli/ ^./var/spool/Tivoli/tmp/ ^./usr/local/IBM_Tivoli/ ^./usr/local/Tivbackup/ ^./usr/local/Tivoli/custom/ if not => # vi /etc/exclude.altdisk e) Run the alternate disk install - Rename LVs with names longer than 11 characters (unmount before and then mount again) # smitty lvm => Rename LV or # chlv -n (if problem to umount /opt/Tivoli => # cd /opt/Tivoli/lcf/dat/1 => # ./lcfd.sh stop => # fuser -xuck /opt/Tivoli) - Create the alt_disk_install # nohup alt_disk_install -BC -e /etc/exclude.altdisk hdisk1 & - Check the job is done # tail -f nohup.out (the end of backup looks like “Fixing file system superblocks...”) or # jobs - Test the backup # alt_disk_install -W hdisk1 (wake up the disk) # lspv -l hdisk1 # alt_disk_install -S (put the disk to sleep) - Recreate the bootlist # bootlist -m normal hdisk0 hdisk1 * MKSYSB - Check for a tape device and for an inserted tape # lsdev -Cc tape # tctl -f /dev/rmt0 rewind - Send an email to this Notes address when you need to mount a tape: ITALY TAPE LIBRARY OAO/Italy/IBM, Marco Bertuzzi/Italy/Contr/IBM - Create an exclude list like the one for alt_disk_install # vi /etc/exclude.rootvg - Run the backup, create map files, exclude FS in exclude.rootvg and make new image.data file # smitty mksysb # nohup mksysb -m -e -i /dev/rmt0 & * MKSYSB on file - If you cannot do backup on the tape or do alt_disk_install, you can do it on a local or NFS FS. # smitty mksysb # nohup /usr/bin/mksysb '-m' '-e' '-i' '-X' /backup/image_date & 4. Check and stop the services on the system - Stop the cluster on 2nd node # smit clstop : node => 2ND NODE ! mode => "graceful or Bring resource groups offline" (on some hacmp versions option "graceful" = "Bring resource groups offline") => wait until one node is in ST_INIT state, then continue on the other node, if some problems occur and cluster hangs, try # smitty hacmp => Problem Determination Tools => Recover From HACMP Script Failure and then again # smitty clstop and/or - Check / stop the services: # ps -ef | grep http; ps -ef | grep maestro; ps -ef | grep pmon; ps -ef | grep lsnr; ps -ef | grep db2sysc; ps -ef | grep mqm; ps -ef | grep was; ps -ef | grep was | wc -l; ps -ef | grep sap; ps -ef | grep sap | wc -l; ps -ef | grep oserv; ps -ef | grep dsm; ps -ef | grep rembak; ps -ef; ps -ef | wc -l or => # services="http maestro pmon lsnr db2sysc mqm was sap oserv dsm rembak"; for service in $services; do running_processes="$(ps -ef | grep $service | grep -v grep)"; if [ -n "$running_processes" ]; then echo; echo $service:; echo; echo "$running_processes"; echo; echo "number of lines: $(echo "$running_processes" | wc -l)"; echo; echo XXXXXXXXXX; fi; done; echo; echo "summary:"; echo; ps -ef; echo; echo "number of lines:$(ps -ef | wc -l)" - Stop Tivoli # /etc/Tivoli/oserv.rc stop or /usr/local/Tivoli/etc/oserv.rc stop - Stop Netview # ovstop - Stop rembak (printing) before performing the backup and roll it back after the backup # cd /etc # mv qconfig qconfig_backup_orig # cat qconfig_backup_orig | grep "^*" > qconfig # lssrc -a | grep spooler # stopsrc -g spooler - Stop DB2 III. Phase three - Installation 1. Commit all software - Before the installation you should always commit all software installed on the server (first do it in PREVIEW mode to ensure correct output) # smitty commit - Check there is no applied software left # lslpp -l | grep APPLIED 2. Installation - Before installation change to the directory containing the extracted packages - Install LPP Install Commands package (again first in PREVIEW mode, COMMIT software - NO) # smitty install => bos.rte.install (in current directory . ) - Install the new TL or SP (again first in PREVIEW mode to ensure correct upgrade) # smitty update_all => set directory containing TL or SP (.) => COMMIT software - NO => Accept new licenses - YES If the installation fails, clean the unsuccessful installation => # installp -C 3. Check the result of the installation and the system status # lppchk -v; oslevel -s; instfix -i | egrep "ML|SP" (All filesets should be present.) - Check again the bootlist # bootlist -m normal -o 4. For GV servers start script fixing routes affected when upgrading from AIX TL5 to a newer one # nohup ./routes.sh TEST & # nohup ./routes.sh CONFIRM & 5. Reboot the server # shutdown -Fr - If the connection to GV servers fails after the reboot, use HMC (HMCWebSM_Power5 from MILSNISM1 farm) => hscroot / abc123, => Server and Partition => Server Management => choose the server with correct serial number => choose the correct hostname => Open terminal window => log on as root => add repository to the routing table: # chdev -l inet0 -a addroute="net,-hopcount,0,-netmask,255.255.255.0,,195.118.230.0,130.103.136.1" 6. Verify if the new TL / SP is installed, all VGs are varied on and all FS are mounted # lppchk -v; oslevel -s; instfix -i | egrep "ML|SP" # lsvg -o (all VGs may be varied on after the cluster is running) if not => # varyonvg # lsvg -l rootvg # mount # mount | wc -l (all NFS may be mounted after the cluster is running) if not => # mount : # df - Print exported FS and what clients are using them # showmount -a - Export all directories listed in /etc/exports file (if there were any and it was not done automatically after the reboot) # exportfs -a if there is a different file used instead of /etc/exports, be sure to do it, as this is not done automatically => # exportfs -a -f (if some problems with unknown hosts occur, check /etc/hosts) 7. Start the cluster or services stopped before the backup # smitty clstart => wait until one node is in ST_STABLE state, then continue on the other node - Check the cluster status # lssrc -ls clstrmgrES 8. Eventually assign the remote PV back to rootvg and mirror the disks # extendvg -f rootvg hdisk2 # nohup mirrorvg -S rootvg hdisk2 & # bosboot -ad /dev/hdisk2 # bootlist -m normal hdisk0 hdisk1 hdisk2 9. Update the change, Inventario DB and check if all apars are involved in the update - Set task to 100%, change code completed, change status completed - Check the apars # instfix -ivk - Update Inventario - enter the new TL, SP and set individual apars as applied IV. Phase four - Recreating the Mirror (3 working days after installation) 1. Remove the alternate install - Check if “alt_disk_install” still exists on hdisk1 # lspv | grep alt - If so, remove it # alt_disk_install -X (or # alt_disk_install -X old_rootvg if a rollback has been done before) - Check if it was changed to “none” # lspv 2. Add hdisk1 to root VG - Check which hdisk was previously included in rootvg and add it # extendvg -f rootvg hdisk1 # lsvg -p rootvg 3. Create the secondary dump device on hdisk1 - Extract the name and size of the primary dump device on hdisk0 (for example dumplv LPs 20 - should be the same number on both disks; you can also calculate the size of the necessary paging space with the command # sysdumpdev -e ) # lsvg -l rootvg - Create sysdump LV similar to the first one # smitty lvm => Logical Volumes => Add a new LV in rootvg => LV name dump2lv, type sysdump, size 20 or # mklv -y dump2lv -t sysdump rootvg 20 hdisk1 - Change the secondary dump device # smitty dump => change 2nd dump device => /dev/dump2lv or # sysdumpdev -P -s /dev/dump2lv 4. Mirror the VG - Mirror the rootvg in background synchronization mode # smitty lvm => Volume Groups => Mirror a Volume Group or # nohup mirrorvg -S rootvg hdisk1 & if there were 3 disks in rootvg => - Remove the remote PV from the mirror # unmirrorvg rootvg hdisk2 - Clear the boot record of the remote PV # chpv -c hdisk2 - Add a new boot image to the first PV # bosboot -ad /dev/hdisk0 - Set the bootlist to the first PV in the current rootvg # bootlist -m normal hdisk0 - Mirror the rootvg including all PVs # nohup mirrorvg -S -c 3 rootvg hdisk1 hdisk2 & - Check the status of mirroring # lsvg -l rootvg - Create the boot image on hdisk1 and eventually on the remote PV # bosboot -ad /dev/hdisk1 (hdisk2) - Add hdisk1 and eventually the remote PV to the bootlist # bootlist -m normal hdisk0 hdisk1 (hdisk2) - Check the bootlist # bootlist -m normal -o - Synchronize LVs on both disks (only if syncd is not running - ps -ef | grep sync) # nohup syncvg -v rootvg &