Now that GI 19c has been released, it is time to upgrade your GI to 18c !
I will be presenting here a well tested procedure to upgrade your Grid Infrastructure to 18c; it has been designed and applied on many Exadatas and the procedure can also be applied on non-Exadata systems. You will notice that it is very close to upgrade your Grid Infrastructure to 12c.
0/ Preparation
Please find few things that are good to know and read before starting upgrading a GI to 18c:
- Download the GI 18c image from edelivery: V978971-01.zip
- Download the RU you want to apply like Patch 28828717: GI RELEASE UPDATE 18.5.0.0.0; As I use to do this kind of maintenance when I patch the Exadata stack, I usually use the GI RU provided in the Exadata bundle like 28980183/Database/18.5.0.0.0/18.5.0.0.190115GIRU/28828717 from the Jan 19 Bundle; you can do either way, it is the same RU.
- GI 18c will be installed on /u01/app/18.1.0.0/grid
- GI 12.1 is running from /u01/app/12.1.0.2/grid on the systems I work on
- This procedure has been successfully applied on many Exadatas half rack and full rack; it also applies to non-Exadata GI system
- I use the rac-status.sh script to check the status of al the resources of my cluster before and after the maintenance to avoid any unpleasantness
- Check your oratab entries to avoid having them deleted during the upgrade as explained in this post
Versions naming:
It is not a secret to anyone, Oracle's version naming has always been changed in a not really consistent manner across the years making it not really easy to follow and then this new style of yearly release also came with its new version numbering. Indeed, we used to name our directories with the version number like 12.1.0.2 and then create a new directory like 12.2.0.1 when we would perform an out of place upgrade to a new major version.
Nowadays, even if 18c is actually version 12.2.0.2, the database came with a 18.1 version and GI came with a 18.0 version and each RU will increment the second number of the version, January 19 RU (patch 28828717) being GI version 18.5, and so on.
I had a discussion with some friends about how to name the GI directory knowing that and also that nowadays, Oracle promotes the Rapid Home Provisionning tool for an "always out of place" patch strategy.
We decided that we wouldn't go with RHP nor an out of place patching for each RU but for every major release as we were doing before. It would then have made no sense to name our directory /u01/app/18.5.0.0/grid for GI upgrade to 18c + Jan 19 Bundle as April 19 Bundle would upgrade the version to 18.6; we then have decided to name our GI 18c directory with the original version number /u01/app/18.1.0.0/grid which makes it clear to everyone (and 18.0 looked weird to me, indeed, a "0" version makes not much sense to me even if /u01/app/18.0.0.0/grid is the path used in Oracle Cloud). Also, we kept 4 numbers in the naming to stay consistent with what we were doing before and with the databases Homes.
And the day you need to double check what exact patch is installed here, we would rely on something like lspatches which you would also do in any case as one off patches may have been installed on your Homes and you would not have renamed the directory.
Having said that, you can name it as you wish :)
Nowadays, even if 18c is actually version 12.2.0.2, the database came with a 18.1 version and GI came with a 18.0 version and each RU will increment the second number of the version, January 19 RU (patch 28828717) being GI version 18.5, and so on.
I had a discussion with some friends about how to name the GI directory knowing that and also that nowadays, Oracle promotes the Rapid Home Provisionning tool for an "always out of place" patch strategy.
We decided that we wouldn't go with RHP nor an out of place patching for each RU but for every major release as we were doing before. It would then have made no sense to name our directory /u01/app/18.5.0.0/grid for GI upgrade to 18c + Jan 19 Bundle as April 19 Bundle would upgrade the version to 18.6; we then have decided to name our GI 18c directory with the original version number /u01/app/18.1.0.0/grid which makes it clear to everyone (and 18.0 looked weird to me, indeed, a "0" version makes not much sense to me even if /u01/app/18.0.0.0/grid is the path used in Oracle Cloud). Also, we kept 4 numbers in the naming to stay consistent with what we were doing before and with the databases Homes.
And the day you need to double check what exact patch is installed here, we would rely on something like lspatches which you would also do in any case as one off patches may have been installed on your Homes and you would not have renamed the directory.
Having said that, you can name it as you wish :)
1/ Install GI 18c from the gold image
Let's enjoy this super new feature and quickly install GI 18c from a gold image:
-- Create the target directories for GI 18c sudo su - dcli -g ~/dbs_group -l root "ls -ltr /u01/app/18.1.0.0/grid" dcli -g ~/dbs_group -l root mkdir -p /u01/app/18.1.0.0/grid dcli -g ~/dbs_group -l root chown grid:oinstall /u01/app/18.1.0.0/grid dcli -g ~/dbs_group -l root "ls -altr /u01/app/18.1.0.0/grid" -- Install GI using this gold image : /patches/V978971-01.zip sudo su - grid unzip -q /patches/V978971-01.zip -d /u01/app/18.1.0./grid
2/ Pre requisites
2.1/ Upgrade opatch
As usual, it is recommended to patch opatch before starting any patching activity. If you work with Exadata, you may have a look at this post where I show how to quickly upgrade opatch with dcli.
2.2/ ASM spfile and password file
Check that the ASM passwordfile and the ASM spfile are located under ASM to avoid issues during the upgrade:
[grid@exadb01]$ asmcmd spget +DATA/mycluster/ASMPARAMETERFILE/registry.253.909449003 [grid@exadb01]$ asmcmd pwget --asm +DATA/orapwASM [grid@exadb01]$If you don't, you may face the below error and ASM won't restart after being upgraded:
Verifying Verify that the ASM instance was configured using an existing ASM parameter file. ...FAILED PRCT-1011 : Failed to run "asmcmd". Detailed error: ASMCMD-8001: diskgroup 'u01' does not exist or is not mountedPlease find a quick procedure to move the ASM passwordfile from a FileSystem to ASM:
[grid@exadb01]$ asmcmd pwcopy /u01/app/12.1.0.2/grid/dbs/orapw+ASM +DBFS_DG/orapwASM [grid@exadb01]$ asmcmd pwset --asm +DBFS_DG/orapwASM [grid@exadb01]$ asmcmd pwget --asm
2.3/ Prepare a responsefile such as this one
[grid@exadb01]$ egrep -v "^#|^$" /tmp/giresponse.rsp | head -10 oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v18.1.0 INVENTORY_LOCATION= oracle.install.option=UPGRADE ORACLE_BASE=/u01/app/oracle oracle.install.asm.OSDBA=dba oracle.install.asm.OSOPER=dba oracle.install.asm.OSASM=dba oracle.install.crs.config.gpnp.scanName= oracle.install.crs.config.gpnp.scanPort= oracle.install.crs.config.ClusterConfiguration= [grid@exadb01]$
2.4/ System pre requisites
Check these system pre requisites:
-- a 10240 limits for the "soft stack" (if not, set it, log off and log on) [root@exadatadb01]# dcli -g ~/dbs_group -l root grep stack /etc/security/limits.conf | grep soft exadatadb01: * soft stack 10240 exadatadb02: * soft stack 10240 exadatadb03: * soft stack 10240 exadatadb04: * soft stack 10240 [root@exadatadb01]# -- at least 1500 huge pages free [root@exadatadb01]# dcli -g ~/dbs_group -l root grep -i huge /proc/meminfo .... AnonHugePages: 0 kB HugePages_Total: 200000 HugePages_Free: 132171 HugePages_Rsvd: 38338 HugePages_Surp: 0 Hugepagesize: 2048 kB .... [root@exadatadb01]#
2.5/ Run the pre requisites
This step is very important and the logs need to be checked closely for any error:
[grid@exadb01]$ cd /u01/app/18.1.0.0/grid [grid@exadb01]$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/12.1.0.2/grid -dest_crshome /u01/app/18.1.0.0/grid -dest_version 18.1.0.0 -fixup -verboseBelow an output which requires a fixup:
CVU operation performed: stage -pre crsinst
Date: Jun 18, 2019 9:47:15 PM
CVU home: /u01/app/18.1.0.0/grid/
User: grid
******************************************************************************************
Following is the list of fixable prerequisites selected to fix in this session
******************************************************************************************
-------------- --------------- ------------- -------------
Check failed. Failed on nodes Reboot Re-Login
required? required?
-------------- --------------- ------------- -------------
Group Membership: asmoper exadb01,exadb02 no no
Execute "/tmp/CVU_18.0.0.0.0_grid/runfixup.sh" as root user on nodes "exadb01,exadb02" to perform the fix up operations manually
Press ENTER key to continue after execution of "/tmp/CVU_18.0.0.0.0_grid/runfixup.sh" has completed on nodes "exadb01,exadb02"
Which I executed on both nodes (do not do that if your a fixup requires a reboot !):
[root@exadb01 ~]# /tmp/CVU_18.0.0.0.0_grid/runfixup.sh All Fix-up operations were completed successfully. [root@exadb01 ~]# ssh exadb02 [root@exadb02 ~]# /tmp/CVU_18.0.0.0.0_grid/runfixup.sh All Fix-up operations were completed successfully. [root@exadb02 ~]#Once the fixup script has been executed, restart the pre-requisites to be sure that everything is now good:
[grid@exadb01]$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/12.1.0.2/grid -dest_crshome /u01/app/18.1.0.0/grid -dest_version 18.1.0.0 -fixup -verbose . . . Pre-check for cluster services setup was successful. CVU operation performed: stage -pre crsinst Date: Jun 18, 2019 9:53:40 PM CVU home: /u01/app/18.1.0.0/grid/ User: grid [grid@exadb01]$
3/ Upgrade to GI 18c
Now that all the pre requisites are successful, we can upgrade GI to 18c.
3.0/ A status before starting the upgrade
I strongly recommend to keep a status of all the resources across your cluster before starting the maintenance to avoid any unpleasantness after the maintenance.
[grid@exadb01]$ ./rac-status.sh -a -w0 | tee status_before_GI_upgrade_to_18 Cluster exadata is a X5-2 Elastic Rack HC 8TB Listener | Port | db01 | db02 | db03 | db04 | Type | ------------------------------------------------------------------------------------------------------------------- LISTENER | TCP:1551 | Online | Online | Online | Online | Listener | LISTENER_ABCD | TCP:1561 | Online | Online | Online | Online | Listener | LISTENER_SCAN1| TCP:1551,1561 | - | - | Online | - | SCAN | LISTENER_SCAN2| TCP:1551,1561 | - | Online | - | - | SCAN | LISTENER_SCAN3| TCP:1551,1561 | Online | - | - | - | SCAN | ------------------------------------------------------------------------------------------------------------------- DB | Service | db01 | db02 | db03 | db04 | ---------------------------------------------------------------------------------------------------- db01 | proddb_1_bkup | Online | - | - | - | | proddb_2_bkup | - | Online | - | - | | proddb_3_bkup | - | - | Online | - | | proddb_4_bkup | - | - | - | Online | db02 | db02svc1_bkup | - | - | Online | - | | db02svc2_bkup | - | - | Online | - | db03 | db03svc1_bkup | Online | - | - | - | | db03svc2_bkup | Online | - | - | - | db04 | db04svc1_bkup | Online | - | - | - | | db04svc2_bkup | - | Online | - | - | | db04svc3_bkup | - | - | Online | - | | db04svc4_bkup | - | - | - | Online | ---------------------------------------------------------------------------------------------------- DB | Version | db01 | db02 | db03 | db04 | DB Type | ------------------------------------------------------------------------------------------------------------------- db01 | 12.1.0.2 (1) | Readonly | Readonly | Readonly | Readonly | RAC (S) | db02 | 12.1.0.2 (1) | - | - | Open | Open | RAC (P) | db03 | 12.1.0.2 (1) | Open | Open | - | - | RAC (P) | db04 | 12.1.0.2 (1) | Readonly | Readonly | Readonly | Readonly | RAC (S) | ------------------------------------------------------------------------------------------------------------------- ORACLE_HOME references listed in the Version column : Primary : White and (P) Standby : Red and (S) 1 : /u01/app/oracle/product/12.1.0.2/dbhome_1 [grid@exadb01]$
3.1/ ASM memory setting
Some recommended memory settings have to be set at ASM instance level:[grid@exadb01]$ sqlplus / as sysasm SQL> alter system set sga_max_size = 3G scope=spfile sid='*'; SQL> alter system set sga_target = 3G scope=spfile sid='*'; SQL> alter system set memory_target=0 sid='*' scope=spfile; SQL> alter system set memory_max_target=0 sid='*' scope=spfile /* required workaround */; SQL> alter system reset memory_max_target sid='*' scope=spfile; SQL> alter system set use_large_pages=true sid='*' scope=spfile /* 11.2.0.2 and later(Linux only) */;
3.2/ Reset miscount to default
The miscount parameter is the maximum time, in seconds, that a network heartbeat can be missed before a node eviction occurs. It needs to be reset to default before upgrading. It has to be done as the GI owner.
[grid@exadb01]$ . oraenv <<< +ASM1 [grid@exadb01]$ crsctl unset css misscount
3.3/ gridSetup.sh
We will be using a script named gridSetup.sh to initiate the GI upgrade to 18c
Please find below a whole output:
Please find below a whole output:
[grid@exadb01]$ cd /u01/app/18.1.0.0/grid [grid@exadb01]$ ./gridSetup.sh -silent -responseFile /tmp/giresponse.rsp -J-Doracle.install.mgmtDB=false -J-Doracle.install.crs.enableRemoteGIMR=false -applyRU /patches/28980183/Database/18.5.0.0.0/18.5.0.0.190115GIRU/28828717 Preparing the home to patch... Applying the patch /patches/28980183/Database/18.5.0.0.0/18.5.0.0.190115GIRU/28828717... Successfully applied the patch. The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2018-11-18_05-11-43PM/installerPatchActions_2018-11-18_05-11-43PM.log Launching Oracle Grid Infrastructure Setup Wizard... . . . You can find the log of this install session at: /u01/app/oraInventory/logs/GridSetupActions2018-11-18_05-11-43PM/gridSetupActions2018-11-18_05-11-43PM.log As a root user, execute the following script(s): 1. /u01/app/18.1.0.0/grid/rootupgrade.sh Execute /u01/app/18.1.0.0/grid/rootupgrade.sh on the following nodes: [exadb01, exadb02, exadb04, exadb03] Run the script on the local node first. After successful completion, you can start the script in parallel on all other nodes, except a node you designate as the last node. When all the nodes except the last node are done successfully, run the script on the last node. Successfully Setup Software. As install user, execute the following command to complete the configuration. /u01/app/18.1.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /tmp/giresponse.rsp [-silent] [grid@exadb01]$Above are some ignorable warnings about OS groups. Is also described the next step which is to start rootupgrade.sh on each node.
3.4/ rootupgrade.sh
As specified by gridSetup.sh in the previous step, we now need to run rootupgrade.sh on each node knowing that you can start rootupgrade.sh in parallel except for the first and the last node; below an example with a half rack (4 nodes):
Here is a sample output; note that rootupgrade.sh is very silent, all logs go to the log file specified:
An interesting thing to note here after a node is patched is that the softwareversion is now the target one (18.5 and not that here CRS shows 18.0.0.0.0) but the activeversion is still the old one (12.1); indeed, the activeversion will be changed to 18.5 when applying rootupgrade.sh on the last node.
- Start rootupgrade.sh on the node 1
- Start rootupgrade.sh in parallel on the nodes 2 and 3
- Start rootupgrade.sh on the node 4
Here is a sample output; note that rootupgrade.sh is very silent, all logs go to the log file specified:
[root@exadb01]# /u01/app/18.1.0.0/grid/rootupgrade.sh Check /u01/app/18.1.0.0/grid/install/root_exadb01_2019-06-29_11-14-09-408446489.log for the output of root script [root@exadb01]#
An interesting thing to note here after a node is patched is that the softwareversion is now the target one (18.5 and not that here CRS shows 18.0.0.0.0) but the activeversion is still the old one (12.1); indeed, the activeversion will be changed to 18.5 when applying rootupgrade.sh on the last node.
[root@exadb01]# . oraenv <<< +ASM1 [root@exadb01]# crsctl query crs softwareversion Oracle Clusterware version on node [exadb01] is [18.0.0.0.0] [root@exadb01]# crsctl query crs activeversion Oracle Clusterware active version on the cluster is [12.1.0.2.0] [root@exadatadb01]#
3.5/ gridSetup.sh -executeConfigTools
Run the gridSetup.sh -executeConfigTools command:
[grid@exadb01]$ /u01/app/18.1.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /tmp/giresponse.rsp -silent Launching Oracle Grid Infrastructure Setup Wizard... You can find the logs of this session at: /u01/app/oraInventory/logs/GridSetupActions2018-11-18_07-11-22PM Successfully Configured Software. [grid@exadb01]$
3.6/ Check that GI is relinked with RDS:
It is worth double checking that the new GI Home is properly relinked with RDS to avoid future performance issues (you may want to read this pdf for more information on what RDS is):
[grid@exadb01]$ dcli -g ~/dbs_group -l oracle /u01/app/18.1.0.0/grid/bin/skgxpinfo exadatadb01: rds exadatadb02: rds exadatadb03: rds exadatadb04: rds [grid@exadb01]$If not, relink the GI Home with RDS:
dcli -g ~/dbs_group -l oracle "ORACLE_HOME=/u01/app/18.1.0.0/grid; make -C /u01/app/18.1.0.0/grid/rdbms/lib -f ins_rdbms.mk ipc_rds ioracle"
3.7/ Check the status of the cluster
Let's have a look at the status of the cluster and the activeversion:
[grid@exadb01]$ /u01/app/18.1.0.1/grid/bin/crsctl check cluster -all ************************************************************** exadatadb01: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** exadatadb02: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** exadatadb03: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** exadatadb04: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** [grid@exadb01]$ dcli -g ~/dbs_group -l oracle /u01/app/18.1.0.0/grid/bin/crsctl query crs activeversion Oracle Clusterware version on node [exadb01] is [18.0.0.0.0] Oracle Clusterware version on node [exadb02] is [18.0.0.0.0] Oracle Clusterware version on node [exadb03] is [18.0.0.0.0] Oracle Clusterware version on node [exadb04] is [18.0.0.0.0] [grid@exadb01]$Let's check the status of all the resources like we did in paragraph 3.0:
[grid@exadb01]$ ./rac-status.sh -a -w0 | tee status_after_GI_upgrade_to_18 Cluster exadata is a X5-2 Elastic Rack HC 8TB Listener | Port | db01 | db02 | db03 | db04 | Type | ------------------------------------------------------------------------------------------------------------------- LISTENER | TCP:1551 | Online | Online | Online | Online | Listener | LISTENER_ABCD | TCP:1561 | Online | Online | Online | Online | Listener | LISTENER_SCAN1| TCP:1551,1561 | - | Online | - | - | SCAN | LISTENER_SCAN2| TCP:1551,1561 | Online | - | - | - | SCAN | LISTENER_SCAN3| TCP:1551,1561 | - | - | Online | - | SCAN | ------------------------------------------------------------------------------------------------------------------- DB | Service | db01 | db02 | db03 | db04 | ---------------------------------------------------------------------------------------------------- db01 | proddb_1_bkup | Online | - | - | - | | proddb_2_bkup | - | Online | - | - | | proddb_3_bkup | - | - | Online | - | | proddb_4_bkup | - | - | - | Online | db02 | db02svc1_bkup | - | - | Online | - | | db02svc2_bkup | - | - | Online | - | db03 | db03svc1_bkup | Online | - | - | - | | db03svc2_bkup | Online | - | - | - | db04 | db04svc1_bkup | Online | - | - | - | | db04svc2_bkup | - | Online | - | - | | db04svc3_bkup | - | - | Online | - | | db04svc4_bkup | - | - | - | Online | ---------------------------------------------------------------------------------------------------- DB | Version | db01 | db02 | db03 | db04 | DB Type | ------------------------------------------------------------------------------------------------------------------- db01 | 12.1.0.2 (1) | Readonly | Readonly | Readonly | Readonly | RAC (S) | db02 | 12.1.0.2 (1) | - | - | Open | Open | RAC (P) | db03 | 12.1.0.2 (1) | Open | Open | - | - | RAC (P) | db04 | 12.1.0.2 (1) | Readonly | Readonly | Readonly | Readonly | RAC (S) | ------------------------------------------------------------------------------------------------------------------- ORACLE_HOME references listed in the Version column : Primary : White and (P) Standby : Red and (S) 1 : /u01/app/oracle/product/12.1.0.2/dbhome_1 [grid@exadb01]$And check for differences:
[grid@exadb01]$ diff status_before_GI_upgrade_to_12.2 status_after_GI_upgrade_to_12.2 8,10c8,10 < LISTENER_SCAN1| TCP:1551,1561 | - | - | Online | - | SCAN | < LISTENER_SCAN2| TCP:1551,1561 | - | Online | - | - | SCAN | < LISTENER_SCAN3| TCP:1551,1561 | Online | - | - | - | SCAN | --- > LISTENER_SCAN1| TCP:1551,1561 | - | Online | - | - | SCAN | > LISTENER_SCAN2| TCP:1551,1561 | Online | - | - | - | SCAN | > LISTENER_SCAN3| TCP:1551,1561 | - | - | Online | - | SCAN | [oracle@exadb01]$We can see here than only the SCAN listeners have been re shuffled by the maintenance which does not matter. You can relocate them but it has no impact whatsoever. It also means that all our instances and services are back as they were before the maintenance. We are then idempotent.
3.8/ Set Flex ASM Cardinality to "ALL"
Starting release 12.2 ASM will be configured as "Flex ASM". By default Flex ASM cardinality is set to 3. This means configurations with four or more database nodes in the cluster might only see ASM instances on three nodes. Nodes without an ASM instance running on it will use an ASM instance on a remote node within the cluster. Only when the cardinality is set to “ALL”, ASM will bring up the additional instances required to fulfill the cardinality setting.
[oracle@exadb01]$ srvctl modify asm -count ALL [oracle@exadb01]$Note that this command provides no output.
3.9/ Update compatible.asm to 18.1
Now that ASM 18c is running, it is recommended to update the compatible.asm to 18.1 to be able to enjoy the 18c new features.
-- Set env and connect [oracle@exadb01]$ . oraenv <<< +ASM1 [oracle@exadb01]$ sqlplus / as sysasm -- List the diskgroups SQL> select name, COMPATIBILITY from v$asm_diskgroup ; -- Set compatible to 12.2 (examples here with some usual DGs) SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'compatible.asm' = '18.1.0.0.0'; SQL> ALTER DISKGROUP DBFS_DG SET ATTRIBUTE 'compatible.asm' = '18.1.0.0.0'; SQL> ALTER DISKGROUP RECO SET ATTRIBUTE 'compatible.asm' = '18.1.0.0.0'; -- Verify the new settings SQL> select name, COMPATIBILITY from v$asm_diskgroup ;
3.10/ Update the Inventory
To wrap this up, let's update the Inventory
[grid@exadb01]$ . oraenv <<< +ASM1 [grid@exadb01]$ /u01/app/12.2.0.1/grid/oui/bin/runInstaller -ignoreSysPrereqs -updateNodeList ORACLE_HOME=/u01/app/18.1.0.0/grid "CLUSTER_NODES={exadb01,exadb02,exadb03,exadb04}" CRS=true LOCAL_NODE=exadb01Note: you may also want to update the new GI Home patch in OEM or any other monitoring tool that would require it.
3.11/ /etc/oratab entries
If you have oratab entries that have disappeared after the upgrade, you may have missed the warning in the 0/ Preparation paragraph of this post. You may want to have a look at this post for an explanation of this behavior.
And you're all done ! enjoy !
No comments:
Post a Comment