Twitter

list-ohpatches.sh: a better Oracle patch reporting tool

Tired of struggling with Oracle patches, I developped lspatches.sh three years ago and I based it on a new feature at that time which was implemented after the opatch -all_nodes option was deprecated; this new feature was a new report -remote option to opatchauto. I must confess that I like new things and this is why I decided to go this new feature instead of basing lspatches.sh on that good old buddy opatch lspatches.

Three years later, I can clearly say that it was a bad decision; indeed many people (and I) had issues mainly because:
  • Oracle clearly dropped the 11g support with this -- but people still have tons of 11g (which may not be a bad idea :))
  • This opatchauto -remote option had many issues just not working properly
  • This option has even been removed and replaced by a -host
Even if this is kind of usual with oracle and the new features, at the end of the day, lspatches.sh was most likely not to work due to the points discussed above.

So I restarted from scratch and developped a new list-ohpatches.sh script based on opatch lspatches which is working forever and also has the advantage of showing the description of the patches which is always good to have. The script is now more reliable, works perfectly with 11g, less code, more simple... in short: better.

How that works ?

list-ohpatches.sh checks the homes from /etc/oratab, runs opatch lspatches on all the nodes of a cluster and makes a nice table, one column by node, showing a green x if the patch is installed on that specific node and a red Missing if a patch is missing as you can see below:
You can then very easily find a missing or extra patch installed on one or many nodes. You can also note that the return code of the script is the number of missing patches so if used by an automatic script, you can easily know if all is good (return code 0) or not OK (return code > 0) and even the number of missing patches.

Below a successful output:

Is it Exadata specific ?

No, it works on any RAC, even on standalone systems.

Is it working for GI and databases Homes ?

Yes, it does work for GI and databases Homes.

Which user to execute the script with ?

opatch lspatches must be run by oracle (not root) so if you run it as root, the script will execute the opatch lspatches commands as oracle (or anothger user if the --oracleuser option is used to specify another "oracle" user) so you can use root or oracle, the script will manage automatically.

Some options

Feel free to check the help of the script (list-ohpatches.sh -h or list-ohpatches.sh --help) for the possible options; it is pretty simple and self-exaplanatory:
# ./list-ohpatches.sh --help

NAME
        list-ohpatches.sh - Show nice tables of the installed and/or missing patches for some GI/DB Oracle Homes

SYNOPSIS
        ./list-ohpatches.sh [-g] [-c] [-G] [-v] [-s] [-u] [-h]

DESCRIPTION
        list-ohpatches.sh Based on oratab, show nice tables of the installed and/or missing patches for some GI/DB Oracle Homes
                       You can then quickly find a missing patch across a RAC system
        list-ohpatches.sh will by default check all the nodes of a cluster (based on olsnodes) which requires ASM to be running
                       and oraenv to be working with the ASM aslias defined in oratab; If you have no ASM alias in oratab,
                       you may suffer from https://unknowndba.blogspot.com/2019/01/lost-entries-in-oratab-after-gi-122.html
                       You can specify a comma separated list of host or a file containing one host per line
        list-ohpatches.sh by default checks all the homes defined in oratab, you can use --grep/--home and --ungrep/--ignore to limit your home selection (see examples below)
        list-ohpatches.sh relies on opatch lspatches which must run as oracle user (and not root); if the script is started as root,
                       the opatch lspatches commands will be run after su - oracle (see -u | --oracleuser for more on this)


OPTIONS
        -g | --groupfile            ) A group file containing a list of hosts
        -c | --commalist  | --hosts ) A comma separated list of hosts
        -G | --grep | --oh | --home ) Pattern to grep from /etc/oratab
        -v | --ungrep | --ignore    ) Pattern to grep -v (ignore) from /etc/oratab
        -s | --showhomes | --show   ) Just show the homes from oratab resolving the grep/ungrep combinations
        -u | --oracleuser           ) User to use to run opatch lspatches if the script is started as root, default is oracle
        -h | --help                 ) Shows this help


EXAMPLES
       ./list-ohpatches.sh                                                       # Analyze and show all the homes of nodes of a cluster
       ./list-ohpatches.sh --show                                                # Show the homes from oratab (only show, dont do anything else)
       ./list-ohpatches.sh --grep grid                                           # Analyze the grid home
       ./list-ohpatches.sh --grep db --ungrep 12                                 # Only the DB homes but not the 12 ones
       ./list-ohpatches.sh --grep db --ungrep 12 --groupfile ~/dbs_group         # Same as above on the hosts contained in the ~/dbs_group file
       ./list-ohpatches.sh --home db --ignore 12 --hosts exa01,exa06             # Same as above but only on hosts exa02 and exa06
       ./list-ohpatches.sh --home db --ignore 12 --hosts exa01,exa06 -u oracle2  # Same as above but started as root; will then su - oracle2 automatically
# 


We now finally have a robust script to show the installed patches across many nodes, enjoy ! You can freely download the script here.

24 comments:

  1. Works well. Thank you for this. Appreciate your kindness and efforts on this.

    ReplyDelete
    Replies
    1. Hello Fred,
      Whatif the DB entries are no longer available in /etc/oratab file ?
      Like in OCI env, the DB's are mostly created with .env files.
      Is it okay to include an option to either check the oratab or the .env file in default loc (/home/oracle/) ?

      Delete
    2. Could you please show how these .env work ?

      Delete
    3. Yes Fred!
      In Exadata setup, each DB provisioning via the OCI-C (Gen1) CloudUI console, creates its corresponding .env file with contents similar to below format.
      And
      each dbaascli (OCI CLI "dbaastools_exa" tool-set), would update this .env file (on parameters whichever needs to be updated).


      SAMPLE
      [oracle@hostname1 ~]$ cat DBNAME.env
      ORACLE_HOME=/u02/app/oracle/product/12.1.0/dbhome_2; export ORACLE_HOME
      OH=/u02/app/oracle/product/12.1.0/dbhome_2; export OH
      ORACLE_UNQNAME=DBNAME_CLUSTER401; export ORACLE_UNQNAME
      ORACLE_BASE=/u02/app/oracle; export ORACLE_BASE
      PATH=/sbin:/bin:/usr/sbin:/usr/bin:/u02/app/oracle/product/12.1.0/dbhome_2/bin:/u02/app/oracle/product/12.1.0/dbhome_2/OPatch; export PATH
      LD_LIBRARY_PATH=/u02/app/oracle/product/12.1.0/dbhome_2/lib; export LD_LIBRARY_PATH
      TNS_ADMIN=/u02/app/oracle/product/12.1.0/dbhome_2/network/admin/DBNAME_CLUSTER401; export TNS_ADMIN
      ORACLE_HOSTNAME=hostname1.fqdn.ocm.domain.oraclecloudatcustomer.com; export ORACLE_HOSTNAME
      ORACLE_SID=DBNAME1; export ORACLE_SID
      [oracle@hostname1 ~]$

      Delete
    4. OK got it; I think that I should more rely on the clusterware to get the DB and the Homes and then run the opatch commands.

      Will do this and update in the coming ... when I have 5 minutes :)

      Delete
  2. Hi,
    I'm getting below error while executing.

    [oracle@]$ ./list-ohpatches.sh
    2021-05-25_115727 [INFO] Starting collecting GI/OH patch information
    2021-05-25_115727 [WARNING] It may be a bit slow if you have many nodes and patches as opatch lspatches is slow
    2021-05-25_115727 [INFO] Proceeding with /u01/app/12.2.0.1/grid . . .
    awk: cmd. line:50: tab_patches[$1][$2] = $2 ;
    awk: cmd. line:50: ^ syntax error
    awk: cmd. line:73: if (length(tab_patches[tab_hosts[i]][patch_id]) > 0){
    awk: cmd. line:73: ^ syntax error
    2021-05-25_115730 [INFO] Proceeding with /u01/app/oracle/product/11.2.0.4/dbhome_1 . . .
    awk: cmd. line:50: tab_patches[$1][$2] = $2 ;
    awk: cmd. line:50: ^ syntax error
    awk: cmd. line:73: if (length(tab_patches[tab_hosts[i]][patch_id]) > 0){
    awk: cmd. line:73: ^ syntax error


    Thank you..

    ReplyDelete
    Replies
    1. Hi,
      No. This is Oracle Enterprise Linux (OEL 7).

      Delete
    2. This comment has been removed by the author.

      Delete
    3. Strange, I have dev it on OEL7; it works perfectly on:

      [root@xxxx ~]# cat /etc/redhat-release
      Red Hat Enterprise Linux Server release 7.9 (Maipo)
      [root@xxxxx ~]#

      Can you try re download the script from https://raw.githubusercontent.com/freddenis/oracle-scripts/master/list-ohpatches.sh

      You may have a weird character, can you wget it or git clone it ? to be sure copy and paste dont mess it up ?

      Thanks,

      Delete
    4. Hi,
      Sure. I'll again try to download and will execute.
      Thank you

      Delete
    5. Hi,
      The script is perfectly working on Linux 7. But failing on 6.

      LINUX - 6
      ==========

      [oracle@ ~]$ cat /etc/redhat-release
      Red Hat Enterprise Linux Server release 6.10 (Santiago)
      [oracle@ ~]$

      [oracle@ ~]$ ./list-ohpatches.sh
      which: no olsnodes in (/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/oracle/bin:/home/oracle/bin)
      2021-05-28_060448 [INFO] Starting collecting GI/OH patch information
      2021-05-28_060448 [WARNING] It may be a bit slow if you have many nodes and patches as opatch lspatches is slow
      2021-05-28_060448 [INFO] Proceeding with /u01/app/oracle/product/12.2.0.1/dbhome_1 . . .
      awk: cmd. line:50: tab_patches[$1][$2] = $2 ;
      awk: cmd. line:50: ^ syntax error
      awk: cmd. line:73: if (length(tab_patches[tab_hosts[i]][patch_id]) > 0){
      awk: cmd. line:73: ^ syntax error

      lINUX - 7
      ==========
      ho1$ cat /etc/redhat-release
      Red Hat Enterprise Linux Server release 7.8 (Maipo)

      ho1$ ./list-ohpatches.sh
      2021-05-28_073025 [INFO] Starting collecting GI/OH patch information
      2021-05-28_073025 [WARNING] It may be a bit slow if you have many nodes and patches as opatch lspatches is slow
      2021-05-28_073025 [INFO] Proceeding with /u01/app/18.1.0.0/grid . . .
      ------------------------------------------------------
      Patch id| | | Patch description
      ------------------------------------------------------
      ------------------------------------------------------
      2021-05-28_073027 [INFO] Proceeding with /u02/app/oracle/product/18.0.0.0/dbhome_6 . . .
      --------------------------------------------------------------------------------------------------------------------------------------------------
      Patch id | ho1 | ho2 | Patch description
      --------------------------------------------------------------------------------------------------------------------------------------------------
      28090604 | x | x | DBMS_STATS FAIL ORA-12899 VALUE TOO LARGE FOR COLUMN SYS.WRI$_OPTSTAT_OPR .JOB_
      29628647 | x | x | INCREASED GETS FOR DC_OBJECTS AND DC_USERS FOR DESCRIBE
      29997959 | x | x | DSTV34 UPDATE - TZDATA2019B - NEED OJVM FIX
      30277589 | x | x | PHX SINGLE INSTANCE CRASH DATAPUMP TRYING TO FREE UNDEFINED HANDLE IN UNEXPECTED EDGE ERROR CASE
      30432098 | x | x | MERGE REQUEST ON TOP OF 18.0.0.0.0 FOR BUGS 28125601 28852325 29997937
      30794929 | x | x | RMV0/RMV1 WATING ON GES GENERIC EVENT AFTER APPLYING 12.2.0.1.20200114 DB RU
      30805598 | x | x | OJVM RELEASE UPDATE: 18.10.0.0.200414
      30872794 | x | x | Database Release Update : 18.10.0.0.200414
      30882568 | x | x | OCW RELEASE UPDATE 18.10.0.0.0
      30994996 | x | x | PROBLEMS IN CATMVRS.SQL OF BUG 30083002
      --------------------------------------------------------------------------------------------------------------------------------------------------
      ho1$
      p

      Delete
    6. This is a feature introduced in gawk 4.0:
      awk: cmd. line:50: tab_patches[$1][$2] = $2 ;

      I checked on some OEL6 and I found gawk 3.1.7 (can you check on yours with awk --version ?)
      For reference, gawk 4.0 has been released in ... June 2011 !(10 years ago -- how come they still ship their OS with gawk 3.1.7 released in ... 2009 !)

      gawk 4.0 release date: https://lists.gnu.org/archive/html/info-gnu/2011-06/msg00013.html
      gawk 3.1.7 released date: https://lists.gnu.org/archive/html/info-gnu/2009-07/msg00018.html

      I'll check next week to reuse the old way for multidimensional arrays and see if I can make it work :)

      Regards,

      Delete
  3. Great job! Thanks for sharing

    ReplyDelete
  4. Great job! Thanks for sharing

    ReplyDelete
  5. Replies
    1. Yes it should work on AIX; let me know by email if you have an issue with it on AIX and I'll fix it.

      Delete
    2. Hi,
      can i have your emailid pls and my user is not oracle.
      I am switching to su - dbuser.

      your help is highly appreciated .

      regards,
      harish.R

      Delete
    3. I am switching to dbuser in my environment.
      su -dbuser
      and my environment is a standalone database.

      please email methe modified script to rhaarrish@gmail.com

      Delete
    4. My email is line 2 of the script

      Delete
    5. Just run it as your "dbuser"

      As discussed, your issue is because AIX getopt does not support long options; one talks about it here: https://stackoverflow.com/questions/27376607/getopt-is-not-working-on-aix-as-it-does-in-linux

      Delete
    6. Can you try the latest version ? it should work for AIX with small options; please let me know.

      Delete
  6. Workaround for Oracle Restart line 222 [ -z $HOSTS ] && HOSTS=$($(which hostname) -s | ${AWK} '{printf ("%s,",$1)}' | sed s'/,$//')

    ReplyDelete

OCI: Manual creation of a 23ai Autonomous Database

Command line is always more powerful than fancy GUIs; let's then push our mouse away and use our keyboard to create an OCI 23ai Autonom...