Exadata: how to manage OS dependencies when upgrading to 19c

Exadata 19c comes with Linux 7 whereas Exadata 12c and 18c come with Linux 6; there will then be an upgrade to Linux 7 when upgrading from 12c or 18c to Exadata 19c or above. In this specific case, you may face specific dependencies issues I will be explaining how to deal with below.
This can be split in two more general cases:

• When there is no OS upgrade, I have documented the way of dealing with dependencies in this blog.
• If you have an OS upgrade so if you upgrade from 12c or 18c to 19c and above, you will have to follow a specific way of dealing with this due to the upgrade to Linux 7 which I describe below.

Database servers pre-requisites

When running the usual database servers pre-requisites when upgrading to 19c, the dependencies error message will be slightly different than the usual one (when no OS upgrade happens):

[root@exacel01 dbserver_patch_19.190411]# ./patchmgr -dbnodes ~/dbs_group -precheck  -iso_repo /tmp/SAVE/p29542809_*_Linux-x86-64.zip -target_version 19.2.1.0.0.190410 -allow_active_network_mounts

************************************************************************************************************
NOTE    patchmgr release: 19.190411 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip)
NOTE
WARNING Do not interrupt the patchmgr session.
WARNING Do not resize the screen. It may disturb the screen layout.
WARNING Do not reboot database nodes during update or rollback.
WARNING Do not open logfiles in write mode and do not try to alter them.
************************************************************************************************************
2019-05-07 19:49:41 -0500        :Working: DO: Initiate precheck on 4 node(s)
2019-05-07 19:50:33 -0500        :Working: DO: Check free space and verify SSH equivalence for the root user to node(s)
2019-05-07 19:50:46 -0500        :SUCCESS: DONE: Check free space and verify SSH equivalence for the root user to node(s)
2019-05-07 19:52:02 -0500        :Working: DO: dbnodeupdate.sh running a precheck on node(s).
2019-05-07 19:57:29 -0500        :ERROR  : dbnodeupdate.sh precheck failed on one or more nodes

    SUMMARY OF WARNINGS AND ERRORS FOR exadb01:

    . . .
    exadb01: ERROR: Custom packages found that require removal before updating to another major Oracle Linux release, see exadb01:/var/log/cellos/unkown_packages-rpt.070519194937.txt for more details.
    exadb01: WARNING: This Exadata update includes an Oracle Linux 6 to Oracle Linux 7 update.
    exadb01: WARNING: Although the system was analyzed for custom rpms, any other custom installed software (such as tar-balls) cannot be detected.
    exadb01: WARNING: After updating to another major OS it may be necessary to update any script used for mounting DBFS so that it can correctly determine whether or not DBFS has been successfully mounted since the expected output from the 'stat' command, which is used to determine the status of the DBFS mount point, has changed. If an early version of mount-dbfs.sh from Note 1054431.1 has been used or extended, download and use/extend the new version that checks for 'fuseblk' as the output of the 'stat' command. Failure to update the script will lead to incorrect status being returned for DBFS, with the script reporting that DBFS has not been mounted successfully when it has. Any CRS resource that makes use of such a script will also show the incorrect status!
    exadb01: The following known issues will be checked for but require manual follow-up:
    exadb01: (*) - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
    exadb01: (*) - DBFS mount scripts may fail after a major os upgrade

And when looking at the *unkown_packages-rpt* file:

[root@exadb01]# cat /var/log/cellos/unkown_packages-rpt.070519194937.txt
#################################################################################
# File initialized at 070519_195509 (runid :070519194937) by dbnodeupdate.sh 19.190411
# NOTE: This list contains rpms which are seen as custom rpms
#################################################################################
. . .
list of customs RPMs here
. . .
#################################################################################
# ALERT       : Custom packages found (see above)
#               These custom packages MUST be removed before proceeding a major Oracle Linux upgrade.
#               Run dbnodeupdate.sh with additional -R flag at update time to have these packages automatically removed - OR -
#               Run /var/log/cellos/remove_unkown_packages.070519194937.sh manually as root before re-running prereq check.
#
# RECOMMENDED : Let dbnodeupdate.sh remove the packages right after the mandatory backup by using the -R flag
# WARNING     : Removing custom packages may impact functionality
# NOTE        : When removed functionality is still required after the update, reinstall the rpm after the update
#################################################################################
[root@exadb01]#

So in this case clearly, the customs RPMs MUST be removed to be able to upgrade to 19c then to Linux 7. We have a few ways to achieve this:

• Using a generated script (/var/log/cellos/remove_unkown_packages.070519194937.sh) right now
• Using a generated script (/var/log/cellos/remove_unkown_packages.070519194937.sh) just before the upgrade
• Run dbnodeupdate.sh with additional -R flag at update time to have these packages automatically removed -- this is the patchmgr option -force_remove_custom_rpms

I personally recommend to go with the patchmgr option -force_remove_custom_rpms which would do everything for us at upgrade time.
Just one thing is that if we check the patchmgr documentation, -force_remove_custom_rpms is still documented as to be used to upgrade from Linux 5 to Linux 6:

    -force_remove_custom_rpms
         Remove any custom rpms when db node update goes from OL5 to OL6.

Oracle may have missed to update the patchmgr help, you can indeed use -force_remove_custom_rpms from OL6 to OL7 as well.

So . . . in case of upgrade to Exadata 19c, you just have to add -force_remove_custom_rpms to your patchmgr command line when patching:

[root@exacel01 dbserver_patch_19.190411]# nohup ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /tmp/SAVE/p29542809_*_Linux-x86-64.zip -target_version 19.2.1.0.0.190410 -allow_active_network_mounts -force_remove_custom_rpms -rolling &

And you are good to go ! (also you have to organize the customs RPMs re-installation after patching or may be use this opportunity for a bit of cleanup !)

A general rule to keep in mind for the dependencies issues when patching Exadata is then:

• When there is no OS upgrade, follow this blog.
• When there is an OS upgrade, add -force_remove_custom_rpms to your patchmgr command line when patching.

Enjoy your Exadata in 19c !