Exadata: reinstall a broken system RPM

The Exadata compute nodes and storage servers are some Linux servers with a specific software which optimizes everything for the Oracle database, the Oracle clusterware, etc... it means that the software is a mix of RPMs like any other Red Hat system and when we patch these servers, Oracle provides an ISO (in note 888828.1) containing the new version of all these RPMs and patchmgr updates all these RPMs. And usually, our job stops here.

But . . . it may happen that a RPM needs to be reinstalled as it may be become corrupted or broken. Let's take as an example the Exadata Management Package which provides the ability to monitor the Exadata compute nodes and storage cells. This Exadata Management Package software is located in /etc/oracle/dbserver so if you remove this directory (don't do that :)), the dbmcli command won't work any more and you will be in trouble (for patching and more).

The situation will then be that the RPM wil still be installed at the system level but dbmcli is not here any more:

[root@node01]# rpm -qa | grep exadata-dbmmgmt
exadata-dbmmgmt-19.2.8.0.0.191119-1.noarch
[root@node01]# dbmcli
dbmcli: command not found

To fix this situation, you need to reinstall this RPM and to start with, you need to get it for your current version (imageinfo or exa-versions.sh will give you the current version of your system). If you have patched your Exadata yourself, you will have the patch already somewhere; if not, you'll find it in note 888828.1.

To get the ISO out of the patch, you need to unzip the patch (thing that patchmgr does for us when patching):

[root@node01]# unzip -q p30398147_192000_Linux-x86-64.zip
[root@node01]# ls -ltr exa*
-rw-rw-r-- 1 root root 1421932544 Nov 20  2019 exadata_ol7_base_repo_19.2.8.0.0.191119.iso

Now, we will mount the ISO:

[root@node01]# mkdir /mnt
[root@node01]# mount -o loop exadata_ol7_base_repo_19.2.8.0.0.191119.iso /mnt
mount: /dev/loop0 is write-protected, mounting read-only

We can now copy the RPM somewhere for future use and umount the ISO (we are done with it, you can delete it):

[root@node01]# cp /mnt/x86_64/exadata-dbmmgmt-19.2.8.0.0.191119-1.noarch.rpm /tmp/.
[root@node01]# umount /mnt

Let's now deinstall the broken RPM:

[root@node01]# rpm -e exadata-dbmmgmt-19.2.8.0.0.191119-1.noarch --nodeps
tee: /opt/oracle/dbserver_19.2.8.0.0.191119/.install_log.txt: No such file or directory
2020-11-03 15:00:05 +1100: Pre uninstallation steps in progress ...
Uninstalling version 19.2.8.0.0.191119.
. . . many errors as the directory has been dropped . . .

And reinstall the good RPM:

[root@node01]# rpm -ivh /tmp/exadata-dbmmgmt-19.2.8.0.0.191119-1.noarch.rpm
Preparing...                          ################################# [100%]
2020-11-03 15:00:24 +1100: Pre Installation steps in progress ...
2020-11-03 15:00:24 +1100: This is a fresh install.
Updating / installing...
   1:exadata-dbmmgmt-19.2.8.0.0.191119################################# [100%]
2020-11-03 15:00:34 +1100: Post Installation steps in progress ...
. . .
[root@node01]#

In the specific case of the management package, we also need to restart the services (dbmcli -e alter dbserver restart services all) and you are good to go as if /etc/oracle/dbserver has never been dropped !

The important thing to understand here is that the patches provided in the note 888828.1 to patch your Exadatas servers are ISO containing RPMs which can then be used to reinstall some RPMs in case of something wrong happens on your systems -- and this can be very very useful !