Exadata: Hack patchmgr

Do no try this at home, this blog is for educational (and fun) purpose only.

Let's take as an example that weird patchmgr behavior when patching Exadata:

• 1/ patchmgr uses the /etc/hosts IP of a host to start the patching
• 2/ patchmgr uses the DNS ip of a host when waiting for a host to reboot

So if it happens that the DNS IP is blocked by a firewall for example (this would be most likely due to a wrong configuration but let's assume this is the situation we are in) then patchmgr will wait for the host te be back after reboot forever -- and you will be in trouble.

To work this around, you can still comment the DNS server(s) out of the /etc/resolv.conf file from the host you start patchmgr from (no DNS server then patchmgr would not be able to use the DNS IP to ping the host waiting to come back after reboot then would use the /etc/hosts IP instead) but, by doing that, the whole host would be unable to use DNS during this patch session and this is not what you want; indeed, if you use an external server to start patchmgr, you will most likely create an incident on this system.

Another way is to ... "adapt" patchmgr to make it not to use the DNS IP when pinging the patched host to wait for it to come back online after reboot. And this kind of easy as patchmgr is made out of Shell script -- great news, right ?
So looking into patchmgr code, you will find that patchmgr uses the host command to resolve the target server IP:

host_name_or_ip=$(host -t A $target | awk '/has address/ {print $NF; exit}')

And if you look at man host, you'll see that host resolves using DNS:

host - DNS lookup utility

So this is where the described issue comes from; we would not have this issue if patchmgr would resolve the host from the /etc/hosts file using for example a simple ping:

host_name_or_ip=$(ping -qc1 $target | head -1 | awk -F "[()]" '{print $2}')

Note that the resolution from /etc/hosts before DNS is default as you can see in /etc/nsswitch.conf:

# grep hosts /etc/nsswitch.conf
hosts:          files dns    <=== we resolve using files before DNS

All that said, we can then update patchmgr to resolve using ping and no more host:

#host_name_or_ip=$(host -t A $target | awk '/has address/ {print $NF; exit}')
host_name_or_ip=$(ping -qc1 $target | head -1 | awk -F "[()]" '{print $2}')

And let's run a test (just a precheck) to see how it goes:

# ./patchmgr -dbnodes ~/dbs_group -precheck -nomodify_at_prereq -target_version 19.3.12.0.0.200905 -iso_repo ../p31720221_193000_Linux-x86-64.zip -allow_active_network_mounts
2020-11-05 15:22:45 +1100        :ERROR  : Incorrect md5sum of /patches/dbserver_patch_20.200911/patchmgr
#

Hey, patchmgr has detected that we have modified it, clever ! Indeed, patchmgr checks the md5 of all the files before starting a patching sesssion; these md5 values are saved in the md5sum_files.lst file, so we just have to get the md5 of our "patched" patchmgr and update md5sum_files.lst with it:

# md5sum patchmgr
8e75bf3c1cae3e2d75229c85e84f8e0e  patchmgr
# cp md5sum_files.lst md5sum_files.lst.orig
# vi md5sum_files.lst
# grep patchmgr md5sum_files.lst
8e75bf3c1cae3e2d75229c85e84f8e0e  patchmgr
40acc292fec697492dd40a6938fb60c4  patchmgr_functions
#

And you are now good to go, you can now run your patched patchmgr !

Again, do no try this at home, this blog is for educational (and fun) purpose only. Nothing here would be supported by Oracle -- but it is still good to know ! :)