How to Patch / Upgrade Exadata to any version -- Part 4 -- The Rollback Procedure

4: The Rollback Procedure

It is very unlikely that you would have to rollback an Exadata Bundle but it is interesting to document it in case of. It is indeed possible to rollback any part of the patch. Please find the procedures below but keep in mind that I have never tested it (which is an exception on this blog as everything I post has already been performed on production).
I will not discuss the GI and Database OH rollback procedure here as there's nothing specific, it is the same as with non-Exadata GI and DB OH.

4.1 - Cell Rollback

For the cells, it is only possible to rollback a successful update; indeed cells with incomplete or failed updates cannot be rolled back. If you face a failed cell update, read my first advice again, check the troubleshooting section of this blog (coming soon), the known issues of the patch readme and if you found nothing there, open a Sev 1 SR to Oracle Support.
If you then really want to rollback a successfully patched cell, here is the procedure you will find in the Readme section "2.3 Rolling Back Successfully Updated Exadata Cells" :

• Check the version that the cells will be rolled back to and the flashCacheMode setting with the following commands :

[root@myclusterdb01 ~]# dcli -l root -g cell_group imageinfo -ver -inactive
[root@myclusterdb01 ~]# dcli -l root -g cell_group cellcli -e 'list cell attributes flashCacheMode'

Cells being rolled back to releases earlier than release 11.2.3.2.0 with write back flash cache enabled need to be converted manually back to write through flash cache before being rolled back. Disable write back flash cache using the script in My Oracle Support note 1500257.1. Cells being rolled back to release 11.2.3.2.0 or later retain the flash cache mode that is currently set.

• Check the prerequisites using the following command:

[root@myclusterdb01 ~]# ./patchmgr -cells cell_group -rollback_check_prereq -rolling
[root@myclusterdb01 ~]# ./patchmgr -cells cell_group -rollback -rolling

• Clean up the cells using the -cleanup option to clean up all the temporary update or rollback files on the cells. This option cleans the stale update and rollback states as well as cleaning up to 1.5 GB of disk space on the cells. Use this option before retrying a halted or failed run of the patchmgr utility :

[root@myclusterdb01 ~]# ./patchmgr -cells cell_group -cleanup

4.2 - DB nodes Rollback

Here is the procedure that can be found in the readme to rollback a database node.

• Check the version of each DB node before the rollback :

[root@myclusterdb01 ~]# ./exa-versions.sh -d

• Umount the NFS :

You can generate all the umount commands with this command:

df -t nfs | awk '{if ($NF ~ /^\//){print "umount " $NF}}'

Note : use the -allow_active_network_mounts option instead if you run 12.1.2.1.1 or later

• Rollback the patch (launch it from the cel01 server) :

[root@myclustercel01 ~]# cd /tmp/dbserver_patch_5.161110
[root@myclustercel01 dbserver_patch_5.161110]# ./patchmgr -dbnodes ~/dbs_group -rollback -iso_repo /tmp/p24669306_121233_Linux-x86-64.zip -target_version "previous_installed_version" -allow_active_network_mounts -rolling

• Check the version of each DB node after the rollback:

[root@myclusterdb01 ~]# ./exa-versions.sh -d

4.3 - IB Switches Rollback

Be sure to be connected to the server myclusterdb01 (a server where the SSH keys are deployed to the IB switches)

• Check the current version installed on the IB switches :

[root@myclusterdb01 ~]# ./exa-versions.sh -i

Downgrade the version to the only possible version : 2.1.6-2

[root@myclusterdb01 ~]# cd /patches/OCT2016_bundle_patch/24436624/Infrastructure/12.1.2.3.3/ExadataStorageServer_InfiniBandSwitch/patch_12.1.2.3.3.161013/
[root@myclusterdb01 ~]# ./patchmgr -ibswitches ~/ib_group -downgrade -ibswitch_precheck
[root@myclusterdb01 ~]# ./patchmgr -ibswitches ~/ib_group -downgrade

• Check the versions on the IB switches after the rollback:

[root@myclusterdb01 ~]# ./exa-versions.sh -i

Quick links to Part 1 / Part 2 / Part 3 / Part 4 / Part 5 / Part 6