Twitter

How to install a brand new Exadata (X2 to X7) -- Part 2


This post is a rebranded post originally posted in August 2016 here; it will be maintained here in the future

Now that all the prerequisites have been performed in Part 1, we can jump in the Exadata installation procedure itself.
All the installation steps are performed by a script named install.sh (provided by OEDA) with the help of the well known <client>-<cluster>.xml XML file. It is quite straightforward. All steps can be launched in one command but I recommend launching them step by step, it is easier to troubleshoot if needed. One important thing is to not jump over a failed step; resolve the failed step first and then continue.
The install.sh script we will be using is located in the /u01/Exadata_software/linux-x64/ directory as well as the <client>-<cluster>.xml configuration file (see in Part 1 for more detail on this).


4.1/ Step 1

As the install.sh script is meant for many different configurations, the step 1 will perform some checks and adapt the list of steps that are needed for your particular configuration.
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 1
Initializing
 Executing Validate Configuration File
 Validating cluster: mycluster
  Locating machines...
  Verifying operating systems...
  Validating cluster networks...
  Validating network connectivity...
  Validating NTP setup...
  Validating physical disks on storage cells...
  Validating users...
 Validating platinum...
 Validating switches...
 Checking disk reclaim status...
 Checking Disk Tests Status....
 Completed validation...
…
Check the output of the script to see if anything is wrong and we can then continue.

4.1.2/ List of Steps

The -l option can be used to list all those steps now that the step 1 has adapted it to your configuration.
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -l
 Initializing
1. Validate Configuration File
2. Setup Required Files
3. Create Users
4. Setup Cell Connectivity
5. Verify Infiniband
6. Calibrate Cells
7. Create Cell Disks
8. Create Grid Disks
9. Install Cluster Software
10. Initialize Cluster Software
11. Install Database Software
12. Relink Database with RDS
13. Create ASM Diskgroups
14. Create Databases
15. Apply Security Fixes
16. Install Exachk
17. Create Installation Summary
18. Resecure Machine
[root@myclusterdb01 linux-x64]#

4.2/ Step 2

This step will check for all the ISO files needed for all the installation steps, you will be prompted for missing files; you would then have to download and copy the missing files in the WorkDir directory and relaunch the step.
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 2
 Initializing
 Executing Setup Required Files
 Copying and extracting required files...
 Required files are:
 /u01/Exadata_software/linux-x64/WorkDir/linuxamd64_12102_grid_1of2.zip
 /u01/Exadata_software/linux-x64/WorkDir/linuxamd64_12102_grid_2of2.zip
 …
Copying file: p6880880_121010_Linux-x86-64.zip to node myclusterdb04
 Completed copying files...
 Extracting required files...
 Copying resourcecontrol and other required files
 Creating databasemachine.xml for EM discovery
 Done Creating databasemachine.xml for EM discovery
 Successfully completed execution of step Setup Required Files [elapsed Time [Elapsed = 345038 mS [5.0 minutes] Tue Apr 05 21:30:32 CDT 2016]]
[root@myclusterdb01 linux-x64]#

4.3/ Step 3

This step will create the required users
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 3
 Initializing
 Executing Create Users
 Creating users...
 Creating users in cluster mycluster
 Validating existing users and groups...
 Creating required directories on nodes in cluster mycluster
 Updating /etc/hosts on nodes in cluster mycluster
 Setting up ssh for users in cluster mycluster
 Completed creating all users...
 Successfully completed execution of step Create Users [elapsed Time [Elapsed = 113443 mS [1.0 minutes] Tue Apr 05 19:55:36 CDT 2016]]
[root@myclusterdb01 linux-x64]#

4.4/ Step 4

This step will set up the cell connectivity creating the /etc/oracle/cell/network-config/cellip.ora and /etc/oracle/cell/network-config/cellinit.ora files
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 4
 Initializing
 Executing Setup Cell Connectivity
 Creating cellip.ora and cellinit.ora  ...
 Creating cellip.ora for cluster mycluster
 Creating cellinit.ora for cluster mycluster
 Creating cellAffinity for cluster mycluster
 Creating cellaffinity if required
 Creating cellaffinity is NOT required
 Creating cellaffinity is NOT required
 Creating cellaffinity is NOT required
 Creating cellaffinity is NOT required
 Done with cellaffinity.ora...
 Done creating cellip.ora and cellinit.ora...
 Successfully completed execution of step Setup Cell Connectivity [elapsed Time [Elapsed = 26897 mS [0.0 minutes] Tue Apr 05 19:57:27 CDT 2016]]
[root@myclusterdb01 linux-x64]#

4.5/ Step 5

This step will check the Infiniband configuration and test the performances of it using the infinicheck script
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 5
 Initializing
 Executing Verify Infiniband
 Validating infiniband network with rds-ping...
 Check Admin networks connection...
 Running infinicheck to verify infiniband fabric for cluster mycluster...
 Checking status of infiniband links in cluster mycluster...
 Running verify topology to verify infiniband network...
 No Infiniband link errors found...
 SUCCESS: Verify topology does not report any errors on node myclusterdb01 ...
…
 Successfully completed execution of step Verify Infiniband [elapsed Time [Elapsed = 823374 mS [13.0 minutes] Mon Apr 18 18:14:06 CDT 2016]]
[root@myclusterdb01 linux-x64]#

4.6/ Step 6

This step will check the cells using the cellcli -e calibrate command. It also tests the performance of the cell disks and report any issue
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 6
 Initializing
 Executing Calibrate Cells
 Calibrating cells...
 Successfully completed execution of step Calibrate Cells [elapsed Time [Elapsed = 471323 mS [7.0 minutes] Tue Apr 05 21:06:39 CDT 2016]]
[root@myclusterdb01 linux-x64]#

4.7/ Step 7

This step will create the cell disks, flashlog and flashcache
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 7
 Initializing
 Executing Create Cell Disks
 Checking physical disks for errors before creating celldisks
 Restarting cell services...
 Initializing cells...
 Cleaning cells...
 Restarting cell services...
 Creating cell disks...
 Creating flashlog on cells...
 Creating flashcache on cells...
 Successfully completed execution of step Create Cell Disks [elapsed Time [Elapsed = 125431 mS [2.0 minutes] Tue Apr 05 21:11:25 CDT 2016]]
[root@myclusterdb01 linux-x64]#

4.8/ Step 8

This step will create the grid disks
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 8
 Initializing
 Executing Create Grid Disks
 Creating grid disks for cluster mycluster
 Successfully completed execution of step Create Grid Disks [elapsed Time [Elapsed = 32878 mS [0.0 minutes] Tue Apr 05 21:15:07 CDT 2016]]
[root@myclusterdb01 linux-x64]#

4.9/ Step 9

This step will install the Grid Infrastructure 12.1.0.2
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 9
 Initializing
 Executing Install Cluster Software
 Installing cluster mycluster
 Getting grid disks using utility in /u01/app/oracle/Oeda/Software/12.1.0.2/grid
 Writing grid response file for cluster mycluster
 Running clusterware installer...
 Setting up Opatch for cluster mycluster
 Patching cluster mycluster...
 Successfully completed execution of step Install Cluster Software [elapsed Time [Elapsed = 945400 mS [15.0 minutes] Wed Apr 13 22:37:02 CDT 2016]]
[root@myclusterdb01 linux-x64]#

4.10/ Step 10

This step will run the root.sh script on all the nodes.
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 10
 Initializing
 Executing Initialize Cluster Software
 Initializing cluster mycluster
 Relinking with RDS protocol...
 Running root.sh on node myclusterdb01.mydomain.com
 Checking file root_myclusterdb01.mydomain.com_2016-04-13_22-38-09.log on node myclusterdb01.mydomain.com
 Running root.sh on node myclusterdb02.mydomain.com
 Checking file root_myclusterdb02.mydomain.com_2016-04-13_22-46-51.log on node myclusterdb02.mydomain.com
 Running root.sh on node myclusterdb03.mydomain.com
 Checking file root_myclusterdb03.mydomain.com_2016-04-13_22-54-03.log on node myclusterdb03.mydomain.com
 Running root.sh on node myclusterdb04.mydomain.com
 Checking file root_myclusterdb04.mydomain.com_2016-04-13_23-01-13.log on node myclusterdb04.mydomain.com
 Running configuration assistants...
 Running configuration tools on machine myclusterdb01.mydomain.com
 Checking status of cluster...
 Completed cluster initialization...
 Successfully completed execution of step Initialize Cluster Software [elapsed Time [Elapsed = 2572955 mS [42.0 minutes] Wed Apr 13 23:20:44 CDT 2016]]
[root@myclusterdb01 linux-x64]#
Note : if any issue occurs during this step, you will have to deinstall the GI:
$ /u01/app/oracle/product/12.1.0.2/dbhome_1/deinstall/deinstall
# dcli -g /root/dbs_group -l root "rm -fr /u01/app/12.1.0.2/grid"
# dcli -g /root/dbs_group -l root "mv /u01/app/oraInventory /u01/app/oraInventory.old"
Then reinstall the GI (step 9) and relaunch root.sh (step 10).

4.11/ Step 11

This step will install Oracle database -- this was 11.2.0.4 in this case (not 12c, I installed it manually later on)
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 11
 Initializing
 Executing Install Database Software
 Installing database software ...
 Installing database home software, database home c0_DbHome_0
 Installing database software ...
 Running database installer on node myclusterdb01.mydomain.com ... Please wait...
 After running database installer...
 Successfully completed execution of step Install Database Software [elapsed Time [Elapsed = 849646 mS [14.0 minutes] Wed Apr 13 23:45:05 CDT 2016]]
[root@myclusterdb01 linux-x64]#

4.12/ Step 12

This step will relink the installed Oracle databases homes with RDS (protocol to use Infiniband)
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 12
 Initializing
 Executing Relink Database with RDS
 Successfully completed execution of step Relink Database with RDS [elapsed Time [Elapsed = 7050 mS [0.0 minutes] Wed Apr 13 23:47:03 CDT 2016]]
[root@myclusterdb01 linux-x64]#

4.13/ Step 13

This step will create the ASM diskgroups according to what has been defined by the client in the XML configuration file (this is the perfect timing to double check, triple check, etc... the name of the diskgroups before creating it to avoid to have to rename the diskgroups later)
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 13
 Initializing
 Executing Create ASM Diskgroups
 Getting grid disks using utility in /u01/app/oracle/Oeda/Software/12.1.0.2/grid
 Getting grid disks using utility in /u01/app/oracle/Oeda/Software/12.1.0.2/grid
 Successfully completed execution of step Create ASM Diskgroups [elapsed Time [Elapsed = 198770 mS [3.0 minutes] Thu Apr 14 20:52:39 CDT 2016]]
[root@myclusterdb01 linux-x64]#

4.14/ Step 14

This step will create the databases as defined in the <client>-<cluster>.xml file. In my case, no database was defined there as the client wanted to create them manually. But you can easily do it like that :
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 14
....
[root@myclusterdb01 linux-x64]#

4.15/ Step 15

This step applies some security fixes; the error below is due to the fact that the Step 14 has not been executed; it can then be ignored
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 15
 Initializing
 Executing Apply Security Fixes
 Checking database dbm01 status on cluster mycluster
 Database instance dbm01 in cluster mycluster is not Running. Restart databases and re-run step...!
 ERROR:
 Database instance dbm01 in cluster mycluster is not Running. Restart databases and re-run step...!
 Collecting diagnostics...
 Errors occurred. Send /u01/Exadata_software/linux-x64/WorkDir/Diag-160414_215154.zip to Oracle to receive assistance.

 ERROR:
 Errors occured...
 Errors occured, exiting...
[root@myclusterdb01 linux-x64]#

4.16/ Step 16

This step installs the Exachk tool
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 16
 Initializing
 Executing Install Exachk
 Installing Exachk
 Exachk has been installed on compute node: mycluster2db01.mydomain.com at: /opt/oracle.SupportTools/exachk and can be run by invoking ./exachk
 Successfully completed execution of step Install Exachk [elapsed Time [Elapsed = 5132 mS [0.0 minutes] Thu Apr 14 21:55:35 CDT 2016]]
[root@myclusterdb01 linux-x64]#

4.17/ Step 17

This step creates an installation summary
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 17
 Initializing
 Executing Create Installation Summary
 Getting system details...
 Running verify topology to verify infiniband network...
 Validating infiniband network with rds-ping...
 Check Admin networks connection...
 Running infinicheck to verify infiniband fabric for cluster mycluster...
 Checking status of infiniband links in cluster mycluster...
 Running verify topology to verify infiniband network...
 No Infiniband link errors found...
…
Installation summary report: /u01/Exadata_software/linux-x64/ExadataConfigurations/Client-AMI_Dev_Exadata-InstallationReport.xml
 Successfully completed execution of step Create Installation Summary [elapsed Time [Elapsed = 877799 mS [14.0 minutes] Thu Apr 14 22:12:13 CDT 2016]]
[root@myclusterdb01 linux-x64]#

4.18/ Step 18

This step is to “Resecure the machine”; this step is meant to :
  • expire all passwords (need to be changed at next login)
  • implement password aging and complexity
  • tightened some permissions
  • drop the root ssh keys
You may want to bypass this step depending on the client security policy.
[root@myclusterdb01 linux-x64]# ./install.sh -cf client-mycluster.xml -s 18
...
[root@myclusterdb01 linux-x64]#


You are now done, and everything is now installed properly. You can now create many databases and start to enjoy Exadata !



1 comment:

Load 2 billion rows with SQL LOADER

I worked on few Extract data / transform data (nor not) / load data into an Oracle database projects during my professional life and SQL ...