YAL : Yet Another Launcher !

Whatever is your database technology, architecture, whether you are with a Cloud vendor, multi Cloud, on-premises or a mix of all of that, you are really likely to have the below way of connecting to your servers:

• 1/ ssh to a jump server as a non privileged user (fdenis for me for example)
• 2/ ssh to a server as a non privileged user (opc for example in Oracle Cloud)
• 3/ sudo to a more privileged user (oracle or grid in the Oracle world for example)
• 4/ do something

. . . and you repeat this for any server which leads to the below workflow:

This is good from a security perspective indeed but it may be a bit more problematic when you want to:

• Execute some commands on each server or a subset of those servers
• Execute a script on each server or a subset of those servers
• Copy a file on each server or a subset of those servers
• . . . well . . . efficiently do your admin job . . .

One could argue that tools already exist to achieve this like Ansible, Chef, Puppet, OEM, name it ... indeed these tools exist but first of all they are not standard (meaning not on all Unix boxes by default) and far more too complicated to do simple things like (and this was my starting point) executing asmdu.sh or rac-status.sh on all my servers.
The most approaching simple and efficient tool I know (may be there are others) to achieve this kind of thing is dcli but this tool is Exadata specific. Also, dcli does not allow to execute a script in sudo on the remote server -- as well a few other things I will describe below in yal's features.

YAL

So, one more time when something does not exist or does not fit my needs, I used my favorite and standard vi to make this standard shell script yal.sh aka Yet Another Launcher with the goal of being able to easily achieve what I described above. The only requirement is to have SSH keys deployed from the jump server to the target server (it is actually not a requirement but being prompted for a password at every server connection would be painful).

A first example

Let's start with a first example and just execute the "uptime" and the "id" commands on a set of servers:

[fdenis@jump]$ ./yal.sh -c server1,server2 -l opc -b "uptime;id"
BEGIN on server1 : Thu Sep 12 02:56:01 UTC 2019
 02:56:01 up 125 days, 14:41,  0 users,  load average: 2.04, 2.36, 2.25
uid=54322(opc) gid=54323(opc) groups=54323(opc)
END   on server1 : Thu Sep 12 02:56:05 UTC 2019
ELAPSED : 7 seconds
BEGIN on server2 : Thu Sep 12 02:56:08 UTC 2019
 02:56:08 up 125 days, 14:38,  0 users,  load average: 3.30, 2.38, 2.09
uid=54322(opc) gid=54323(opc) groups=54323(opc)
END   on server2 : Thu Sep 12 02:56:11 UTC 2019
ELAPSED : 6 seconds
[fdenis@jump]$

A first explanation of the options used:

• -c: a comma separated list of target servers
• -l: the user to use to login to the target servers
• -b: commands to execute "before". Before what ? before copying or executing a script as I will show below. There is also a -a option to execute a command "after" a script is executed or copied on a target server

You can note that the output also shows the servername, the current date and the elapsed time of the operation for each server. These headers can be modified in the script (you may not want to modify them though):

      HEADER="echo BEGIN on \`hostname -s\` : \`date\`"     # Header to print before execution on  target
      FOOTER="echo END\ \ \ on \`hostname -s\` : \`date\`"  # Footer to print after  execution on a target
SHOW_ELAPSED="yes"                                          # Show the elapsed time

Target servers

The previous example shows that -c is used to select a comma separated list of target servers; you can also use a -g option to specify a file containing a list of files:

[fdenis@jump]$ cat alist 
server1
server2
[fdenis@jump]$ ./yal.sh -g alist -l opc -b "uptime;id"
BEGIN on server1 : Thu Sep 12 03:11:38 UTC 2019
 03:11:38 up 125 days, 14:57,  0 users,  load average: 3.89, 2.88, 2.48
uid=54322(opc) gid=54323(opc) groups=54323(opc)
END   on server1 : Thu Sep 12 03:11:41 UTC 2019
ELAPSED : 6 seconds
BEGIN on server2 : Thu Sep 12 03:11:44 UTC 2019
 03:11:44 up 125 days, 14:53,  0 users,  load average: 2.46, 2.21, 2.15
uid=54322(opc) gid=54323(opc) groups=54323(opc)
END   on server2 : Thu Sep 12 03:11:47 UTC 2019
ELAPSED : 6 seconds
[fdenis@jump]$

sudo

As described earlier, you may not want to run your commands / copy / execute your scripts with the user you use to login as this one is most likely a non-privileged user; you may want to sudo to do so; yal can do that by specifying the user you want to execute with using the -e option:

[fdenis@jump]$ ./yal.sh  -g alist -l opc -b "uptime;id" -e oracle
BEGIN on server1 : Thu Sep 12 03:15:55 UTC 2019
 03:15:55 up 125 days, 15:01,  0 users,  load average: 1.88, 2.26, 2.32
uid=101(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(dbaoper),1003(dba),1006(asmdba)
END   on server1 : Thu Sep 12 03:15:58 UTC 2019
ELAPSED : 6 seconds
BEGIN on server2 : Thu Sep 12 03:16:02 UTC 2019
 03:16:02 up 125 days, 14:58,  0 users,  load average: 1.76, 1.92, 2.05
uid=101(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(dbaoper),1003(dba),1006(asmdba)
END   on server2 : Thu Sep 12 03:16:05 UTC 2019
ELAPSED : 6 seconds
[fdenis@jump]$

On older systems, you may not be able to sudo through a script as no terminal is allocated here and you would then receive a sudo: sorry, you must have a tty to run sudo. error. This behavior has been acknowledged by Red Hat and Red Hat has fixed the issue as very well explained in this post. So you would basically face this issue if your /etc/sudoers still contains:

Defaults requiretty

Execute a script

yal can indeed execute a script (with the -x option) on each specified target server. The script has to be on your jump server, where you execute yal, not on the target serverl; yal will then:

• Copy the script on the target server (in /tmp by default, this can be modified with the -d option)
• Execute the script on the target server
• Remove the script from the target server

The below example will do on each target server contained in the alist file:

• Connect as user opc
• Sudo to user oracle
• Execute what is specified in the -b option before executing the script
• Execute the rac-status.sh script
• Execute what is specified in the -a option after executing the script
• Show everything server by server with the elapsed time on each server

Let's have a look at this in action:

[fdenis@jump]$ ./yal.sh  -g alist -l opc -b "echo before" -e oracle -a "echo after" -x rac-status.sh 
BEGIN on server1 : Thu Sep 12 03:32:41 UTC 2019
before
Script rac-status.sh successfully copied to /tmp/rac-status.sh71743

                Cluster blablacluster

      Listener   |      Port      |   server1   |     Type     |
  -------------------------------------------------------------
   LISTENER      | TCP:1521       |    Online   |   Listener   |
   LISTENER_SCAN1| TCP:1521       |      -      |     SCAN     |
  -------------------------------------------------------------
  
         DB      |     Version    |   server1   |    DB Type   |
  -------------------------------------------------------------
   a_database    | 11.2.0.4   (1) |     Open    |  SINGLE (P)  |
  -------------------------------------------------------------
  ORACLE_HOME references listed in the Version column 
  
         1 : /u01/app/oracle/product/11.2.0.4/dbhome_1  oracle oinstall

Script /tmp/rac-status.sh71743 removed successfully
after
END   on server1 : Thu Sep 12 03:32:50 UTC 2019
ELAPSED : 13 seconds
BEGIN on server2 : Thu Sep 12 03:32:54 UTC 2019
before
Script rac-status.sh successfully copied to /tmp/rac-status.sh71743

                Cluster anothercluster

      Listener   |      Port      |   server2   |     Type     |
  -------------------------------------------------------------
   LISTENER      | TCP:1521       |    Online   |   Listener   |
   LISTENER_SCAN1| TCP:1521       |      -      |     SCAN     |
  -------------------------------------------------------------
  
         DB      |     Version    |   server2   |    DB Type   |
  -------------------------------------------------------------
   myprod_db     | 12.1.0.2   (1) |     Open    |  SINGLE (P)  |
  -------------------------------------------------------------
  ORACLE_HOME references listed in the Version column 
  
         1 : /u01/app/oracle/product/12.1.0.2/dbhome_1  oracle oinstall

Script /tmp/rac-status.sh71743 removed successfully
after
END   on server2 : Thu Sep 12 03:33:02 UTC 2019
ELAPSED : 11 seconds

Note that are shown the copies and deletion of the temporary files used to execute the script; you can deactivate that with the -q option (quiet).

Options to execute scripts

As you may want to use options to the scripts you execute, I made the -O option (upper case letter o as lower case o is already taken):

[fdenis@jump]$ ./yal.sh  -g alist -l opc -e oracle -x rac-status.sh -O "-n -d" -q

Note that you'll have to double quote your script options not to mess up with the yal options.

Copy a file

You can also simply copy a file to target servers using the -f option and the -d option to specify the target directory:

[fdenis@jump]$ ./yal.sh  -g alist -l opc -b "ls -ltr /tmp/asmdu.sh" -a "ls -ltr /tmp/asmdu.sh" -f asmdu.sh -d /tmp
BEGIN on server1 : Thu Sep 12 03:46:57 UTC 2019
ls: cannot access /tmp/asmdu.sh: No such file or directory
Script asmdu.sh successfully copied to /tmp/asmdu.sh
-rwxr--r-- 1 opc opc 12292 Sep 12 03:47 /tmp/asmdu.sh
END   on server1 : Thu Sep 12 03:47:04 UTC 2019
ELAPSED : 9 seconds
BEGIN on server2 : Thu Sep 12 03:47:07 UTC 2019
ls: cannot access /tmp/asmdu.sh: No such file or directory
Script asmdu.sh successfully copied to /tmp/asmdu.sh
-rwxr--r-- 1 opc opc 12292 Sep 12 03:47 /tmp/asmdu.sh
END   on server2 : Thu Sep 12 03:47:14 UTC 2019
ELAPSED : 10 seconds
[fdenis@jump]$ 

SSH options

To best adapt to your environment and the potential SSH security in place in your environments, you can change the SSH and SCP options using the -s and -S options

A config file

You can also specify your favorite options in a config file named .yal.config by default to avoid repeting them every time you use a command line:

[fdenis@jump]$ cat .yal.config 
SSH_OPTIONS="-qT -i /tmp/fred.pub"
       USER_TO_EXEC = root             # comment
[fdenis@jump]$ 

Note that a value assigned to a parameter in the config file overwrites the default but is overwritten by a command line option. Consider the config file as your defaults.

Show the options

As you may end up with many options in your command line, I made a -o option to show (and only show, not execute anything) the options of the current command line:

[fdenis@jump]$ ./yal.sh  -g alist -l opc -b "ls -ltr /tmp/asmdu.sh" -e oracle -a "ls -ltr /tmp/asmdu.sh" -f asmdu.sh -d /tmp -o
Option              | Value                  |
----------------------------------------------
-a: After           | ls -ltr /tmp/asmdu.sh  |
-b: Before          | ls -ltr /tmp/asmdu.sh  |
-c: Servers List    | server1,server2        |
-d: Dest            | /tmp                   |
-e: User to exec    | oracle                 |
-f: File to copy    | asmdu.sh               |
-g: Group File      | alist                  |
-l: User to login   | opc                    |
-q: Quiet           | NO                     |
-s: SSH Options     | -qT                    |
-S: SCP Options     | -q                     |
-x: Script to exec  |                        |
    Config File     | .yal.config            |
----------------------------------------------
[fdenis@jump]$

-h for help and -V for the script version

Feel free to use -h to show the script help and -V to show the script version.

yal instead of dcli ?

I am a big fan of dcli so I used the same options names for yal and added more that I found nice to have and I must admint that I sometimes use yal instead of dcli on Exadata but to each his own :)

Get the code

The code is available and always up to date here.

I hope you'll be enjoying yal as much as I do !