Twitter

asmdu.sh : a far better "du" for asmcmd, version 3

It's been a while since I released asmdu version 1 and version 2. These versions were cool but few features were missing. For those who don't know what asmdu is about, it is a shell script which is meant to provide a clear output of the ASM diskgroups usage.

 

ASM disk usage : The old fashioned way

When it is about to have an ASM output about the diskgroups usage, you may use the asmcmd lsdg command and you got this kind of output :



Let's have a quick look at some interesting elements of this output :
  1. The diskgroups
  2. The diskgroups redundancy
  3. Four columns presenting some sizes are shown in the yellow shape on the above screenshot, they are interesting but not really clear IMO
This output is interesting but I see few things that would benefit from a clearer presentation :
  1.  Unit is MB and cannot be changed
  2. Then it makes the output not really clear with the big diskgroup sizes we have nowadays
  3. Also the sizes shown here are non mirrored and mirrorred sizes which makes the output perfectible. Also the "Req_mir_free_MB" column may be tricky; please have a look at this very good blog on the subject.

A new & better way with asmdu.sh

The asmdu.sh script will then try to resolve these issues by showing you in a glimpse a nice and clear output :



Let's have a look at the above screenshot's colored features :
  1. The script will show you a list of running instances on the local node; I found it very handy to have an idea on what's running on the node I am connected to
  2. The diskgroups and the redundancy are also shown in the two first coolumns
  3. You can see that the Unit is TB here which is beter for this kind of sizes.
  4. Then is shown the Total diskgroup size and the Usable size. Also note that only mirrored sizes are shown here then you can easily compare them and quickly see how much space is available for the databases.
  5. The last column shows the %Free space for each diskgroup which is a very valuable information. You can then know if a diskgroup is under space pressure in a glimpse.
  6. This % Free is presented with colored thresholds (Red, Yellow and Green) to be able to quickly identify any potential space pressure on any diskgroup. The defaut thresholds are Green < 75% < Yellow < 90%  < Red. You can customize these thresholds by modifying these values in the script :


Using different Units

As we all have different diskgroup sizes, I've added options to be able to choose the Unit you want to show the output on :
  • -m : to show in MB
  • -g : to show in GB
  • -t : to show in TB




Please note that the default Unit  (when no -m, -g nor -t is specified on the command line) is TB; you can change the default Unit by editing the script and change the DEFAULT_UNIT variable (the last uncommented value wins) :



A verbose mode

Few information that asmcmd lsdg shows are not shown by asmdu.sh by default :
  • Free_MB : Free space (in MB) in the diskgroup, this is with no redundancy
  • Req_mir_free_MB : Required size (in MB) to restore redundancy after the worst failure that can tolerate ASM

I have a bit modified this output to show the below one with the -v option  : 

  1. Raw Total is the total non mirrored size of the diskgroup
  2. Raw Free is the non mirrored free space of the diskgroup
  3. Reserved size is the amount of free space required to be available in order to restore redundancy after one or more disk failures without affecting data availability

It is then strongly recommended to not use this reserved size, the real Usable size being (Free_MB - Req_mir_free_MB) / Redundancy.


If you want to show these information by default, just edit the script  and modify this DEFAULT_VERBOSE variable (the last uncommented value wins) :



External redundancy

Please note that in case of External Redundancy the Reserved Size is 0.




Handling subdirectories

In the case you have a space shortage on a diskgroup like in the below example where only 4% left, you would need to know which database uses the most space in order to work on freeing some space.

Naturally, I would do a "du *" under asmcmd as I would do under any Unix shell and here is the accident : "du *" under asmcmd acts like a "du ." under any Unix shell (it shows the total size used on the diskgroup instead of the size used by each subdirectory)  as you can see with the below screenshot :



You would then have to do a du for each directory to finally find the biggest(s). It is not handy and may be painful if you have many databases running there as you would have to make a loop with asmcmd commands to achieve this goal :


It is indeed doable but it may not be what you want to face when you may be in a rush with a 4% free space diskgroup.


A subdirectories example

The purpose of the -d option is to easily show you all the subdirectories under a directory (it also shows you the diskgroup characteristics where resides this directory) :


You can then identify the databases that use the most space in a glimpse. "In a glimpse" is a figure of speech here; indeed this -d option id based  on the "real" asmcmd du command and as this one may be quite slow, this output may take a bit of time as well depending on how big are your databases.


Another subdirectories example

The example below is to show that the -d option also works with a whole path and not only a diskgroup name. The below screenshot shows archivelogs directories size and you will quickly notice that there's some 2012 archivelogs still using lots of space on your diskgroup. You may want to delete them to save some space.



If you want no color

If you want a non colored output (to, for example, automatically send the output by email), you can use the -n option:


 This option is compatible with any other option.
If you want a non colored output as default, edit the script and comment the default behavior you do not want:
DEFAULT_NOCOLOR="Yes"   # Print with no color
DEFAULT_NOCOLOR="No"    # Print with colors


Option -h for the syntax

To sum that up, you may want to use the -h option that will show you all the options and a description of each option :





Restrictions

There're not many restrictions that could prevent asmdu to work :

  1. oraenv has to work to set the ASM environment
  2. asmdu has to be started as the ASM owner (the GI owner)to be able to run asmcmd commands.

The code


The code is hosted and maintained on my github repository, help yourself and enjoy asmdu !



I hope you'll be enjoying this script as much as I enjoy it on a daily basis !

Any suggestion, comment or bug, please let me know in the comments sections.


14 comments:

  1. if i want to redirect log in email or in .log file , the output is not proper

    > asm.log

    DiskGroup Redundancy^[[1;37m Total GB^[[m^[[1;37m Usable GB^[[m^[[1;37m % Free^[[m
    --------- -----------^[[1;37m --------^[[m^[[1;37m ---------^[[m ------
    CRS EXTERN^[[1;37m 30.00^[[m^[[1;37m 29.71^[[m^[[1;32m 99^[[m
    DATA EXTERN^[[1;37m 100.00^[[m^[[1;37m 0.00^[[m^[[1;31m 0^[[m
    FRA EXTERN^[[1;37m 70.00^[[m^[[1;37m 34.45^[[m^[[1;32m 49^[[m

    how to fix this ?

    ReplyDelete
    Replies
    1. Hi Neoniel,

      As we discussed through email, I have implemented a "-n" option to have a non colored output and then fix your issue.

      I will be updating this post to reflect this option.

      Have a good day,

      Fred

      Delete
  2. This comment has been removed by the author.

    ReplyDelete
  3. Hi Fred,
    good post as usual.
    Thanks

    ReplyDelete
  4. shea@pythian.comJuly 17, 2019 at 2:50 AM

    Thanks Fred, very useful script.
    I copied the latest source but got the error below. The ASM is version 11.2.0.3. (I also replaced the DG name for client privacy.)
    Subdir Used GB Raw Used GB
    ------ ------- -----------
    Unknown option: nocp
    usage: asmcmd [-V] [-v ] [--privilege ] [-p] [command]
    help: help
    / 0.00 0.00


    I reviewed your comments about the nocp option then commented out line 252 and uncommented line 253 and the script ran fine.
    252:# echo ${DIR} `asmcmd --nocp du ${D}/${DIR} | tail -1` # Please look at the "About the --nocp option" notes in the header for more information
    253- echo ${DIR} `asmcmd du ${D}/${DIR} | tail -1`

    ReplyDelete
  5. Is there a way to add -r option, to be used along with -d?

    This would perform a recursive tree search on directory, show each size (both directory and files)

    And with a -nf option the files would me ommited, showing only directory subtree sizes.

    ReplyDelete
  6. Great script!

    However, I am trying to reconcile the difference between the output of this and the output from the Oracle supplied script from MOS note 1551288.1

    The Raw Total and Raw free match, but the Disk Usable File MB from the MOS note does not tally with the Usable MB from your script?

    ReplyDelete
    Replies
    1. Hi Neil,

      Thanks ! I checked the MOS note you pointed out and it seems that asmdu.sh calculates the disk usable size in the same way as the MOS note. Would you have an example with numbers to find out ?

      Thanks,

      Delete
    2. Not sure it will copy and paste that well but let me try..

      ASMDU

      [oracle@dm1db1$ /staging/oracle/scripts/asmdu.sh -v -m

      Instances running on dm1db1 : +ASM1, DBPROD

      DiskGroup Redundancy Total MB Raw Total MB Raw Free MB Reserved MB Usable MB % Free
      --------- ----------- -------- ------------ ----------- ----------- --------- ------
      DATA HIGH 78471168.00 235413504.00 48895764.00 13078528.00 11939078.00 15
      RECO HIGH 33634560.00 100903680.00 36810972.00 5605760.00 10401737.00 30

      Note : Usable = (Raw Free - Reserved)/Redundancy

      $


      MOS 1551288.1

      ASM Version is 19.0.0.0
      -------------------------------------------------------------------------------------------------------------------------------------------------
      | | | | | | | | |Disk Req'd | | | |
      | |DG |Num |Num |Disk Size |DG Total |DG Used |DG Free |Mirror Free |Disk Usable |PCT | |
      |DG Name |Type |FGs |Disks|MB |MB |MB |MB |MB |File MB |Util |DFC |
      -------------------------------------------------------------------------------------------------------------------------------------------------
      |DATA |HIGH | 3| 36| 6,539,264| 235,413,504| 186,754,080| 48,659,424| 35,312,026| 4,449,133| 79.3%|PASS|
      |RECO |HIGH | 3| 36| 2,802,880| 100,903,680| 63,709,440| 37,194,240| 15,135,552| 7,352,896| 63.1%|PASS|
      -------------------------------------------------------------------------------------------------------------------------------------------------
      . . .
      Script completed.

      PL/SQL procedure successfully completed.

      [oracle@dm1db1]$

      Delete
    3. Hi Neil,

      I checked on one of my instance and indeed, numbers are different. I checked the MOS node's code and found how they calculate the usable size:

      => v_usable_mb := ROUND((dg.free_mb - v_required_free_mb)/2);

      and this v_required_free_mb is interesting:

      => v_required_free_mb := v_space_reserve_factor * dg.total_mb;

      We can see specified in the note that:

      => Oracle recommends having free space in the diskgroup equal to or greater than the percentage of the total diskgroup capacity as follows:

      Grid Infrastructure Version Number of Failgroups Required % Free of Diskgroup Capacity
      12.1.0 Any 15
      12.2, 18.1+ less than 5 15
      12.2, 18.1+ 5 or more 9


      And indeed, we can see this in the code:

      IF ((v_db_version like '12.2%') or (v_db_version like '18%')) THEN
      IF v_fg_count < 5 THEN
      v_space_reserve_factor := 0.15 ;
      ELSE
      v_space_reserve_factor := 0.09 ;
      END IF;
      ELSIF ( (v_db_version like '12.1%' ) or (v_db_version like '11.2.0.4%') ) THEN
      v_space_reserve_factor := 0.15 ;
      ELSE
      v_space_reserve_factor := 0.15 ;
      END IF;


      So actually, the MOS note code shows you the usable size respecting this 15% free which is as recommended but this may not be the real life, asmdu relies on what asmdg lsdg shows in the column Req_mir_free_MB which is what is really needed to restore the redundancy safely in case of disk loss.

      In short, the MOS node code uses 15% for the reserved size which is a generic number taken whereas asmdu relies on the real required reserved size from the Req_mir_free_MB column. This is then where the differences in numbers come from.

      Fred

      Delete
    4. Hi Fred,

      Appreciate you taking the time to review this, and what you have written makes sense.

      Many thanks!

      Delete

Exadata: ILOM hostname change

I am not sure about the root cause of this one but it was very weird finding that an ILOM had a wrong hostname (yes, wrong hostname) -- belo...