Exadata: read this before creating or moving datafiles in RECO !

Data growth looks infinite nowadays and one day or another you will face this kind of situation with your ASM diskgroups:

Output from asmdu.sh

Your DATA diskgroup is clearly under pressure here and it may take a while since you can purge or compress data to release some space (not that it is technically difficult but it usually needs to be organized, planned, decided, etc ...). It is then very tempting to create the new datafiles or even move some datafiles to RECO, right ?

Very tempting indeed but keep in mind the "too good to be true" idiom here as there are important things to know before going further with using RECO for datafiles:

• Different redundancy:
In the above example, DATA has a HIGH redundancy whereas RECO has NORMAL redundancy. It would then make no sense to have some datafiles of a database on a HIGH redundancy diskgroup and other datafiles on a NORMAL redundancy database. To overcome this, you could move all the datafiles of the less important databases to RECO and keep the more important databases on DATA.


• cachingPolicy=none:
From the 11.2.3.2.0 version of Exadata (which is pretty old and you may have a more recent version as I am sure you patch Exadata on a regular basis), the cachingPolicy of the RECO diskgroup is set to none meaning that the Flash Cache will not be used for any blocks stored on the diskgroup (the underlying grid disks actually). This behavior (feature) is described in note 2074156.1.

Let's have a look at a real example:

CellCLI> list griddisk attributes name, asmDiskgroupName, cachingPolicy
         DATA_CD_00_exacel02  DATA       default
         DATA_CD_01_exacel02  DATA       default
         DATA_CD_02_exacel02  DATA       default
         DATA_CD_03_exacel02  DATA       default
         DATA_CD_04_exacel02  DATA       default
         DATA_CD_05_exacel02  DATA       default
         DATA_CD_06_exacel02  DATA       default
         DATA_CD_07_exacel02  DATA       default
         DATA_CD_08_exacel02  DATA       default
         DATA_CD_09_exacel02  DATA       default
         DATA_CD_10_exacel02  DATA       default
         DATA_CD_11_exacel02  DATA       default
         RECO_CD_00_exacel02  RECO       none
         RECO_CD_01_exacel02  RECO       none
         RECO_CD_02_exacel02  RECO       none
         RECO_CD_03_exacel02  RECO       none
         RECO_CD_04_exacel02  RECO       none
         RECO_CD_05_exacel02  RECO       none
         RECO_CD_06_exacel02  RECO       none
         RECO_CD_07_exacel02  RECO       none
         RECO_CD_08_exacel02  RECO       none
         RECO_CD_09_exacel02  RECO       none
         RECO_CD_10_exacel02  RECO       none
         RECO_CD_11_exacel02  RECO       none
CellCLI>

Here indeed, none of the underlying grid disks used by the RECO diskgroup has a cachingPolicy set to default so none of your datafiles would use the Flash Cache which would lead to erractic performances with some datafiles able to use the Flash Cache and some not able to do so.

So keep his feature in mind and if you really need to use RECO to store datafiles, you'll have to set the cachingPolicyto default for all the grid disks of the RECO diskgroup across all the cell storages:

CellCLI> alter griddisk RECO_CD_00_exacel02 cachingPolicy=default
CellCLI> alter griddisk RECO_CD_01_exacel02 cachingPolicy=default
. . .

If you enjoyed this post, spread the word !