Twitter

Some bash tips -- 19 -- A generic pretty table output function

Output is key to present our findings and this is even more important in Bash as, for most people, all of this is just raw text on a geek black and white terminal. Also we have to admit that it oftenly takes more time to make a pretty output than to collect the data we are interested in so the output formatting is often treated as an afterthought. I have already shown how to get a nice output using column, all of my scripts have nice outputs but they are specific and the formatting needs to be redone for each script.
This is why I came up with a generic function formatting data from arrays which I have been using for a while now. I will just be using the first 5 lines of a generic /etc/passwd file as an example below.


Sourcing the Script

The idea is to keep this function in a dedicated file (let's name it print_table.sh) and then source it when you need it.
[frdenis@myvm ~]$ source print_table.sh
[frdenis@myvm ~]$


Preparing the data

The print_table() function expects a Bash array of comma-separated strings (CSV style). The first element of the array (index 0) is always treated as the header row.
Let's take the first 5 lines of /etc/passwd as our data source:
[frdenis@myvm ~]$ head -5 /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
[frdenis@myvm ~]$


Basic Table Usage

To render this as a table, we construct a Bash array, define our header as the first element, replace the ":" delimiters with commas "," and pass the array by reference:
source ./print_table.sh
# Initialize array and create data[0] which is our table header
data=("Username,Password,UID,GID,Comment,Home,Shell")

# Read /etc/passwd line by line and append into the ""data" array
while IFS= read -r line; do
    data+=("$(echo "$line" | tr ':' ',')")
done < <(head -5 /etc/passwd)

# Print the table
print_table data

| Username | Password | UID |  GID  | Comment |   Home    |       Shell       |
------------------------------------------------------------------------------
|   root   |    x     |  0  |   0   |  root   |   /root   |     /bin/bash     |
|  daemon  |    x     |  1  |   1   | daemon  | /usr/sbin | /usr/sbin/nologin |
|   bin    |    x     |  2  |   2   |   bin   |   /bin    | /usr/sbin/nologin |
|   sys    |    x     |  3  |   3   |   sys   |   /dev    | /usr/sbin/nologin |
|   sync   |    x     |  4  | 65534 |  sync   |   /bin    |     /bin/sync     |
------------------------------------------------------------------------------


Customizing Alignment & Row Count

By default, all column contents are centered. You can customize alignment per column by passing a second array containing positional alignment codes:
  • l : Left alignment
  • c : Center alignment (default)
  • r : Right alignment
You can also display the total number of processed rows at the bottom by setting L_SHOW_TOTAL="True".
[frdenis@myvm ~]$ cat demo_align.sh
#!/bin/bash
source ./print_table.sh

data=("Username,Password,UID,GID,Comment,Home,Shell")
while IFS= read -r line; do
    data+=("$(echo "$line" | tr ':' ',')")
done < <(head -5 /etc/passwd)

# Define column alignments: Left, Center, Right, Right, Left, Left, Left
align=("l" "c" "r" "r" "l" "l" "l")

# Enable total row count display
export L_SHOW_TOTAL="True"

print_table data align
[frdenis@myvm ~]$ ./demo_align.sh

| Username | Password | UID |   GID | Comment | Home      | Shell             |
----------------------------------------------------------------------------------
| root     |    x     |   0 |     0 | root    | /root     | /bin/bash         |
| daemon   |    x     |   1 |     1 | daemon  | /usr/sbin | /usr/sbin/nologin |
| bin      |    x     |   2 |     2 | bin     | /bin      | /usr/sbin/nologin |
| sys      |    x     |   3 |     3 | sys     | /dev      | /usr/sbin/nologin |
| sync     |    x     |   4 | 65534 | sync    | /bin      | /bin/sync         |
----------------------------------------------------------------------------------
Number of rows: 5
[frdenis@myvm ~]$


Generating Value Summaries (Comma Lists)

A unique feature of this function is the ability to automatically generate comma-separated summaries of specific columns beneath the table. This is very useful when you want to feed list outputs directly into subsequent script commands or quickly copy/paste them.
Pass the 1-based column indices as parameter 3, and an optional row limit as parameter 4:
[frdenis@myvm ~]$ cat demo_summary.sh
#!/bin/bash
source ./print_table.sh

data=("Username,Password,UID,GID,Comment,Home,Shell")
while IFS= read -r line; do
    data+=("$(echo "$line" | tr ':' ',')")
done < <(head -5 /etc/passwd)

# Request summary list for Column 1 (Usernames) and Column 3 (UIDs), max 3 rows
print_table data "" "1,3" 3
[frdenis@myvm ~]$ ./demo_summary.sh

| Username | Password | UID | GID   | Comment | Home      | Shell             |
----------------------------------------------------------------------------------
|   root   |    x     |  0  |   0   |  root   |   /root   |     /bin/bash     |
|  daemon  |    x     |  1  |   1   | daemon  | /usr/sbin | /usr/sbin/nologin |
|   bin    |    x     |  2  |   2   |   bin   |   /bin    | /usr/sbin/nologin |
|   sys    |    x     |  3  |   3   |   sys   |   /dev    | /usr/sbin/nologin |
|   sync   |    x     |  4  | 65534 |  sync   |   /bin    |     /bin/sync     |
----------------------------------------------------------------------------------

Username values (up to 3): root,daemon,bin
UID values (up to 3): 0,1,2

[frdenis@myvm ~]$

Scripting Integration: RAW Output Mode

When passing data downstream in pipeline operations, ASCII borders can get in the way. You can bypass the table rendering entirely by enabling L_RAW="True". This outputs the raw array content without borders or alignments:
[frdenis@myvm ~]$ L_RAW="True" ./demo.sh
Username,Password,UID,GID,Comment,Home,Shell
root,x,0,0,root,/root,/bin/bash
daemon,x,1,1,daemon,/usr/sbin,/usr/sbin/nologin
bin,x,2,2,bin,/bin,/usr/sbin/nologin
sys,x,3,3,sys,/dev,/usr/sbin/nologin
sync,x,4,65534,sync,/bin,/bin/sync
[frdenis@myvm ~]$
Pro Tip: Using L_RAW="True" makes your scripts dual-purpose. They can produce human-friendly interactive CLI tables by default, or clean CSV data when piped into other tools.



< Previous shell tip / Next shell tip coming soon >

Python venv

A lot of application running on our VMs use python. As python evolves a lot and is extensible through modules, it is very frequent that a user do not have the correct modules / modules versions he needs for his project.
To keep it clean and flexible, we do not want to install python packages at the system level, we want each non-privileged user to be using python virtual environment. Note that the base python version requested by the user will need to be installed at the system level.


Proxy

Depending on your company, you may need a proxy to be able to install any python version or module:
export https_proxy=http://www-proxy-your_company.com:80


Example

Below example will be using a non-privileged user "frdenis" on the myvm VM.
[frdenis@myvm ~]$ id
uid=1003(frdenis) gid=1003(frdenis) groups=1003(frdenis) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[frdenis@myvm ~]$
The default python of the system is 3.6.8:
[opc@myvm ~]$ python -V
Python 3.6.8
[opc@myvm ~]$
Python 3.11 will need to be installed as the example will showcase a python3.11 virtual environment:
[opc@myvm ~]$ sudo dnf install -y python3.11 python3.11-pip
. . .
[opc@myvm ~]$ python3.11 -V
Python 3.11.13
[opc@myvm ~]$


Python venv


Check the system python

[frdenis@myvm ~]$ type python
python is /usr/bin/python
[frdenis@myvm ~]$ /usr/bin/python --version
Python 3.6.8
[frdenis@myvm ~]$

Create and activate a venv

We will name this virtual environment "venv" in this example which is the usage.
[frdenis@myvm ~]$ python -V
Python 3.6.8
[frdenis@myvm ~]$ python3.11 -m venv venv             <== the second "venv" is the name of your venv
[frdenis@myvm ~]$ source venv/bin/activate
(venv) [frdenis@myvm ~]$ python -V            <== note that starting "(venv)" showing you are IN the virtual env
Python 3.11.13
(venv) [frdenis@myvm ~]$

Install packages in the venv

Once the venv activated, we can install any module we want in it and as our non-privileged user:
(venv) [frdenis@myvm ~]$ python
Python 3.11.13 (main, Apr 27 2026, 16:44:16) [GCC 8.5.0 20210514 (Red Hat 8.5.0-28.0.1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy                               <== numpy is not default
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
>>> quit()
(venv) [frdenis@myvm ~]$ export https_proxy=http://www-proxy-hqdc.us.oracle.com:80
(venv) [frdenis@myvm ~]$ pip install numpy
Collecting numpy
  Downloading numpy-2.4.6-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.9/16.9 MB 9.2 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-2.4.6
[notice] A new release of pip available: 22.3.1 -> 26.2.1
[notice] To update, run: pip install --upgrade pip
(venv) [frdenis@myvm ~]$ python
Python 3.11.13 (main, Apr 27 2026, 16:44:16) [GCC 8.5.0 20210514 (Red Hat 8.5.0-28.0.1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>

Showcase

Let's write a simple python script using numpy:
[frdenis@myvm ~]$ cat test_env.py
import sys
import numpy as np
print("Python Version :", sys.version.split()[0])
print("Numpy Version  :", np.__version__)
print("Numpy Test     :", np.array([10, 20, 30]) * 2)
[frdenis@myvm ~]$
It works in the venv:
(venv) [frdenis@myvm ~]$ python test_env.py
Python Version : 3.11.13
Numpy Version  : 2.4.6
Numpy Test     : [20 40 60]
(venv) [frdenis@myvm ~]$
But not outside of the venv (as numpy is not default) and has only been installed in the venv:
(venv) [frdenis@myvm ~]$ deactivate
[frdenis@myvm ~]$ python test_env.py
Traceback (most recent call last):
  File "test_env.py", line 2, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'
[frdenis@myvm ~]$


Automatically use the venv

As it may not be nice to source the venv each time we want to run a python script (from a shell for example), 2 solutions exist for that:

Run the script using the venv python

[frdenis@myvm ~]$ ~/venv/bin/python test_env.py
Python Version : 3.11.13
Numpy Version  : 2.4.6
Numpy Test     : [20 40 60]
[frdenis@myvm ~]$

Use a shebang which points to the venv

[frdenis@myvm ~]$ cat test_env.py
#!/home/frdenis/venv/bin/python                             <== here, this is the shebang, adapt to your path
import sys
import numpy as np
print("Python Version :", sys.version.split()[0])
print("Numpy Version  :", np.__version__)
print("Numpy Test     :", np.array([10, 20, 30]) * 2)
[frdenis@myvm ~]$ chmod u+x test_env.py                   <== has to be done only once when setting it up
[frdenis@myvm ~]$ ./test_env.py
Python Version : 3.11.13
Numpy Version  : 2.4.6
Numpy Test     : [20 40 60]
[frdenis@myvm ~]$


Transfer a venv to another user/machine

Note: Do NOT copy the "venv" directory to another user or machine, it contains hardcoded absolute paths and machine specifics which wont make it work.

Export the module list

[frdenis@myvm ~]$ source venv/bin/activate
(venv) [frdenis@myvm ~]$ pip freeze > requirements.txt
(venv) [frdenis@myvm ~]$ cat requirements.txt
numpy==2.4.6
(venv) [frdenis@myvm ~]$

Re-apply to another virtual env

Once this requirements.txt file has been transferred:
[new_user@another_vm ~]$ python3.11 -m venv new_venv             <== create a new venv
[new_user@another_vm ~]$ source new_venv/bin/activate            <== activate the new venv
[new_user@another_vm ~]$ pip install -r requirements.txt         <== reinstall the modules from the original venv
. . .
[new_user@another_vm ~]$

OCI: Datapump between 23ai ADB and 19c ADB using database link

Now that we know how to manually create a 23ai ADB in OCI, that we also know how to create a database link between a 23ai ADB and a 19C ADB, let's now try a datapump between a 19c ADB and a 23ai ADB (oh, this is also a migration method, right?) using a database link.
First, let's remind our environment:
[fred@myvm ~]$ cd wallet_23ai
[fred@myvm wallet_23ai]$ export TNS_ADMIN=$(pwd)
[fred@myvm wallet_23ai]$ sqlplus admin/"Iw0ntt3lly0u"@sandbox01_high
SQL> select banner_full from v$version;
BANNER_FULL
--------------------------------------------------------------------------------
Oracle Database 23ai Enterprise Edition Release 23.0.0.0.0 - Production
Version 23.6.0.24.10
SQL> select banner_full from v$version@TO_19C;
BANNER_FULL
--------------------------------------------------------------------------------
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.25.0.1.0
SQL>
Let's datapump that ~ 50 million rows table from our 19C ADB:
SQL> select count(*) from USER_19C.TABLE_19C@TO_19C;
COUNT(*)
----------
 48930604
SQL>
Let's create a simple datapump parfile (nothing specific compared to a "classic" Oracle database):
[fred@myvm ~]$ cat imp.par
TABLES=USER_19C.TABLE_19C
TABLE_EXISTS_ACTION=append
REMAP_SCHEMA=USER_19C:USER_23C
network_link=TO_19C
encryption_pwd_prompt=no
directory=DATA_PUMP_DIR
logfile=imp.log
[fred@myvm ~]$
And run it:
[fred@myvm ~]$ impdp admin/"Iw0ntt3lly0u"@sandbox01_high parfile=imp.par
Import: Release 21.0.0.0.0 - Production on Thu Nov 21 15:47:06 2024
Version 21.14.0.0.0
Copyright (c) 1982, 2024, Oracle and/or its affiliates.  All rights reserved.
Connected to: Oracle Database 23ai Enterprise Edition Release 23.0.0.0.0 - Production
Starting "ADMIN"."SYS_IMPORT_TABLE_01":  admin/********@sandbox01_high parfile=imp.par
Processing object type TABLE_EXPORT/TABLE/TABLE
. . imported "USER_23C"."TABLE_19C"               48930604 rows
Processing object type TABLE_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
Job "ADMIN"."SYS_IMPORT_TABLE_01" completed successfully at Thu Nov 21 16:09:50 2024 elapsed 0 00:22:41
[fred@myvm ~]$
That was very easy and pretty fast: 22 minutes for 48 million lines (with no parallel, this example table was not partitioned).

Some bash tips -- 19 -- A generic pretty table output function

Output is key to present our findings and this is even more important in Bash as, for most people, all of this is just raw text on a geek b...