PostgreSQL Filesystem Migration

Goal: relocate the PostgreSQL cluster from /var/lib/pgsql/data (on /) to a dedicated filesystem /pgsql/data, and reclaim the freed space on /.

A1 - Create the new volume from the new LUN

multipath -ll

for d in /dev/mapper/mpath*; do echo -n "$d: "; pvs "$d" 2>/dev/null && echo IN-USE || echo FREE; done

# identify the new FREE device matching Storage's serial, then:
pvcreate /dev/mapper/mpathX

vgextend vgl01 /dev/mapper/mpathX

lvcreate -L 500G -n <lv name> <vgname>

mkfs.xfs /dev/mapper/<vgname>-<lv name>

A2 - Mount the new filesystem: /pgsql

mkdir -p /pgsql

# add to /etc/fstab:

# /dev/mapper/<vgname>-<lv name>  /pgsql  xfs  rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota  0 0

mount /pgsql

chown postgres:postgres /pgsql

chmod 700 /pgsql

df -h /pgsql

A3 - Confirm no external tablespaces (must be clean before copy)

su - postgres -c "psql -x -c '\db+'"      # expect pg_default + pg_global, both empty LOCATION

ls -l /var/lib/pgsql/data/pg_tblspc/      # expect empty

# If any custom tablespace shows a real LOCATION, STOP and handle separately.

A4 - Stop XorMon

su - xormon -c 'cd ~/xormon-ng && ./xormon.sh stop'

A5 - Stop PostgreSQL and copy the cluster

systemctl stop postgresql

systemctl is-active postgresql                # expect stopped

rsync -aHAX /var/lib/pgsql/data/ /pgsql/data/ # trailing slashes deliberate

du -sh /var/lib/pgsql/data /pgsql/data        # sizes match

ls -l /pgsql/data/PG_VERSION /pgsql/data/postgresql.conf   # present

ls -ld /pgsql/data                            # postgres:postgres, 700

A6 - Repoint PostgreSQL to the new location (manual vi edit)

# Make backup of the cfg at first
cp -p /etc/sysconfig/postgresql /etc/sysconfig/postgresql.bak.$(date +%s)

vi /etc/sysconfig/postgresql

# Change the line and add the annotation above it so it reads:

#

#   # Data dir relocated to dedicated /pgsql filesystem (). Do NOT revert to ~postgres/data.

#   POSTGRES_DATADIR="/pgsql/data"

#

grep -A1 'relocated' /etc/sysconfig/postgresql

grep '^POSTGRES_DATADIR' /etc/sysconfig/postgresql     # shows /pgsql/data

A7 - Start PostgreSQL and VERIFY the move took

systemctl start postgresql

systemctl status postgresql --no-pager | head -5      # active (running)

su - postgres -c "psql -c 'SHOW data_directory;'"     # MUST show /pgsql/data

su - postgres -c "psql -d xormon -c 'select 1;'"

# Do NOT proceed unless data_directory = /pgsql/data. 

A8 - Restart XorMon and validate end-to-end

su - xormon -c 'cd ~/xormon-ng && ./xormon.sh start'   

sleep 30

ss -ltn | grep -E ':(9443|7162)'                       # both listening

su - xormon -c 'cd ~/xormon-ng && ./xormon.sh status' 

# Bear in mind that this step will only tell you everything is fine if there were jobs that had run since the database was moved/started, so I would check it after atleast 5 minutes

su - postgres -c "psql -P pager=off -d xormon -c \"SELECT max(last_successful_finish) FROM timescaledb_information.job_stats;\""   # current

A9 - Settle-in safety rename (does NOT reclaim space yet)

mv /var/lib/pgsql/data /var/lib/pgsql/data.OLD_premove

# Short bed-in window only (hours, up to ~1 day).

# NOTE: / is still full until A11 - do not treat A9 as the end state.

A10 - Reclaim root space (the actual goal - MANDATORY, do not skip)

# Once verified stable across a normal cycle:

rm -rf /var/lib/pgsql/data.OLD_premove

df -h /            # / now genuinely reclaimed