We are about to upgrade one of our GlusterFS-based storage systems at work. Fortunately, I was worrying about the upgrade procedure for the Debian packages not being tested by the maintainers. It turns out I was right. Simply upgrading the packages without manual intervention will apparently render your GlusterFS server unusable.

Basic setup

I have only tested the most basic distributed GlusterFS setup. No replication whatsoever. We have two GlusterFS servers, storage1 and storage2. A peering between both has been established, and a very basic volume has been configured:

storage1:~# gluster
gluster> peer status
Number of Peers: 1

Hostname: storage2
Uuid: 2d22cc13-2252-4cf1-bfe9-3d27fa2fbc29
State: Peer in Cluster (Connected)
gluster> volume create data storage1:/srv/data storage2:/srv/data
...
gluster> volume start data
...
gluster> volume info

Volume Name: data
Type: Distribute
Volume ID: e2bd5767-4b33-4e57-9320-91ca76f52d56
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: storage1:/srv/data
Brick2: storage2:/srv/data

For the test setup, I populated the volume with a number of files.

Upgrading from Wheezy to Jessie

To be save, stop the volume before you begin with the package upgrade:

gluster> volume stop data

And now perform your dist-upgrade.

After the upgrade, you will have to perform two manual clean ups. Both actions have to be performed on all storage servers.

/etc/glusterd is now /var/lib/glusterd

The package maintainers have apparently neglected to take care of this one. You manually need to copy the old configuration files over.

storage1:~# cd /var/lib/glusterd && cp -r /etc/glusterd/* .

Put volume-id in extended attribute

GlusterFS 3.5 requires the volume-id in an extended directory attribute. This is also not automatically handled during package upgrade.

storage1:~# vol=data
storage1:~# volid=$(grep volume-id /var/lib/glusterd/vols/$vol/info | cut -d= -f2 | sed 's/-//g')
storage1:~# setfattr -n trusted.glusterfs.volume-id -v 0x$volid /srv/data

With these two steps performed on all GlusterFS servers, you should now be able to start and mount your volume again in Debian Jessie.

Do not forget to explicitly stop the volume again before continueing with the next upgrade step.

Upgrading from Jessie to Stretch

After you have dist-upgraded to Stretch, there is yet another manual step you have to take to convert the volume metadata to the new layout in GlusterFS 3.8. Make sure you have stopped your volumes and the GlusterFS server.

storage1:~# service glusterfs-server stop

Now run the following command:

storage1:~# glusterd --xlator-option *.upgrade=on -N

Now you should be ready to start your volume again:

storage1:~# service glusterfs-server start
storage1:~# gluster
gluster> volume start data

And mount it:

client:~# mount -t glusterfs storage1:/data /mnt

You should now be running GlusterFS 3.8 and your files should still all be there.