Master-Slave Metadata Servers - Tutorial

This document describes how to configure master-slave metadata servers.

1. Installation

2. Configuration

2.1 Master metadata server

# useradd -c "Gfarm gfmd" -m _gfarmmd

Configure the master metadata server by the root user. Add extra -r option to the the config-gfarm command which is described in the SETUP.en docuemnt to enable replicated metadata servers.

# config-gfarm -r

The metadata journal file is stored in /var/gfarm-metadata/journal directory by default. The location can be changed by the -j option. If there is an existing metadata server configuration, use config-gfarm-update with the -r option to update the configuration after stopping the gfmd. For config-gfarm-update, the backend PostgreSQL should be running.

install newer version of Gfarm # /etc/init.d/gfmd stop # config-gfarm-update -r --update # /etc/init.d/gfmd start

For now, the PostgreSQL backend database is only supported for the replicated metadata servers.

Register host-b as a slave metadata server.
The gfmdhost command has to be invoked by a Gfarm system administrator, who has "gfarmadm" group privilege. Default setting of Gfarm system administrator is the user who is specified by the -A option of the config-gfarm command.

$ gfmdhost -c host-b $ gfmdhost -l + master - m (default) host-a 601 - slave sync c (default) host-b 601

Insert a metadb_server_list directive to list all metadata hostnames and ports in gfarm2.conf.

metadb_server_list host-a:601 host-b:601

Dump the metadata by the root user.
If you specified --prefix or --l option to the config-gfarm command, you have to specify same options to the gfdump.postgresql command too.

# gfdump.postgresql -d -f gfarm-pgsql.dmp

Create the shared secret key for a user '_gfarmmd' which is used for inter-gfmd communication.

# su _gfarmmd $ gfkey -f -p 31536000

Note that the inter-gfmd communication only supports the shared secret authentication for now.

2.2 Slave metadata server

# useradd -c "Gfarm gfmd" -m _gfarmmd copy the shared secret key (~_gfarmmd/.gfarm_shared_key) from host-a

Configure a slave metadata server by the root user. To enable replicated metadata servers, execute config-gfarm with the -r option.

# config-gfarm -r

Restore the metadata by the root user.
If you specified --prefix or --l option to the config-gfarm command, you have to specify same options to the gfdump.postgresql command too.

# gfdump.postgresql -r -f gfarm-pgsql.dmp

NOTE: Currently, you cannot do "config-gfarm -r" on multiple slave metadata servers simultaneously, even if you'd like to configure multiple slave gfmd at once. Instead, you have to do "config-gfarm -r" and then "gfdump.postgresql -r" on each slave server, and then you can proceed to next server. Copy gfarm2.conf from the master metadata server in which the metadb_server_list directive is inserted. This is because some gfarm client commands like gfmdhost may be invoked on slave metadata servers as well.

2.3 gfsd and client

See SETUP document for further configuration.

If the gfsd configuration is already set up, restart gfsd by the root user.

on each gfsd host:

# /etc/init.d/gfsd restart

Now, master-slave metadata servers are set up.

2.4 Confirmation

$ gfmdhost -l + master - m (default) host-a 601 + slave sync c (default) host-b 601

gfjournal displays the metadata journal. Check whether the metadata updates are successfully transferred to the slave metadata server.

# gfjournal /var/gfarm-metadata/journal/0000000000.gmj records seqnum(min/max) record length(min/max/ave) 11 63/ 73 24/ 118/ 43

Update the metadata, for example, by 'gfmkdir tmp'. If 'max' of 'seqnum' on the master metadata server is same as it on the slave metadata server, the updates are transferred.

3. Upgrade a slave gfmd to a master gfmd

Confirm that the master gfmd has been stopped. If it is still running, stop it by the root user on host-a.

host-a:

# /etc/init.d/gfmd stop

Note that one of the following conditions should be met when a slave gfmd is upgraded to a master gfmd.

• The current master gfmd has been stopped.
• The current master gfmd could communicate to neither any slave gfmd, any gfsd, and any client.

To upgrade, send USR1 signal to the slave gfmd on host-b.

host-b:

# kill -USR1 <process id of gfmd>

By the way, there is a gfarm_zabbix package which has a feature to automatically upgrade a slave gfmd to a master gfmd, when a fault is detected on a master gfmd.

4. Restore the original master gfmd

4.1. restarting gfmd as a slave

# /etc/init.d/gfmd slavestart

4.2. Confirmation

$ gfmdhost -l + slave sync c (default) host-a 601 + master - m (default) host-b 601

If "+" is displayed, the gfmd correctly starts as a slave.

If "x" is displayed, please proceed to "4.3. Manual synchronization".

4.3. Manual synchronization

# gfdump.postgresql -d -f gfarm-pgsql.dmp

restore the metadata by root user on host-a:

# gfdump.postgresql -r -f gfarm-pgsql.dmp

Then, try again from "4.2 Confirmation".