[GFARM LOGO] documents > User's Manual > Master-Slave Metadata Servers - Tutorial

Master-Slave Metadata Servers - Tutorial

English | Japanese

This document describes how to configure master-slave metadata servers.

1. Installation

Let host-a be a master gfmd and host-b be a slave gfmd. Install the Gfarm file system on both host-a and host-b. See INSTALL document for details.

2. Configuration

2.1 Master metadata server

Create a user '_gfarmmd' which is used as the privilege for inter-gfmd communication.

# useradd -c "Gfarm gfmd" -m _gfarmmd

Configure the master metadata server by the root user. Add extra -r option to the the config-gfarm command which is described in the SETUP.en docuemnt to enable replicated metadata servers.

# config-gfarm -r
The metadata journal file is stored in /var/gfarm-metadata/journal directory by default. The location can be changed by the -j option. If there is an existing metadata server configuration, use config-gfarm-update with the -r option to update the configuration after stopping the gfmd. For config-gfarm-update, the backend PostgreSQL should be running.

install newer version of Gfarm
# /etc/init.d/gfmd stop
# config-gfarm-update -r --update
# /etc/init.d/gfmd start
For now, the PostgreSQL backend database is only supported for the replicated metadata servers.

Register host-b as a slave metadata server.
The gfmdhost command has to be invoked by a Gfarm system administrator, who has "gfarmadm" group privilege. Default setting of Gfarm system administrator is the user who is specified by the -A option of the config-gfarm command.

$ gfmdhost -c host-b
$ gfmdhost -l
+ master -     m (default)    host-a 601
- slave  sync  c (default)    host-b 601
Insert a metadb_server_list directive to list all metadata hostnames and ports in gfarm2.conf.

metadb_server_list host-a:601 host-b:601
Dump the metadata by the root user.
If you specified --prefix or --l option to the config-gfarm command, you have to specify same options to the gfdump.postgresql command too.

# gfdump.postgresql -d -f gfarm-pgsql.dmp

Create the shared secret key for a user '_gfarmmd' which is used for inter-gfmd communication.

# su _gfarmmd
$ gfkey -f -p 31536000
Note that the inter-gfmd communication only supports the shared secret authentication for now.

2.2 Slave metadata server

Create a user '_gfarmmd' and copy the shared secret key from the master metadata server for inter-gfmd communication.

# useradd -c "Gfarm gfmd" -m _gfarmmd
copy the shared secret key (~_gfarmmd/.gfarm_shared_key) from host-a
Configure a slave metadata server by the root user. To enable replicated metadata servers, execute config-gfarm with the -r option.

# config-gfarm -r
Restore the metadata by the root user.
If you specified --prefix or --l option to the config-gfarm command, you have to specify same options to the gfdump.postgresql command too.

# gfdump.postgresql -r -f gfarm-pgsql.dmp

NOTE: Currently, you cannot do "config-gfarm -r" on multiple slave metadata servers simultaneously, even if you'd like to configure multiple slave gfmd at once. Instead, you have to do "config-gfarm -r" and then "gfdump.postgresql -r" on each slave server, and then you can proceed to next server. Copy gfarm2.conf from the master metadata server in which the metadb_server_list directive is inserted. This is because some gfarm client commands like gfmdhost may be invoked on slave metadata servers as well.

2.3 gfsd and client

Copy gfarm2.conf from the master metadata server in which the metadb_server_list directive is inserted.

See SETUP document for further configuration.

If the gfsd configuration is already set up, restart gfsd by the root user.

on each gfsd host:
# /etc/init.d/gfsd restart

Now, master-slave metadata servers are set up.

2.4 Confirmation

gfmdhost -l displays status of metadata servers. '+' means the slave metadata server is connected to the master metadata server.

$ gfmdhost -l
+ master -     m (default)    host-a 601
+ slave  sync  c (default)    host-b 601

gfjournal displays the metadata journal. Check whether the metadata updates are successfully transferred to the slave metadata server.

# gfjournal /var/gfarm-metadata/journal/0000000000.gmj
records  seqnum(min/max)            record length(min/max/ave)
     11            63/          73          24/    118/     43

Update the metadata, for example, by 'gfmkdir tmp'. If 'max' of 'seqnum' on the master metadata server is same as it on the slave metadata server, the updates are transferred.

3. Upgrade a slave gfmd to a master gfmd

This section explains how to upgrade a slave gfmd to a master gfmd when the master gfmd fails.

Confirm that the master gfmd has been stopped. If it is still running, stop it by the root user on host-a.

host-a:
# /etc/init.d/gfmd stop

Note that one of the following conditions should be met when a slave gfmd is upgraded to a master gfmd.

If one of above conditions is not met, there is a possibility of inconsistency among the master gfmd and slave gfmds.

To upgrade, send USR1 signal to the slave gfmd on host-b.

host-b:
# kill -USR1 <process id of gfmd>
By the way, there is a gfarm_zabbix package which has a feature to automatically upgrade a slave gfmd to a master gfmd, when a fault is detected on a master gfmd.

4. Restore the original master gfmd

This section describe how to restart old master gfmd as a new slave gfmd.

4.1. restarting gfmd as a slave

Invoke the following command as root user to do so:

host-a:
# /etc/init.d/gfmd slavestart

4.2. Confirmation

Check whether the gfmd correctly starts as a slave or not by the gfmdhost -l command:

$ gfmdhost -l
+ slave  sync  c (default)    host-a 601
+ master -     m (default)    host-b 601

If "+" is displayed, the gfmd correctly starts as a slave.

If "x" is displayed, please proceed to "4.3. Manual synchronization".

4.3. Manual synchronization

If "x" is displayed, it means automatic synchronization is impossible, because the metadata changes while host-a was stopping were too much.
In that case, you have to copy the database on host-b to host-a by the following procedure:

dump metadata by root user on host-b:

# gfdump.postgresql -d -f gfarm-pgsql.dmp


restore the metadata by root user on host-a:

# gfdump.postgresql -r -f gfarm-pgsql.dmp


Then, try again from "4.2 Confirmation".


Gfarm File System <gfarmfs at gmail.com>