documents >
User's Manual >
Master-Slave Metadata Servers - Tutorial
Master-Slave Metadata Servers - Tutorial
English | Japanese
This document describes how to configure master-slave metadata servers.
1. Installation
Let host-a be a master gfmd and host-b be a slave gfmd.
Install the Gfarm file system on both host-a and host-b.
See INSTALL document for details.
2. Configuration
2.1 Master metadata server
Create a user '_gfarmmd' which is used as the privilege for inter-gfmd
communication.
# useradd -c "Gfarm gfmd" -m _gfarmmd
|
Configure the master metadata server by the root user.
Add extra -r option to the the config-gfarm command which is described
in the SETUP.en docuemnt to enable replicated metadata servers.
The metadata journal file is stored in /var/gfarm-metadata/journal
directory by default. The location can be changed by the -j option.
If there is an existing metadata server configuration, use
config-gfarm-update with the -r option to update the configuration
after stopping the gfmd. For config-gfarm-update, the backend
PostgreSQL should be running.
install newer version of Gfarm
# /etc/init.d/gfmd stop
# config-gfarm-update -r --update
# /etc/init.d/gfmd start
|
For now, the PostgreSQL backend database is only supported for the
replicated metadata servers.
Register host-b as a slave metadata server.
The gfmdhost command has to be invoked by a Gfarm system administrator,
who has "gfarmadm" group privilege.
Default setting of Gfarm system administrator is
the user who is specified by the -A option of the config-gfarm command.
$ gfmdhost -c host-b
$ gfmdhost -l
+ master - m (default) host-a 601
- slave sync c (default) host-b 601
|
Insert a metadb_server_list directive to list all metadata hostnames and
ports in gfarm2.conf.
metadb_server_list host-a:601 host-b:601
|
Dump the metadata by the root user.
If you specified --prefix or --l option to the config-gfarm command,
you have to specify same options to the gfdump.postgresql command too.
# gfdump.postgresql -d -f gfarm-pgsql.dmp
|
Create the shared secret key for a user '_gfarmmd' which is used
for inter-gfmd communication.
# su _gfarmmd
$ gfkey -f -p 31536000
|
Note that the inter-gfmd communication only supports the shared secret
authentication for now.
2.2 Slave metadata server
Create a user '_gfarmmd' and copy the shared secret key from the
master metadata server for inter-gfmd communication.
# useradd -c "Gfarm gfmd" -m _gfarmmd
copy the shared secret key (~_gfarmmd/.gfarm_shared_key) from host-a
|
Configure a slave metadata server by the root user.
To enable replicated metadata servers, execute config-gfarm with the
-r option.
Restore the metadata by the root user.
If you specified --prefix or --l option to the config-gfarm command,
you have to specify same options to the gfdump.postgresql command too.
# gfdump.postgresql -r -f gfarm-pgsql.dmp
|
NOTE:
Currently, you cannot do "config-gfarm -r" on multiple slave metadata servers
simultaneously, even if you'd like to configure multiple slave gfmd at once.
Instead, you have to do "config-gfarm -r" and then "gfdump.postgresql -r" on
each slave server, and then you can proceed to next server.
Copy gfarm2.conf from the master metadata server in which the
metadb_server_list directive is inserted.
This is because some gfarm client commands like gfmdhost may be
invoked on slave metadata servers as well.
2.3 gfsd and client
Copy gfarm2.conf from the master metadata server in which the
metadb_server_list directive is inserted.
See SETUP document for further configuration.
If the gfsd configuration is already set up, restart gfsd by the root
user.
on each gfsd host:
# /etc/init.d/gfsd restart
|
Now, master-slave metadata servers are set up.
2.4 Confirmation
gfmdhost -l displays status of metadata servers. '+' means the slave
metadata server is connected to the master metadata server.
$ gfmdhost -l
+ master - m (default) host-a 601
+ slave sync c (default) host-b 601
|
gfjournal displays the metadata journal. Check whether the metadata
updates are successfully transferred to the slave metadata server.
# gfjournal /var/gfarm-metadata/journal/0000000000.gmj
records seqnum(min/max) record length(min/max/ave)
11 63/ 73 24/ 118/ 43
|
Update the metadata, for example, by 'gfmkdir tmp'. If 'max' of
'seqnum' on the master metadata server is same as it on the slave
metadata server, the updates are transferred.
3. Upgrade a slave gfmd to a master gfmd
This section explains how to upgrade a slave gfmd to a master gfmd
when the master gfmd fails.
Confirm that the master gfmd has been stopped. If it is still
running, stop it by the root user on host-a.
host-a:
Note that one of the following conditions should be met when a slave
gfmd is upgraded to a master gfmd.
- The current master gfmd has been stopped.
- The current master gfmd could communicate to neither any slave
gfmd, any gfsd, and any client.
If one of above conditions is not met, there is a possibility of
inconsistency among the master gfmd and slave gfmds.
To upgrade, send USR1 signal to the slave gfmd on host-b.
host-b:
# kill -USR1 <process id of gfmd>
|
By the way, there is a gfarm_zabbix package which has a feature
to automatically upgrade a slave gfmd to a master gfmd, when
a fault is detected on a master gfmd.
4. Restore the original master gfmd
This section describe how to restart old master gfmd
as a new slave gfmd.
4.1. restarting gfmd as a slave
Invoke the following command as root user to do so:
host-a:
# /etc/init.d/gfmd slavestart
|
4.2. Confirmation
Check whether the gfmd correctly starts as a slave or not
by the gfmdhost -l command:
$ gfmdhost -l
+ slave sync c (default) host-a 601
+ master - m (default) host-b 601
|
If "+" is displayed, the gfmd correctly starts
as a slave.
If "x" is displayed, please proceed to
"4.3. Manual synchronization".
4.3. Manual synchronization
If "x" is displayed, it means automatic synchronization
is impossible, because the metadata changes while host-a was stopping
were too much.
In that case, you have to copy the database on host-b to host-a
by the following procedure:
dump metadata by root user on host-b:
# gfdump.postgresql -d -f gfarm-pgsql.dmp
|
restore the metadata by root user on host-a:
# gfdump.postgresql -r -f gfarm-pgsql.dmp
|
Then, try again from "4.2 Confirmation".
Gfarm File System <gfarmfs at gmail.com>