Professional Documents
Culture Documents
Pascal Hofmann: Mysql Master-Master Replication Manager
Pascal Hofmann: Mysql Master-Master Replication Manager
Pascal Hofmann
Table of Contents
1 2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Typical use-cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.1 2.2 Two node setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Two masters + one/many slaves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.1 3.2 3.3 Requirements - monitoring host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Requirements - nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Additional requirements for MMM tools . . . . . . . . . . . . . . . . . . . . . . . . 5
4 5
ii 6.10 6.11 6.12 6.13 set_passive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . move_role role host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . move_role --force role host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . set_ip ip host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 13 13 13
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
7.1 7.2 Config files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuration syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.1 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.2 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.3 Specifying multiple values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.4 Sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.4.1 Unique sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.4.2 Named sections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.4.3 Default sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.5 Includes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Configuration variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 Global variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.2 monitor section . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.3 host sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.4 role sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.5 check sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.6 socket section . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.7 copy_method sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 14 14 14 14 14 14 15 15 15 15 15 16 17 20 21 21 22
Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
8.1 8.2 Logging config files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Log4perl sample configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Chapter 1: Overview
1 Overview
MMM (Master-Master Replication Manager for MySQL) is a set of flexible scripts to perform monitoring/failover and management of MySQL Master-Master replication configurations (with only one node writable at any time). The toolset also has the ability to read balance standard master/slave configurations with any number of slaves, so you can use it to move virtual IP addresses around a group of servers depending on whether they are behind in replication. In addition to that, it also has scripts for data backups, resynchronization between nodes etc. The main functionality is provided through the following three scripts: mmm_mond mmm_agentd agent daemon which runs on each MySQL server and provides monitoring node with simple set of remote services. mmm_control simple script dedicated to management of the mmm mond processes by commands. monitoring daemon which does all monitoring work and makes all decisions about roles moving and so on.
2 Typical use-cases
Typical use cases of MMM will be described in the following two sections.
In two node master-master setup, MMM uses five IPs: single permanent IP for each node that is never changed, 2 reader IPs (read-only) and 1 writer IP (updates). Last three IPs are migrating between the nodes depending on node availability.
Normally (no replication failures, no replication delay etc) active master has 2 IPs (reader and writer), standby master - 1 IP (reader). In case of a failure, both - writer and reader roles migrate to the working node.
Chapter 3: Requirements
3 Requirements
For an MMM setup with n MySQL servers, youll need n + 1 hosts One host for each MySQL server; one host for the MMM monitor. 2 * (n + 1) IPs One IP for each host (see above); one IP for the writer role; n IPs for one reader role per host. monitor user A MySQL user with privileges REPLICATION CLIENT for MMM monitor. agent user A MySQL user with privileges SUPER, REPLICATION CLIENT, PROCESS for MMM agent. replication user A MySQL user with privileges REPLICATION SLAVE used for replication. tools user A MySQL user with privileges SUPER, REPLICATION CLIENT, RELOAD for MMM tools.
Chapter 3: Requirements
Only hosts with state ONLINE may have roles. When a host switches from ONLINE to any other state, all roles will be removed from it. A host that was in state REPLICATION_DELAY or REPLICATION_FAIL will be switched back to ONLINE if everything is OK again, unless it is flapping (see Section 5.6 [Flapping], page 8). A host that was in state HARD_OFFLINE will be switched to AWAITING_RECOVERY if everything is OK again. If its downtime was shorter than 60 seconds and it wasnt rebooted or auto set online is > 0 it will be switched back to ONLINE automatically, unless it is flapping (see Section 5.6 [Flapping], page 8 again). Replication backlog or failure on the active master isnt considered to be a problem, so the active master will never be in state REPLICATION_DELAY or REPLICATION_FAIL. Replication backlog or failure will be ignored on hosts whos peers got ONLINE less than 60 seconds ago (Thats the default value of master-connect-retry). If both checks rep_backlog and rep_threads fail, the state will change to REPLICATION_FAIL.
5.2 Roles
There are two types of roles: exclusive roles Exclusive roles may only have one ip and are assigned to one host at a time. You may specify a preferred host, to which the role will always be moved, if its online. Note: You cant move roles which are assigned to the preferred host, because they would be moved back to it immediately. balanced roles Balanced roles may have multiple ips. The ips will be balanced between the hosts, so that no host has two roles more than another host.
TODO describe active master role allow writes change master of all hosts with mode=slave to this host
5.4 Checks
mmm mond performs four different checks on each host to determine if it is OK. These checks are: ping host is pingable mysql MySQL server on the host is alive rep threads replication threads are running replication backlog is not too big rep backlog
5.6 Flapping
mmm mond supports the detection of hosts that are "flapping". Flapping occurs if a host which is ONLINE changes its state to HARD_OFFLINE / REPLICATION_FAIL / REPLICATION_ DELAY too often and each time gets switched back to ONLINE (because of auto set online or because it has been down for less than 60 seconds). This may lead to roles getting switched between hosts very often. To prevent this mmm mond has a built in flap-detection which can be tuned in the configuration file. If a host goes down for more than flap count times within flap duration seconds it is considered as flapping and will not be set ONLINE automatically. It will stay in state AWAITING_RECOVERY until it gets set online (with mmm_control set_online host ). If auto set online is > 0, flapping hosts will automatically be set to ONLINE after flap duration seconds.
5.7 Modes
5.7.1 Active mode
The monitor will remove roles from failed hosts and assign them to other hosts automatically.
5.8 Startup
Initial network check If network is down startup will be delayed until its reachable again. Initial host checks reads status information from ... status file agents (agent info) hosts (system info) and tries to figure out the cluster status.
10
2. Change master to the new master Writer role is added to the new master: 1. MySQL is made writable 2. IP is configured
11
6.1 help
Show help message.
6.2 ping
Ping the monitor deamon to check if its still alive. # mmm_control ping OK: Pinged successfully! When the monitor is down: # mmm_control ping ERROR: Cant connect to monitor daemon!
6.3 show
Show the current cluster status. See Section 5.1 [States], page 7 for an explanation of the different node states.
# mmm_control show db1(192.168.0.31) master/ONLINE. Roles: writer(192.168.0.50), reader(192.168.0.51) db2(192.168.0.32) master/AWAITING_RECOVERY. Roles: db3(192.168.0.33) slave/ONLINE. Roles: reader(192.168.0.52), reader(192.168.0.53)
# mmm_control checks mmm-master1 mmm-master1 ping [last change: mmm-master1 mysql [last change: mmm-master1 rep_threads [last change: mmm-master1 rep_backlog [last change:
# mmm_control checks mmm-master1 rep_backlog mmm-master1 rep_backlog [last change: 2010/02/11 12:43:16] # mmm_control checks all rep_backlog
12
mmm-master1 mmm-master2
rep_backlog rep_backlog
All roles have been moved to hosts db1 and db3 as db2 has failed. Now that its recovered, we should set it online:
# mmm_control set_online db2 OK: State of db2 changed to ONLINE. Now you can wait some time and check its new roles! # mmm_control show db1(192.168.0.31) master/ONLINE. Roles: writer(192.168.0.50), reader(192.168.0.51) db2(192.168.0.32) master/ONLINE. Roles: reader(192.168.0.52) db3(192.168.0.33) slave/ONLINE. Roles: reader(192.168.0.53)
set_online is used to bring the node back online again. See Section 6.5 [set online], page 12.
6.7 mode
Print current mode. In the following example, the monitor is running in ACTIVE mode: # mmm_control mode ACTIVE Here the monitor is in PASSIVE mode: # mmm_control mode PASSIVE See Section 5.7 [Modes], page 8.
13
6.8 set_active
Switch the monitor into ACTIVE mode: # mmm_control set_active OK: Switched into active mode. See Section 5.7 [Modes], page 8.
6.9 set_manual
Switch the monitor into MANUAL mode: # mmm_control set_manual OK: Switched into manual mode. See Section 5.7 [Modes], page 8.
6.10 set_passive
Switch the monitor into PASSIVE mode: # mmm_control set_passive OK: Switched into passive mode. See Section 5.7 [Modes], page 8.
# mmm_control move_role writer db1 OK: Role writer has been moved from db2 to db1. Now you can wait some time and check new roles info # mmm_control show db1(192.168.0.31) master/ONLINE. Roles: writer(192.168.0.50), reader(192.168.0.51) db2(192.168.0.32) master/ONLINE. Roles: reader(192.168.0.52) db3(192.168.0.33) slave/ONLINE. Roles: reader(192.168.0.53)
Chapter 7: Configuration
14
7 Configuration
7.1 Config files
Config files may resist in /etc, /etc/mmm or /etc/mysql-mmm. /etc/mysql-mmm should be preferred. Program mmm agentd mmm mond mmm control mmm backup mmm clone mmm restore Filename mmm_agent.conf mmm_mon.conf or mmm_mon_CLUSTER.conf mmm_mon.conf or mmm_mon_CLUSTER.conf mmm_tools.conf mmm_tools.conf mmm_tools.conf
7.2.2 Variables
asdsadasd debug 0
7.2.4 Sections
The configuration is divided into two kinds of sections: unique and named.
Chapter 7: Configuration
15
7.2.5 Includes
It is possible to split up the configuration into several files and include them via include. include common.conf
Chapter 7: Configuration
16
default copy method Description: Default value: Used by: clone dirs Description: Default value: Used by: path(s) containing mysql data/logs (relative to mount point of logical volume) tools name of the default copy method tools
Chapter 7: Configuration
17
Default value: Used by: flap count Description: Default value: Used by: auto set online Description: Default value: Used by: kill host bin Description:
3600 monitor Maximum number of downtimes within flap_duration seconds after which a host is considered to be flapping. 3 monitor How many seconds to wait before switching node status from AWAITING_RECOVERY to ONLINE. 0 = disabled. 0 monitor Binary used to kill hosts if roles couldnt be removed because the agent was not reachable. You have to provide a custom binary for this which takes the hostname as first argument and the state of check ping (1 ok; 0 - not ok) as second argument. bin path/monitor/kill host monitor Startup carefully i.e. switch into passive mode when writer role is configured on multiple hosts. true/yes/1/on false/no/0/off 0 monitor
Allowed values: Default value: Used by: mode Description: Default mode of monitor. Allowed values: active manual wait passive Default value: active Used by: monitor wait for other master Description: How many seconds to wait for other master to become ONLINE before switching from mode WAIT to mode ACTIVE. 0 = infinite. Default value: 120 Used by: monitor
Chapter 7: Configuration
18
Default value: Used by: peer Description: Default value: Used by: pid path Description: Default value: Used by: bin path Description: Default value: Used by: agent port Description: Default value: Used by: cluster interface Description: Default value: Used by: mysql port Description: Default value: Used by: mysql pidfile Description: Default value: Used by: mysql rcscript Description: Default value: Used by: mysql cnf Description: Default value: Used by: agent user Description: Default value: Used by:
agent, monitor Name of peer host (if mode is master) agent, monitor Location of pid-file agent Path to directory containing MMM binaries agent Port on which mmm agentd listens 9989 agent, monitor network interface on which the IPs of the roles should be configured agent Port on which mysqld is listening 3306 agent, monitor, tools location of mysql pid-file /var/run/mysqld/mysqld.pid tools location of mysql rc-script /etc/init.d/mysql tools location of my.cnf /etc/my.cnf tools mysql user for MMM Agent agent
Chapter 7: Configuration
19
agent password Description: mysql password for MMM Agent Default value: Used by: agent monitor user Description: mysql user for MMM Monitor Default value: Used by: monitor monitor password Description: mysql password for MMM Monitor Default value: Used by: monitor replication user Description: mysql user used for replication Default value: Used by: agent, tools replication password Description: mysql password used for replication Default value: Used by: agent, tools ssh user Description: SSH user for MMM Tools Default value: Used by: tools ssh port Description: SSH port for MMM Tools Default value: 22 Used by: tools ssh parameters Description: additional SSH parameters for MMM Tools Default value: Used by: tools tools user Description: mysql user for MMM Tools Default value: Used by: tools tools password Description: mysql password for MMM Tools Default value: Used by: tools backup dir Description: Target directory for backups
Chapter 7: Configuration
20
Default value: Used by: restore dir Description: Default value: Used by: lvm bin lvcreate Description: Default value: Used by: lvm bin lvremove Description: Default value: Used by: lvm snapshot size Description: Default value: Used by: lvm logical volume Description: Default value: Used by: lvm volume group Description: Default value: Used by: lvm mount dir Description: Default value: Used by: lvm mount opts Description: Default value: Used by:
tools Directory where backups should be restored to tools Path to lvcreate binary lvcreate tools Path to lvremove binary lvremove tools Size of LVM snapshot tools Logical volume where mysql data and logs reside tools Volume group of logical volume with mysql data and logs tools Mount point for LVM snapshot tools Mount options used when mounting LVM snapshot tools
Chapter 7: Configuration
21
Used by: hosts Description: Default value: Used by: ips Description: Default value: Used by: prefer Description: Default value: Used by:
monitor Hosts which may take over the role monitor One or multiple IPs associated with the role agent, monitor The preferred host for this role. Only allowed for exclusive roles. monitor
Chapter 7: Configuration
22
cert file Description: Default value: Used by: key file Description: Default value: Used by: ca file Description: Default value: Used by:
location of SSL certificate (if type is ssl) agent, control, monitor location of RSA private key (if type is ssl) agent, control, monitor location of file containing certificate(s) of the reputable certificate authorities (if type is ssl) agent, monitor
Chapter 8: Logging
23
8 Logging
8.1 Logging config files
MMM uses Log4perl for logging, so the logging abilities are very flexible. If the default logging configuration doesnt fit your needs you can create a config file which may resist in /etc, /etc/mmm or /etc/mysql-mmm. /etc/mysql-mmm should be preferred. The name of the file depends on the program you want to create it for: Program mmm agentd mmm mond mmm control mmm backup mmm clone mmm restore Filename mmm_agent_log.conf mmm_mon_log.conf or mmm_mon_log_CLUSTER.conf mmm_mon_log.conf or mmm_mon_log_CLUSTER.conf -
It is also possible to log into multiple files (separated by severity) or let MMM send you mails (this was the default logging configuration in versions below 2.1.1):
log4perl.logger = INFO, FileInfo, FileWarn, FileError, FileFatal, MailFatal log4perl.appender.FileInfo = Log::Log4perl::Appender::File log4perl.appender.FileInfo.Threshold = INFO log4perl.appender.FileInfo.filename = /var/log/mysql-mmm/progam.info log4perl.appender.FileInfo.recreate = 1 log4perl.appender.FileInfo.layout = PatternLayout log4perl.appender.FileInfo.layout.ConversionPattern = %d %5p %m%n log4perl.appender.FileWarn = Log::Log4perl::Appender::File log4perl.appender.FileWarn.Threshold = WARN log4perl.appender.FileWarn.filename = /var/log/mysql-mmm/progam.warn
Chapter 8: Logging
24
log4perl.appender.FileWarn.recreate = 1 log4perl.appender.FileWarn.layout = PatternLayout log4perl.appender.FileWarn.layout.ConversionPattern = %d %5p %m%n log4perl.appender.FileError = Log::Log4perl::Appender::File log4perl.appender.FileError.Threshold = ERROR log4perl.appender.FileError.filename = /var/log/mysql-mmm/progam.error log4perl.appender.FileError.recreate = 1 log4perl.appender.FileError.layout = PatternLayout log4perl.appender.FileError.layout.ConversionPattern = %d %5p %m%n log4perl.appender.FileFatal = Log::Log4perl::Appender::File log4perl.appender.FileFatal.Threshold = FATAL log4perl.appender.FileFatal.filename = /var/log/mysql-mmm/progam.fatal log4perl.appender.FileFatal.recreate = 1 log4perl.appender.FileFatal.layout = PatternLayout log4perl.appender.FileFatal.layout.ConversionPattern = %d %5p %m%n log4perl.appender.MailFatal = Log::Dispatch::Email::MailSend log4perl.appender.MailFatal.Threshold = FATAL log4perl.appender.MailFatal.to = root log4perl.appender.MailFatal.buffered = 0 log4perl.appender.MailFatal.subject = FATAL error in progam log4perl.appender.MailFatal.layout = PatternLayout log4perl.appender.MailFatal.layout.ConversionPattern = %d %m%n
Index
25
Index
A
Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
N
Named sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Network check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
C
checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Config files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Configuration syntax . . . . . . . . . . . . . . . . . . . . . . . . . . 14
O
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
R
Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Role transition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
F
Flapping, flap detection . . . . . . . . . . . . . . . . . . . . . . . . . 8
S
Sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 set offline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 set online . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 show . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Startup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Status file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
H
help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
K
Kill host functionality . . . . . . . . . . . . . . . . . . . . . . . . . 10
L
Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Logging config files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
T
typical use-cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
M
mmm agentd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 mmm control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 mmm mond . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
U
Unique Sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
V
Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14