This document serves as the starting point to plan, execute, and verify your hardware setup for a High Availability (HA) environment.
This document provides some advice on the initial planning, the installation and cabling, and the test and verification of the overall setup. We use the word takeover to mean transferring some kind of server functionality from a broken entity to a sane one. In this context, entities can be network adapters, computers, or something else. Our current focus is to provide HA capabilities among PC's which we will call "nodes" from now on.
These two nodes have to be connected in some way to exchange status information and to monitor each other. The more channels our nodes have to talk to each other, the better it is. We will use the term "medium" for such a communication channel.
In general we work from the assumption that we use standard hardware where ever possible. This means that we do not modify our PC's other than to expand them with components off the shelf. And we use only cabling that can be bought without "special orders" such as split serial cables or the like. After all we want solutions that can be installed and used by everyone, not just some experts.
So how will the hardware be planned? Well, straight forward.
+-------------------+ +-------------------+
| | | |
| Node A | | Node B |
| | | |
| eth0 | | eth0 |
+---------+---------+ +---------+---------+
| |
| |
|---------+----------------------+---------| LAN (Ethernet, etc.)
As was mentioned before, this design obviously provides
insufficiently reliable monitoring capabilities.
In a LAN, there are many different points of failure.
Another issue is that the LAN is a public medium and that there are several
levels of possible failures.
So we would be well-advised to look for more robust options to
use in addition to the LAN for heartbeats.
The main idea is to have a simple private medium like one or more serial cables. We can use the standard serial ports, provided they are not already occupied by modems, mice, or other vermin. If you have a server with a PS/2 mouse, it probably has two such ports available.
So here's what this configuration looks like.
+----------------------+ (Nullmodem Cable)
| |
+---------+---------+ +---------+---------+
| ttyS0 | | ttyS0 |
| | | |
| Node A | | Node B |
| | | |
| eth0 | | eth0 |
+---------+---------+ +---------+---------+
| |
| |
|---------+----------------------+---------| LAN (Ethernet, etc.)
What do we gain? We have now two media to exchange the heartbeat.
This provides greater reliability in the case of failure.
Of course
the restriction with the LAN still holds true, but now Node A could use
the serial line to initiate a takeback of the service. And if just the
serial connection should fail, we still have the LAN.
Reliable intracluster communications is very important, and this design
is a low-cost improvement over the previous one.
The serial lines are now arranged in a ring structure. As you will have noticed, this occupies two serial ports on each node as per our discussion in the previous chapter. But on the other hand we do now have a general setup that can easily be extended and also provides a good level of redundancy. We can now send our heartbeat now in both directions over the ring, thus reaching every other node even in case of a (single) cable defect (or down system) anywhere on the ring.
Another facet of our high end design covers the LAN access. Having two adapters connected to the wire allows us to provide intra-node failover capabilities in case of an interface or LAN cable breakdown. Plus it gives us the chance to take over the IP address of Node A, eth0 onto Node B, eth1 and keeping Node B, eth0 as it is. In fact, this is the primary operation mode of several professional systems, including IBM's HACMP or HP's MC/ServiceGuard. Which doesn't imply that we are not professional, of course :-)
So, here is the block diagram for this third design.
(Nullmodem Cables)
+-----------------------------------------------------+
| +--------------+ +-------------+ |
| | | | | |
+-----+-------+-----+ +-----+--------+----+ +-----+-------+-----+
| ttyS0 ttyS1 | | ttyS0 ttyS1 | | ttyS0 ttyS1 |
| | | | | |
| Node A | | Node B | | Node C |
| | | | | |
| eth0 eth1 | | eth0 eth1 | | eth0 eth1 |
+-----+--------+----+ +-----+--------+----+ +-----+--------+----+
| | | | | |
| | | | | |
|-----+--------+-------------+--------+-------------+--------+----|
LAN (Ethernet, etc.)
Future releases of the heartbeat software will support such a
configuration, but current takeover code restricts the configuration
to a single interface and two nodes in the network.
Of course we could also use other media for the heartbeat exchange. Recent
suggestions include SCSI buses in target mode and IrDA ports "connected"
with a mirror. Another candidate that comes to mind is the USB found in
many modern PC's. As I said before, the more (and more different) the better.
25-pin 9-pin 9-pin 25-pin
2 TxD 3 -------------------------- 2 RxD 3
3 RxD 2 -------------------------- 3 TxD 2
4 RTS 7 -------------------------- 8 CTS 5
5 CTS 8 -------------------------- 7 RTS 4
7 GND 5 -------------------------- 5 GND 7
6 DSR 6 ---+---------------------- 4 DTR 20
8 DCD 1 ---+ +--- 1 DCD 8
20 DTR 4 ----------------------+--- 6 DSR 6
Once you have these cable(s) in place you will want to test them. This
is fairly easy since the serial ports are usually configured with decent
default values. On a freshly booted Linux system we can assume the ports
to be in a "sane" state, with the speed set to 9600 baud. If not, you can
do a "stty sane 9600 </dev/ttyS0"
with ttyS0 replaced by the actual device. Please note the input
redirection which selects the device.
Then you can set up one node as receiver ("cat </dev/ttyS0") and the other one as transmitter ("echo hello >/dev/ttyS0"). Voila! What you expect is that the "hello" is printed out at the receiver. Pressing Ctrl-C on the receiver's keyboard will return you to the prompt. Then do the same test with mutually exchanged roles.
If you are planning to use more than one adapter per node (usually called "Standby Adapters"), please make sure to connect them to the same physical medium as the primary adapters. Otherwise you will of course not be able to takeover the IP address. Having them in a different subnet is perfectly okay. More than that: it's preferred. [TODO: this is what I learned with HACMP. Can anyone please give the *correct* rationale --- or rephrase the whole paragraph?]
Note: The heartbeat software does not yet support this kind of configuration.
1: uart:16550A port:2F8 irq:3 baud:19200 tx:24423 rx:4680 RTS|CTS|DTR|DSR|CDThis particular line corresponds to a working "raw" serial port, /dev/ttyS1 with both sides cabled up correctly, and heartbeat active on both sides. The first number on the line is the port number. The built-in serial ports on PCs are numbered 0 and 1. With heartbeat only active on the local side (and not the far side), it looks like this instead:
1: uart:16550A port:2F8 irq:3 baud:19200 tx:43558 rx:12277 RTS|DTRNote the lack of the CTS (Clear To Send), DSR (Data Set Ready), and CD (Carrier Detect) bits on the interface. When heartbeat is only running on the far side interface, it looks like this:
1: uart:16550A port:2F8 irq:3 baud:19200 tx:55039 rx:12277 CTS|DSR|CDNote that when the local port isn't active, the RTS (Request To Send), and DTR (Data Terminal Ready) bits aren't active. When heartbeat isn't running on either interface, the line looks like this:
1: uart:16550A port:2F8 irq:3 baud:19200 tx:55039 rx:12277This is essentially a software breakout box.
Harald Milz' Linux-HA HOWTO that started the whole thing can be found at: http://metalab.unc.edu/pub/Linux/ALPHA/linux-ha/High-Availability-HOWTO.html
A comprehensive survey on professional HA solutions is here:
http://www.sun.com/clusters/dh.brown.pdf
[TODO: should we include links to HACMP, Veritas, Wizard, ... ???]
$Id: HardwareGuide.html,v 1.3 2001/03/08 14:34:55 alan Exp $