Introduction
A common configuration of failover in ISC DHCP was the "hub and spoke" model, where there was a central (hub) server that had a failover relationship with several satellite servers (sometimes called branch or spokes) at (possibly) geographically disparate locations. Kea, until 2.5.5, did not support such a configuration. This article is an exploration of what is possible with the "hub and spoke" model and is a result of testing each relationship represented here. It should be noted that the case where a Kea server is both a hub and a spoke has not been tested by the ISC QA team. There is other documentation, such as a design document describing the model. And, of course, there is a description and configuration explanation in the ARM.
A description of the test
There were a couple questions that we in support wanted to answer surrounding this model. The intent here is to explore what is possible so that some creative administrator might, using these results, come up with a solution that works for his use case.
- Can multiple servers participate with a single backup server?
- Answer: Yes
- Is it possible for one of the satellite servers to participate in multiple relationships?
- Answer: Yes
- Can a satellite server also be a hub in the case of another HA relationship?
- Answer: Yes
- Is the hub server required to be the standby server?
- Answer: No
- Is load-balancing supported?
- Answer: No
To find the answers that have already been given away above, five servers were configured as follows (the actual Kea configuration used on each server is available at the end of this document).
Consider this diagram:
There are five white boxes of various shapes and sizes there. Each is labeled with a server name. Additionally, each relationship is color coded with color coded arrows representing the connection between each server and, essentially, their subnets. The color coded boxes each contain a name (e.g., "server1"). These names were used in the HA parameters requiring a name. The dotted lines represent the backup server relationships. The tables below describe each HA relationship between the subnets configured in each server.
| HA Server Name | Subnet | Server | Primary | Standby | Backup |
|---|---|---|---|---|---|
| server1 | 172.28.0.0/24 | Kea 2 | ✓ | ||
| server2 | 172.28.0.0/24 | Kea 1 | ✓ | ||
| server7 | 172.28.0.0/24 | Kea Backup Server | ✓ |
| HA Server Name | Subnet | Server | Primary | Standby | Backup |
|---|---|---|---|---|---|
| server3 | 172.29.0.0/24 | Kea 3 | ✓ | ||
| server4 | 172.29.0.0/24 | Kea 1 | ✓ | ||
| server8 | 172.29.0.0/24 | Kea Backup Server | ✓ |
| HA Server Name | Subnet | Server | Primary | Standby | Backup |
|---|---|---|---|---|---|
| server5 | 172.30.0.0/24 | Kea 3 | ✓ | ||
| server6 | 172.30.0.0/24 | Kea 4 | ✓ | ||
| server9 | 172.30.0.0/24 | Kea Backup Server | ✓ |
Note that "Kea 3" appears in two subnets above. Currently, it is listed as "Primary" for 172.29.0.0/24 but "Standby" for 172.30.0.0/24. During testing, we flipped these roles to test both open questions.
Using perfdhcp, we were able to test that all of these relationships worked. These were not performance tested. It should also be noted that, though we did test shutting down various Kea daemons on each server for failover purposes, perfdhcp is not a real client. An administrator that wanted to use one or more of these novel relationships should test with real clients, and under load, in both normal and failover scenarios, before deployment.
There were multiple copies of perfdhcp executed from a couple of different servers that were specific to that purpose. Each of these servers had multiple addresses on enp0s3 and enp0s8 so that multiple copies could be run. A typical command line for these perfdhcp runs might look like this:
sudo perfdhcp -4 -r 1 -R 59 -l 192.168.20.251 -p 600 -Y 1 -y 3600 192.168.20.220
See the perfdhcp manual page in the ARM for further information about the parameters passed.
To check the status of each server, the API call status-get was used. This API call is documented in the ARM. The command line looked the same on all servers:
echo '{"command":"status-get" }' | sudo socat UNIX:/tmp/kea-dhcp4-socket -,ignoreeof | jq
The output was similar to this
Note that there are multiple elements in the high-availability list returned in the results. There is one per relationship. In this way, it is possible to individually track the status of each relationship.
{
"arguments": {
"high-availability": [
{
"ha-mode": "hot-standby",
"ha-servers": {
"local": {
"role": "standby",
"scopes": [],
"server-name": "server2",
"state": "hot-standby"
},
"remote": {
"age": 2,
"analyzed-packets": 0,
"communication-interrupted": false,
"connecting-clients": 0,
"in-touch": true,
"last-scopes": [
"server1"
],
"last-state": "hot-standby",
"role": "primary",
"server-name": "server1",
"unacked-clients": 0,
"unacked-clients-left": 0
}
}
},
{
"ha-mode": "hot-standby",
"ha-servers": {
"local": {
"role": "standby",
"scopes": [],
"server-name": "server4",
"state": "hot-standby"
},
"remote": {
"age": 2,
"analyzed-packets": 0,
"communication-interrupted": false,
"connecting-clients": 0,
"in-touch": true,
"last-scopes": [
"server3"
],
"last-state": "hot-standby",
"role": "primary",
"server-name": "server3",
"unacked-clients": 0,
"unacked-clients-left": 0
}
}
}
],
"multi-threading-enabled": true,
"packet-queue-size": 28,
"packet-queue-statistics": [
0.998886,
0.493383,
0.0657395
],
"pid": 2661,
"reload": 91,
"sockets": {
"status": "ready"
},
"thread-pool-size": 4,
"uptime": 91
},
"result": 0
}
The configurations used follow in the collapsed sections as they are quite large. But first: a couple of notes about the configurations and the test in general.
- Some of the servers below feature multiple interfaces in use during the testing. This is not strictly necessary. Some of these were pre-existing VMs used during other testing. Multiple addresses were needed for the
"relay"parameter to select the proper subnet during the test which made it convenient to use the pre-existing interfaces. - The VMs running
perfdhcpwere separate from any of the Kea servers. One could contain all of theperfdhcpon a single VM utilizing different addresses configured there. This could even be done on the "Kea backup server" VM if so desired. - All of these VMs were running on the same host.
Kea 1 configuration
This server used a single Ethernet port during the testing. perfdhcp instances used enp0s8 or one of the following IP addresses 172.28.0.250, 172.28.0.251, 172.28.0.249, or 172.28.0.252 in the -l argument to select the subnet to be tested below.
Kea 1 configuration
{
"Dhcp4": {
"control-socket": {
"socket-type": "unix",
"socket-name": "/tmp/kea-dhcp4-socket"
},
"interfaces-config": {
"interfaces": [
"enp0s8"
]
},
"lease-database": {
"type": "memfile",
"persist": true,
"name": "/tmp/kea-dhcp4-leases"
},
"multi-threading": {
"enable-multi-threading": true,
"thread-pool-size": 4,
"packet-queue-size": 28
},
"cache-threshold": 0.25,
"calculate-tee-times": true,
"decline-probation-period": 15,
"hooks-libraries": [
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [
{
"this-server-name": "server2",
"mode": "hot-standby",
"peers": [
{
"name": "server1",
"url": "http://172.28.0.254:8000/",
"role": "primary"
},
{
"name": "server2",
"url": "http://172.28.0.253:8000/",
"role": "standby"
},
{
"name": "server7",
"url": "http://192.168.20.219:8000/",
"role": "backup"
}
]
},
{
"this-server-name": "server4",
"mode": "hot-standby",
"peers": [
{
"name": "server3",
"url": "http://172.28.0.220:8001/",
"role": "primary"
},
{
"name": "server4",
"url": "http://172.28.0.253:8001/",
"role": "standby"
},
{
"name": "server8",
"url": "http://192.168.20.219:8001/",
"role": "backup"
}
]
}
]
}
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so"
}
],
"subnet4": [
{
"subnet": "172.28.0.0/24",
"id": 1,
"option-data": [
{
"name": "routers",
"data": "172.28.0.1"
}
],
"pools": [
{
"pool": "172.28.0.2-172.28.0.219"
}
],
"user-context": {
"ha-server-name": "server2"
},
"interface": "enp0s8"
},
{
"subnet": "172.29.0.0/24",
"id": 2,
"option-data": [
{
"name": "routers",
"data": "172.29.0.1"
}
],
"pools": [
{
"pool": "172.29.0.2-172.29.0.254"
}
],
"user-context": {
"ha-server-name": "server4"
},
"relay": { "ip-addresses": [ "172.28.0.250","172.28.0.251","172.28.0.249","172.28.0.252" ] }
}
],
"loggers": [
{
"name": "kea-dhcp4",
"severity": "INFO",
"output_options": [
{
"output": "stdout"
}
]
}
]
}
}
Kea 2 configuration
This server also used only a single Ethernet port for the test. This one had only one relationship in which to participate. This was marked as a "local" subnet and so perfdhcp -l enp0s8 was used during the testing.
Kea 2 configuration
{
"Dhcp4": {
"control-socket": {
"socket-type": "unix",
"socket-name": "/tmp/kea-dhcp4-socket"
},
"interfaces-config": {
"interfaces": [
"enp0s8"
]
},
"lease-database": {
"type": "memfile",
"persist": true,
"name": "/tmp/kea-dhcp4-leases"
},
"multi-threading": {
"enable-multi-threading": true,
"thread-pool-size": 4,
"packet-queue-size": 28
},
"cache-threshold": 0.25,
"calculate-tee-times": true,
"decline-probation-period": 15,
"hooks-libraries": [
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [
{
"this-server-name": "server1",
"mode": "hot-standby",
"peers": [
{
"name": "server1",
"url": "http://172.28.0.254:8000/",
"role": "primary"
},
{
"name": "server2",
"url": "http://172.28.0.253:8000/",
"role": "standby"
},
{
"name": "server7",
"url": "http://192.168.20.219:8000/",
"role": "backup"
}
]
}
]
}
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so"
}
],
"subnet4": [
{
"subnet": "172.28.0.0/24",
"id": 1,
"option-data": [
{
"name": "routers",
"data": "172.28.0.1"
}
],
"pools": [
{
"pool": "172.28.0.2-172.28.0.219"
}
],
"user-context": {
"ha-server-name": "server1"
},
"interface": "enp0s8"
}
],
"loggers": [
{
"name": "kea-dhcp4",
"severity": "INFO",
"output_options": [
{
"output": "stdout"
}
]
}
]
}
}
Kea 3 configuration
This server was unique in that it was participating in HA relationships with both "Kea 1" and "Kea 4". The "172.29.0.0/24" subnet in the "server3" <-> "server4" relationship featured perfdhcp instances with 172.28.0.250, 172.28.0.251, 172.28.0.249, or 172.28.0.252 passed as arguments to -l. The other relationship of "server5" <-> "server6" with subnet "172.30.0.0/24" featured instances of perfdhcp with 192.168.20.250, 192.168.20.251, 192.168.20.249, or 192.168.20.252 passed in -l.
Kea 3 configuration
{
"Dhcp4": {
"control-socket": {
"socket-type": "unix",
"socket-name": "/tmp/kea-dhcp4-socket"
},
"interfaces-config": {
"interfaces": [
"enp0s8","enp0s3"
]
},
"lease-database": {
"type": "memfile",
"persist": true,
"name": "/tmp/kea-dhcp4-leases"
},
"multi-threading": {
"enable-multi-threading": true,
"thread-pool-size": 4,
"packet-queue-size": 28
},
"cache-threshold": 0.25,
"calculate-tee-times": true,
"decline-probation-period": 15,
"hooks-libraries": [
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [
{
"this-server-name": "server3",
"mode": "hot-standby",
"peers": [
{
"name": "server3",
"url": "http://172.28.0.220:8001/",
"role": "primary"
},
{
"name": "server4",
"url": "http://172.28.0.253:8001/",
"role": "standby"
},
{
"name": "server8",
"url": "http://192.168.20.219:8001/",
"role": "backup"
}
]
},
{
"this-server-name": "server5",
"mode": "hot-standby",
"peers": [
{
"name": "server5",
"url": "http://172.28.0.220:8002/",
"role": "standby"
},
{
"name": "server6",
"url": "http://172.28.0.221:8002/",
"role": "primary"
},
{
"name": "server9",
"url": "http://192.168.20.219:8002/",
"role": "backup"
}
]
}
]
}
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so"
}
],
"subnet4": [
{
"subnet": "172.29.0.0/24",
"id": 2,
"option-data": [
{
"name": "routers",
"data": "172.29.0.1"
}
],
"pools": [
{
"pool": "172.29.0.2-172.29.0.254"
}
],
"user-context": {
"ha-server-name": "server3"
},
"relay": { "ip-addresses": [ "172.28.0.250","172.28.0.251","172.28.0.249","172.28.0.252" ] }
},
{
"subnet": "172.30.0.0/24",
"id": 3,
"option-data": [
{
"name": "routers",
"data": "172.30.0.1"
}
],
"pools": [
{
"pool": "172.30.0.2-172.30.0.254"
}
],
"user-context": {
"ha-server-name": "server5"
},
"relay": { "ip-addresses": [ "192.168.20.250","192.168.20.251","192.168.20.249","192.168.20.252" ] }
}
],
"loggers": [
{
"name": "kea-dhcp4",
"severity": "INFO",
"output_options": [
{
"output": "stdout"
}
]
}
]
}
}
Kea 4 configuration
This server had only one relationship in which it was the "primary". Note that at one point, this was also tested as the "standby" while "Kea 3" was the primary. That also worked. The subnet "172.30.0.0/24" is part of the "server5" <-> "server6" relationship. perfdhcp was passed one of 192.168.20.250, 192.168.20.251, 192.168.20.249, or 192.168.20.252 in the -l argument.
Kea 4 configuration
{
"Dhcp4": {
"control-socket": {
"socket-type": "unix",
"socket-name": "/tmp/kea-dhcp4-socket"
},
"interfaces-config": {
"interfaces": [
"enp0s8","enp0s3"
]
},
"lease-database": {
"type": "memfile",
"persist": true,
"name": "/tmp/kea-dhcp4-leases"
},
"multi-threading": {
"enable-multi-threading": true,
"thread-pool-size": 4,
"packet-queue-size": 28
},
"cache-threshold": 0.25,
"calculate-tee-times": true,
"decline-probation-period": 15,
"hooks-libraries": [
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [
{
"this-server-name": "server6",
"mode": "hot-standby",
"peers": [
{
"name": "server5",
"url": "http://172.28.0.220:8002/",
"role": "standby"
},
{
"name": "server6",
"url": "http://172.28.0.221:8002/",
"role": "primary"
},
{
"name": "server9",
"url": "http://192.168.20.219:8002/",
"role": "backup"
}
]
}
]
}
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so"
}
],
"subnet4": [
{
"subnet": "172.30.0.0/24",
"id": 3,
"option-data": [
{
"name": "routers",
"data": "172.30.0.1"
}
],
"pools": [
{
"pool": "172.30.0.2-172.30.0.254"
}
],
"user-context": {
"ha-server-name": "server6"
},
"relay": { "ip-addresses": [ "192.168.20.250","192.168.20.251","192.168.20.249","192.168.20.252" ] }
}
],
"loggers": [
{
"name": "kea-dhcp4",
"severity": "INFO",
"output_options": [
{
"output": "stdout"
}
]
}
]
}
}
Kea backup server configuration
This server participated in all relationships as a backup server. This could, of course, be separate Kea servers. There is no reason that a single server be used for multiple relationships. We chose this configuration to cut down on the amount of VMs that needed to be created.
Kea backup server configuration
{
"Dhcp4": {
"control-socket": {
"socket-type": "unix",
"socket-name": "/tmp/kea-dhcp4-socket"
},
"interfaces-config": {
"interfaces": [
"enp0s3"
]
},
"lease-database": {
"type": "memfile",
"persist": true,
"name": "/tmp/kea-dhcp4-leases"
},
"multi-threading": {
"enable-multi-threading": true,
"thread-pool-size": 4,
"packet-queue-size": 28
},
"cache-threshold": 0.25,
"calculate-tee-times": true,
"decline-probation-period": 15,
"hooks-libraries": [
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [
{
"this-server-name": "server7",
"mode": "hot-standby",
"peers": [
{
"name": "server1",
"url": "http://172.28.0.254:8000/",
"role": "primary"
},
{
"name": "server2",
"url": "http://172.28.0.253:8000/",
"role": "standby"
},
{
"name": "server7",
"url": "http://192.168.20.219:8000/",
"role": "backup"
}
]
},
{
"this-server-name": "server8",
"mode": "hot-standby",
"peers": [
{
"name": "server3",
"url": "http://172.28.0.220:8001/",
"role": "primary"
},
{
"name": "server4",
"url": "http://172.28.0.253:8001/",
"role": "standby"
},
{
"name": "server8",
"url": "http://192.168.20.219:8001/",
"role": "backup"
}
]
},
{
"this-server-name": "server9",
"mode": "hot-standby",
"peers": [
{
"name": "server5",
"url": "http://172.28.0.220:8002/",
"role": "standby"
},
{
"name": "server6",
"url": "http://172.28.0.221:8002/",
"role": "primary"
},
{
"name": "server9",
"url": "http://192.168.20.219:8002/",
"role": "backup"
}
]
}
]
}
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so"
}
],
"subnet4": [
{
"subnet": "172.28.0.0/24",
"id": 1,
"option-data": [
{
"name": "routers",
"data": "172.28.0.1"
}
],
"pools": [
{
"pool": "172.28.0.2-172.28.0.219"
}
],
"user-context": {
"ha-server-name": "server7"
},
//"interface": "enp0s8"
},
{
"subnet": "172.29.0.0/24",
"id": 2,
"option-data": [
{
"name": "routers",
"data": "172.29.0.1"
}
],
"pools": [
{
"pool": "172.29.0.2-172.29.0.254"
}
],
"user-context": {
"ha-server-name": "server8"
},
"relay": { "ip-addresses": [ "172.28.0.250","172.28.0.251","172.28.0.249","172.28.0.252" ] }
},
{
"subnet": "172.30.0.0/24",
"id": 3,
"option-data": [
{
"name": "routers",
"data": "172.30.0.1"
}
],
"pools": [
{
"pool": "172.30.0.2-172.30.0.254"
}
],
"user-context": {
"ha-server-name": "server9"
},
"relay": { "ip-addresses": [ "192.168.20.250","192.168.20.251","192.168.20.249","192.168.20.252" ] }
}
],
"loggers": [
{
"name": "kea-dhcp4",
"severity": "INFO",
"output_options": [
{
"output": "stdout"
}
]
}
]
}
}