Kea: Use unique databases

Summary

When configuring Kea to use a database for storage of leases or host reservations, use a unique database for each Kea server.

Within a single Kea high availability group (HA group), the database may be shared, subject to certain considerations.

Guidance

Use a unique database for every Kea server.

A database can be made unique by creating a different database name on the same database server, or by using different database servers. For example, if you use a central database server, create a different database for each Kea server, and specify the appropriate name in the Kea config files. Another example would be using a local database server on each Kea server: The host can be 127.0.0.1 and the name can be the same on each, as each database server will be unique to the local host.

There are some exceptions to this guidance. If you have already deployed Kea using shared databases, and none of the exceptions apply, see Deviations.

Background

The Kea DHCP server uses JSON-format config files to store configuration information, such as kea-dhcp4.conf and kea-dhcp6.conf. Optionally, Kea can also use an external database ("SQL database") for certain kinds of data, most commonly leases and host reservations.

In the config files, lease storage is specified in the lease-database section. Specifying anything other than memfile (CSV file) means an external database engine will be used.

Host reservations may be defined directly in a config file, in a reservations section. Alternatively, the config file may specify a hosts-database section, telling Kea to use an external database engine for storage of host reservations.

For both of these, specifying an external database requires specifying several parameters. The host and port together generally identify a database server; name identifies a particular database on that server. Using a separate database server instance also means the database itself will be separated.

Exceptions

High Availability

Certain high availability (HA) configurations may use a shared database in a supported fashion. Other HA configurations allow or require separate databases for each Kea server in an HA group.

Leases

A single HA group can use the same database for leases if you are employing the shared lease database concept. In this scenario, the database engine is responsible for replicating lease data, and presenting it to Kea servers in the HA group.

If you are using one of the HA modes provided by Kea (using the libdhcp_ha.so hook), such as hot-standby or load-balancing, you have two choices.

The default is for the HA hook to replicate lease data between the two Kea servers. In this scenario, a separate database must be used for each Kea server's lease data. This may be desirable; see Impact on Availability below.

Alternatively, one can combine the HA hook with the shared lease database strategy. In this case, the HA hook will manage which Kea server answers DHCP requests, but the database will be responsible for lease replication. This requires setting send-lease-updates and sync-leases to false. See Lease Information Sharing in the ARM.

Host Reservations

A single HA group can typically use the same database for host reservations, as both servers will be serving the same subnets and offering the same reservations.

If you are sharing the host reservation database, when making changes, you should send the API commands to one server only. With Stork, select only one daemon. Since the database is shared, the one change will take effect for both Kea servers. If you were to send the API command to both servers, one of the commands would fail (being seen as a duplicate of the other command). This error is generally harmless, but may confuse people or programs.

Hub-and-spoke configurations, or any other configuration where different Kea servers in the HA group will offer different subnets or reservations, should have separate databases, and manage reservations individually.

Impact on Availability

Note that using a shared database in HA scenarios makes the entire HA group dependent on that database. If the database malfunctions, that will impact the entire HA group. Thus, you should ensure the database is also highly available.

Alternatively, use a different database server for each member of the HA group, just as you would for two unrelated Kea servers. This will make Kea independent of any one database server. For lease information, use the HA hook to replicate between databases. For host reservations, send the same add/delete API commands to both servers when making changes.

Configuration Backend

The use of the configuration backend (CB) assumes a single shared database for all configuration data. The rule against shared databases does not apply to anything within a config-databases section.

Reasons

Why does ISC recommend this?

General

The design and implementation of Kea generally assumes each database table will be a unique dataset for that server. No mechanism is provided to ensure records or identifiers are unique. Thus, if more than one Kea server is using the same database, identifier collisions become possible.

This might result in unexpected "duplicate entry" errors when adding/updating records in the database, leaving you unable to make your change. A query might return the same records for what should be distinct items. A data change might overwrite a record when it should create a new one. The overall behavior would be confusing to both people and programs.

Schema

Each Kea version targets a specific database schema version, and cannot run against other schema versions. When each Kea instance has its own database, one simply upgrades the schema during the Kea upgrade, and continues. With multiple Kea servers running against a single database, schema upgrades must be carefully coordinated.

Stork

The Stork management suite, which works with Kea, also assumes that databases are unique to each Kea server. Inconsistent behavior and errors can be seen if this assumption is violated.

Deviations

First, see if one of the exceptions applies; you may be fine. Otherwise, the following advice can help minimize any problems.

Strategy

If you have already deployed Kea using the same database for multiple Kea servers, and things are working to your satisfaction, ISC generally recommends continuing in that mode of operation. Separating a Kea database into multiple databases is a very intrusive process, involving downtime and increased chance of error.

If you need to add new Kea servers to such a deployment, give them their own database, per the guidance herein. This avoids compounding the problem. The pre-existing servers can be considered a legacy configuration.

Mitigation

The following practices will reduce (but not eliminate) the chance of a shared Kea database causing problems.

Avoid the use of global reservations
Make your subnet IDs be unique across all Kea servers
Avoid overlapping IP address ranges across servers
For HA, configure per High Availability above
Send Kea API commands to only one server when making changes
With Stork, select only one Kea daemon at a time when making changes

Schema updates

Upgrades to Kea versions which include updating the database schema will require special care. You need to ensure both of the following:

Older Kea does not run against the newer schema
Newer Kea does not run against the older schema

The suggested upgrade procedure would look something like this:

Stop all Kea daemons on all servers
On one Kea server:
a. Upgrade the Kea software
b. Update the database schema
c. Start the Kea daemon(s)
d. Confirm correct operation
On all other Kea servers:
a. Upgrade the Kea software
b. Start the Kea daemon(s)
c. Confirm correct operation

Step 3 can be done one server at a time, or across all servers in parallel, as long as the Kea software upgrade (step 3a) is completed first for any given server.

Rehabilitation

As noted above, separating a Kea database is a major undertaking, and may not be strictly necessary. For ISC customers with a support agreement, we recommend contacting us. Our support engineers can analyze the specifics of your scenario and provide appropriate assistance.