Kea Performance Optimization

We often receive requests for suggestions about how to optimize Kea's performance. It's difficult to give recommendations, as your hardware and software choices, the size and complexity of your configuration, and local factors such as network latency all contribute, but here's some general guidance.

1. Choose the right lease file backend

Each lease must be written to disk, and where this file is written significantly impacts performance. The default "memfile" backend is the fastest. Writing the lease to an external database engine (MySQL or PostgreSQL) is expensive and is much less performant than memfile. Consider whether you really need to store the lease database in an external database engine; often, it is not necessary as it offers no benefit to Kea itself. Please note that if you consider storing the leases in an external database engine because you wish to access lease data, this is often easier with the API, which allows the use of memfile.

2. Choose the right hardware

The kea-dhcp4 and kea-dhcp6 processes are CPU-intensive. All supported Kea versions can run in multi-threaded mode, considerably improving performance. Multi-threading is enabled by default from Kea 2.4.0 onward.

It is often best to spec a system with a Solid State Drive (SSD) system of some sort. This greatly improves disk I/O when compared with the traditional spinning magnetic head-based hard drives. This can be important for logging or lease storage, depending on the number of leases served per second.

It should be noted that performance of the Kea server starts to fall off with more than 16 cores in use, likely due to I/O contention among the threads. The number of threads in use by a single process can be influenced by the thread-pool-size parameter.

If running both kea-dhcp4 and kea-dhcp6 on the same server, be sure to split this parameter appropriately between the two, based on the lease-per-second volume of each. You might find that kea-dhcp4 is busier than kea-dhcp6 and therefore should be allocated more cores. Take care that the two added together do not exceed the number of CPUs available on the hardware. In especially high-volume networks, consider running the kea-dhcp4 and kea-dhcp6 processes on separate servers.

3. Minimize network latency between Kea components

When using Kea with a database backend of any kind, it is ideal to locate both Kea and the database on the same host to minimize latency. Given the advice above, to maximize clock speed for Kea and disk performance for the backend, you will need a host with both a fast clock and high-performance storage for the best performance. Measuring the latency from your Kea instance to the database is important when using clustering technologies.

4. Help Kea find an available lease quickly

The more complex your configuration, the more structures must be searched to find and confirm an address. Until Kea 2.4.0, a simple iterative allocator was used to find the next available lease. Since Kea 2.4.0, there are three different allocators to choose from: the Iterative Allocator (default), the Random Allocator, and the Free Lease Queue Allocator. General guidelines are:

• Use fewer subnets if you can.
• Avoid using the iterative allocator when employing a shared lease database. When multiple Kea servers configured with the Iterative Allocator share a single lease backend (e.g., a cluster of databases serving as the lease backend with multiple Kea instances sharing the same pools of addresses for allocation), they will run into contention for assigning addresses. Multiple Kea instances will attempt to assign the next available address; only the first one will succeed, and the others will have to retry.
• One large pool is better than many smaller ones; keeping pool utilization below 80% is recommended. Consider using the Free Lease Queue Allocator in subnets with highly utilized address pools. Still, if your pool utilization is over 90%, consider extending your subnet or using a shared network.
• Avoid shared networks if you can. Shared networks are a powerful feature that lets Kea overlay multiple IP subnets on the same physical link; this is very useful, but it has significant performance implications. In many cases, Kea has to go through a more complex decision algorithm (akin to "Am I out of addresses in this subnet? Let's change to a different subnet in this shared network and try again.").
• kea-dhcp4 does two lease lookups: by client ID (if the client sent a client ID) and another one by MAC address if a lease is not found. If the administrator doesn't care about the client ID (which is sometimes not provided by the client) that lookup can be disabled via the match-client-id setting. The default value of "true" indicates that the server will use the client-identifier for lease lookups and chaddr if the first lookup returns no results. The value "false" means that the server will use only the chaddr to search for the client’s lease. Setting match-client-id to "false" cuts the number of searches for a lease in half and may provide a significant performance boost. See here for more details.

5. Follow these tips for host reservations

• Searching for possible host reservations (HRs) is expensive. If you use HRs, decide which part of your address space should be used for reserved addresses and which for dynamic assignment. When an address is considered, Kea asks the database: "Is this candidate address used by anyone right now?" If the answer is no, the next question is: "Is this address reserved for anyone?" You can omit the second question if you tell Kea there are no reservations for addresses from the dynamic pool. See "reservations-out-of-pool": true.
• If you don't need HRs, you can tell Kea not to look for them by setting "reservations-global": false, "reservations-in-subnet": false. If you only use HRs in a few subnets, disable host reservations globally and enable them on a per-subnet basis. This can be done by disabling global and in-subnet reservations at the global level and setting "reservations-in-subnet": true in only the subnets that contain reservations.
• Eliminate unused host identifiers. Kea can identify host reservations in several ways: circuit ID, MAC (hardware) address, DUID, or client ID in DHCPv4, and DUID or MAC (hardware) address in DHCPv6. Kea checks all identifier types in sequence. Suppose you know that you don't use certain types (e.g. you always identify your hosts by MAC address alone): in that case, you can tell Kea not to bother trying different identifiers, e.g. host-reservation-identifiers: [ "hw-address" ]. If you want to use several identifier types, order the list from most to least frequent, e.g. if you have 990 reservations by MAC address and ten reservations by client ID, set host-reservation-identifiers: [ "hw-address", "client-id" ].

See here (DHCPv4) or here (DHCPv6) for host reservation tuning information, including host identifier tuning.

6. Remember that hooks add latency

Dynamically linked Kea hooks are a powerful way to add functionality but can limit performance. In particular, a hook that queries an external system, such as an external database, adds latency. Depending on the external system, it may be a significant bottleneck. For example, the ISC subscriber-only hook that integrates a look-up to a RADIUS database can give a significant performance hit. RADIUS was designed for lower throughput than many administrators expect from their DHCP systems.

7. Consider the impact of High Availability

With High Availability (HA) enabled, one server has to wait for its partner's confirmation before sending a reply to a client, which has slight performance implications. The performance impact was previously more drastic; as of Kea 2.0.0, it is possible to configure the Kea HA hook to use multi-threading and communicate directly with the HA peer rather than through the kea-ctrl-agent. This removed a significant bottleneck in the performance of the HA hook, and is now the default setting since Kea 2.4.1. See here for further information.

There are still some additional tips that may increase performance of HA-hook enabled Kea servers:

• Minimize the latency (round-trip time) between two cooperating HA servers. A Kea server will not respond to a client until its HA partner has properly processed and responded to the lease update. The faster the partner server can respond, the faster the client will receive a response.

• "Hot-standby" mode is more efficient than "load-balancing" mode. It is also easier to configure, as "load-balancing" mode requires manual splitting of the pools and assignment of special client classes to the pools (see here for further information). In hot-standby mode, only one of the servers processes the received DHCP messages and responds to clients. However, in load-balancing mode, both servers process half of the client updates. Load-balanced servers must still be able to process all packets when one of the servers is down, so nothing is really gained here.

8. Don't log at debug level in production

Hopefully, this advice is self-explanatory.

9. Note that REST operations are not "free"

Kea processes some REST API commands in a synchronous manner, which means that DHCP packets are not processed while the commands are being received. Other commands may not block packet processing, but they may still degrade performance while being processed. The general rule is that if a command writes something to the configuration, then it probably blocks packet processing (e.g., reservation-add). If the command is a read command, then it probably doesn't block (e.g., stat-lease4-get). In the past, before multi-threading, all of these commands blocked packet processing while the command was processed and answered. Even read commands can cause some moderate performance degradation, if, for example, the command requires complex operations (such as retrieving statistics from all subnets when there are thousands of subnets configured). Also, If many small commands are being sent frequently, they can still have a degrading impact on performance.

10. Don't use client classification expressions to enumerate a long list of specific clients

Enumerating individual clients in class test expressions can be inefficient and will grow more so as the number of expressions increases. If you find yourself doing this, you should use global host reservations to assign clients to classes instead.

Kea evaluates every test statement in a client classification operation before proceeding. Even after finding a match, Kea must continue evaluating subsequent statements in case additional matches exist. An excessively long list of test cases in a classification statement can impact performance.

11. Avoid lengthy regular expressions

Evaluating regular expressions is expensive. Suppose you are using the Forensic Logging feature, for example. In that case, the default log is relatively high-performing compared to the forensic log, which requires evaluating an expression to determine what to log. When using custom forensic log output expressions, use the shortest regular expression that will work for your situation.