DHCP 4.4.0 has a serious defect. ISC recommends users postpone upgrades until DHCP 4.4.1 can be issued
A serious flaw has been discovered in the recent release DHCP 4.4.0 that may cause the server to not properly write lease changes to the lease file. As a result, after restart the server may incorrectly treat leases that are assigned as available for allocation and eventually report them as abandoned.
Posting date: 07 February 2018
Program Impacted: DHCP
Versions affected: DHCP 4.4.0 (including development releases DHCP 4.4.0b1 and DHCP 4.4.0rc1)
DHCP 4.4.0 is the first release of DHCP 4.4, a new branch of DHCP. DHCP 4.4 is the first branch to have the delayed-ack feature compiled on by default. By default, the value of the delayed-ack parameter is 0. Due to a logic error, when these two conditions are satisfied (delayed-ack is enabled and the value for the delayed-ack parameter is 0) the server will incorrectly assume that the lease writes are to be delayed but never actually write them to a lease file. The server may appear to operate normally, responding to requests and granting leases, but the lease file will not be updated properly. This problem will become most apparent after the server is restarted, whereupon the addresses that had been earlier leased to clients will be incorrectly treated as available for allocation.
In the DHCP 4.4 default configuration, servers will confirm that an address is not in use before assigning it to a client (ping-check.) Since some addresses will have been assigned before the restart, the ping check test for those addresses is likely to indicate a conflict, causing the server to abandon it, choose another address and try again. This will cause unexpectedly high numbers of leases being marked as abandoned.
If the ping check feature is disabled, this can result in the server assigning an address to a new client despite it already being in use.
All 4.4.0 installations that do not have delayed-ack parameter set to non-zero value are affected. After restart the server may incorrectly report a significant number of addresses being used by unknown devices and mark those leases as abandoned.
First and most importantly: if you have not already upgraded to DHCP 4.4.0, avoid doing so until after DHCP 4.4.1 (which will correct the problem) is made available.
If you HAVE upgraded to 4.4.0 already, you may have some options for mitigating the severity of the problem. To begin with: if you performed a backup on your leases database before upgrading, you may be able to recover information for most of your clients, especially those that have been active and renewing since the upgrade. Even if you did NOT back up your leases database before the upgrade it may still contain information which will limit the scope of the issue. Back it up now.
While your server is running it will be keeping track of granted leases in volatile memory; the problem is that it is probably not saving new leases to non-volatile storage. The severity of this problem will depend to some extent on the amount of time since you upgraded and the rate at which new devices enter your network and request new leases. In network environments where client assignments are very stable the problem will develop slowly and most information is probably still in the existing copy of the leases file which was written before the upgrade. If your network changes slowly, you should pick a time to restart the server which will minimize disruption and restart the server after either:
- downgrading to a version without this issue,
- recompiling 4.4.0 without delayed-ack (i.e. use --disable-delayed-ack as a configure-time option when rebuilding ISC DHCP), or
- setting delayed-ack to a non-zero value using the delayed-ack configuration statement in dhcpd.conf
Any of those alternatives will prevent the problem from getting worse, but your server will still have to recover from those leases which were assigned to new clients but not recorded in non-volatile storage.
The greatest problems will be experienced by those operating networks in which clients frequently need new lease assignments. In such networks the longer you run an affected build of DHCP 4.4.0 the more your network state will diverge from the state information which is recorded in non-volatile storage. As soon as a convenient time can be found to restart the server (after applying one of the three options above to prevent recurrence after restart) the server should be restarted. Expect for it to take some time for the network to recover and be forewarned that conflicting leases may be assigned due to the failure of the server to record them before the restart. It may help to add an abandon-lease-time setting to your config (e.g. "abandon-lease-time 30;" to aid in recovery of leases marked as abandoned. You should also ensure that that ping-check will be turned on after any restart, as this will help to minimize the chance of handing out conflicting leases.
Solution: Do not upgrade to 4.4.0; wait for 4.4.1 to be released. If you have already deployed 4.4.0 use one of the mitigations described above.
Internet Systems Consortium (ISC) is providing this notice on an "AS IS" basis. No warranty or guarantee of any kind is expressed in this notice and none should be implied. ISC expressly excludes and disclaims any warranties regarding this notice or materials referred to in this notice, including, without limitation, any implied warranty of merchantability, fitness for a particular purpose, absence of hidden defects, or of non-infringement. Your use or reliance on this notice or materials referred to in this notice is at your own risk. ISC may change this notice at any time. A stand-alone copy or paraphrase of the text of this document that omits the document URL is an uncontrolled copy. Uncontrolled copies may lack important information, be out of date, or contain factual errors.