Why is BIND re-priming the roots from hints more often than it should?
A quick recap on root priming
When handling a client query for which it has no answer, a recursive server needs to know which authoritative server is responsible for the domain of the name it has been asked to resolve. So for example, if a client queries for hostname.example.com, a recursive server that has just been restarted will not know where to find the authoritative servers for example.com, nor the authoritative servers for com. It will need to start at the 'top', which it does by sending a the query for hostname.example.com to one of the root nameservers.
But how does the recursive server know where the root nameservers are? The answer to this is found in the root hints.
The root hints are a list of the servers that are authoritative for the root domain ".", along with their IPv4 and IPv6 addresses. In other words, this is a collection of NS, A, and AAAA records for the root nameservers.
There is a subtle clue in the name we give the root hint zone - the word 'hint'. Although the names and addresses for the root nameservers don't change very often, there are occasional updates. Thus the root hint zone is assumed to be mostly accurate, but it is not treated as the authoritative list of the roots. Instead, named will query the servers in the root hints list in turn, asking them for the NS records for ".". When an authoritative response is received, the answer is added to cache, and it is the newly primed set of root nameservers that is used, rather than the set from the hints file.
For more detailed information on root priming, please see: Root hints - a collection of operational and configuration FAQs.
An unfortunate and almost invisible bug
BIND versions 9.12.0, 9.11.3, 9.10.7 and 9.9.12 contain a fix for the following bug:
4770. [bug] Cache additional data from priming queries as glue. Previously they were ignored as unsigned non-answer data from a secure zone, and never actually got added to the cache, causing hints to be used frequently for root-server addresses, which triggered re-priming. [RT #45241]
This too-frequent root priming problem has existed for a very long time, but the fix for bug RT #22842 in versions 9.8.1, 9.7.4 and 9.6-ESV-R10 introduced as a side effect an increase of the potential frequency of this from 30 minutes to 1 second.
Most of the time, resolver operators would be completely unaware that their servers were re-priming the roots more often than they should because there is no operational impact to them and nothing is logged when root priming occurs except when there is a mismatch between the configured or built-in root hints and what is returned in the priming query response. When there is a mismatch, for example when one of the root nameservers has changed an address, then named will log a checkhints warning - for example:
13-Nov-2017 11:15:02.792 general: warning: checkhints: b.root-servers.net/A (22.214.171.124) missing from hints 13-Nov-2017 11:15:02.792 general: warning: checkhints: b.root-servers.net/A (126.96.36.199) extra record in hints
The mismatch itself is not a problem (more information about this is available in Root hints - a collection of operational and configuration FAQs). But the logged message means that it's possible to see how often BIND is sending priming queries.
What caused the bug?
First we need to look at the way BIND assigns levels of trust for what it learns and stores in cache: Trust levels for RRsets in BIND cache.
The root hints RRsets are loaded from local storage, and are thus regarded as having the ULTIMATE trust level - but they are a special case, because we want them to be replaced as soon as possible by sending a query to the root zone for up-to-date information. Therefore, when the root NS RRset is needed to resolve a query but is not in the cache, root hint RRsets are loaded into the cache with TTL=0 (meaning they will expire at the end of the current second). The action of loading root hints into the cache automatically triggers a priming query - that's a query sent to one of the root nameservers for the NS RRset for "." - so that the cached hints will be replaced right away.
The root zone itself is DNSSEC-signed, and the answers provided by the root nameservers are authoritative, therefore the NS RRset for "." is loaded into cache with trust level AUTH-ANSWER or SECURE, (depending on whether or not DNSSEC-validation is configured) replacing the hints root NS RRset. Note that the root priming query has specifically requested this RRset, so the Answer Section in the query response is used to populate cache.
Meanwhile, the addresses (A and AAAA RRsets) for the root nameservers have been included in the Additional Section of the priming query response. These are the RRs that named needs for its list of servers by address that are authoritative for the root zone. The zone root-servers.net is not DNSSEC-signed, so even on a DNSSEC-validating server, the trust level for the A and AAAA RRs provided during root priming is only ADDITIONAL (not GLUE - this not a delegation referral). Therefore the A/AAAA RRsets for the root nameservers received in the priming response are discarded and the cache is not updated. After the root hint RRsets have expired, if another query for a new TLD is processed, the resolver will have to fall back to the root hints again. The hints are once again loaded into the cache with TTL=0, triggering yet another priming query, the results of which will once again be discarded, and so on.
The fix has been to promote the additional data received in the response from priming queries to GLUE. This means that it will be kept in cache in the same way as nameserver addresses received in a delegation referral response, so that when named needs to contact those servers, it can query the zone root-servers.net for confirmation of their addresses, which will then be updated in cache as AUTH-ANSWER. (Essentially, the solution has been to treat the information from root hints as being the equivalent of a delegation response from a parent to a subdomain where the addresses received in the additional section are cached as necessary glue.)
How can I get rid of the logged checkhints warnings?
Although they can safely be ignored (and will go away or become very infrequent when BIND is upgraded to a version that fixes bug #45241), some DNS administrators may prefer to eliminate the logged messages. There are two workarounds that should be effective:
- Eliminate the checkhints warnings by updating the root hints.
This solution does not change how often named is re-priming the roots, but it addresses the hints/action mismatch.
- If you are using a manually-configured root hints file, then update it with the current root NS/A/AAAA RRsets for the root nameservers.
- If you are using the built-in root hints, then (temporarily) configure it to use a manual root hints file which contacts the current root NS/A/AAAA RRsets.
- Temporarily eliminate the root re-priming by populating cache manually with the root nameserver A and AAAA RRsets.
- To do this, you will need to send queries to your resolver for each root nameserver's A and AAAA RRset.
- The query responses will be added to cache and will persist per their TTL (although they may be discarded sooner than this if max-cache-size is reached or if max-cache-ttl in named.conf is lower than the TTL on the received RRset).