Tuning your BIND configuration effectively for zone transfers (particularly with many frequently-updated zones)

Zone transfers - AXFR and IXFR

When a primary nameserver is updated (irrespective of the mechanism through which this happens), the working contents of the zone held in memory that have changed need to be transferred to the other servers that are authoritative for that zone (the secondary servers).  This process is called 'zone transfer'.

There are two types of zone transfer - full (AXFR) and incremental (IXFR).  A full zone transfer is of the entire zones.  An Incremental zone transfer is a list of changes that, when applied, bring the secondary zone up to date.  The use of incremental zone transfers requires that both the provider and the recipient are maintaining a separate journal file of changes for each zone.  The default is that secondaries will request IXFR (unless otherwise configured) but they will always handle being given an AXFR instead, if no IXFR is available.

The critical component that controls the decision on whether or not the zone has changed is the serial number that is maintained in the zone's SOA record (although when a primary reloads, it does also check the timestamp on the zone data file first, before checking the zone's SOA).   BIND assumes that if the serial number remains the same, then the zone is unchanged.

Zone transfers are always initiated by the secondary server (that is, they are 'requested' or 'pulled').  While there are mechanisms in place to alert secondaries that they should check to see if a zone transfer is needed, the transfer itself is always started by the secondary.

Zone notifies and refreshes

Apart from when a server is restarted, or when rndc commands are used to force a zone transfer, there are two mechanisms that cause secondary servers to check whether or not their copy of a zone is current.  The process of checking and updating is called a zone refresh :

• Refresh timer expiry:
  Each zone's SOA record holds a refresh timer that all slave servers receiving a copy of the zone should use.  The refresh timer tells a secondary server how often it should ask one of the primary servers to which it has been configured to refer for an SOA record.  This is then compared to the SOA the secondary is holding - if it is the same, there is nothing to be done (and the secondary waits until the next refresh interval expiry).  There is also a retry timer which is applied if the refresh attempt fails (none of the primary servers could be contacted).  And finally, there's an expire timer - if the secondary has been unable to contact another primary for this period, then it stops serving data from that zone.  (Note that restarting the secondary will reset any 'expired' zones, so if they have a copy of the zone backed up to file, they will then resume serving the expired zone.)
• Notify received:
  When an authoritative (primary or secondary) server updates a zone, it can send out notifications to other servers that it has changed.  This causes the recipients to set their 'next refresh' time to 'now' and to queue a zone refresh.

For a detailed description of how a secondary zone performs a zone refresh, particularly if it has more than one primary configured, see How does BIND choose the primary for a zone refresh (zone timer or notify)?

Notification and refresh options unrelated to tuning/performance

There are a number of options that can be used to control (both on primaries and secondaries) which servers are notified when a zone is updated and which primary servers should be queried by secondaries when they are refreshing as well as handling the security implications of both via ACLs and TSIG.  This article assumes that the reader is familiar with all of these and that the zone transfer mechanism is both secure and functioning as expected, but needs some attention to tuning.

Potential problems

Where servers handle large numbers of zone transfers - either because they have zones that are highly dynamic (many and frequent changes) or are simply serving a very large number of zones, then some consideration needs to be given to tuning to avoid:

• Delays in propagation of zone updates (some authoritative servers providing old data for several minutes or hours after the zone has been updated)
• Refresh or transfer failures
• Servers non-responsive to clients (too busy handling zone transfers or network traffic peaks are causing dropped/lost/failed queries).

The last problem on the list is the one that historically has led to tuning options that 'throttle' zone transfers, but higher-speed hardware and networks developed in recent years mean that most servers should be able to handle a significant rate of zone maintenance activity in parallel with servicing client queries.

Configuration options for tuning zone transfers that are set too low may therefore cause other issues for zone administrators.  Some commonly-reported log messages are detailed below, along with the explanation of what is happening to cause them to be logged - but note that this is not a complete list!  Some messages are logged from category general , and others from categories xfer-in and xfer-out .  On the server providing the zone transfers you might also see some relevant messages logged from category client.

Commonly-reported Logfile Messages

16-Nov-2011 16:31:07.044 xfer-in: error: transfer of 'testzone.example.com/IN' from 192.0.2.1#53: failed to connect: timed out

This means that a zone transfer was started but failed.  Sometimes this is due to the TCP connection between the master and slave being reset but it can also be due to the primary being inaccessible from the secondary due to routing or firewall issues.  The outcome is doubly bad:

• The zone will not be refreshed again until the retry timer period or another notification is received.

• The primary server is temporarily cached as 'unreachable' and won't be retried for future transfers until at least 10 minutes has passed since it was last discovered to be unreachable or until a notify is received from it.

failed zone transfer attempts will now be removed from the
"unreachable" list (i.e. considered reachable again) if the secondary receives a NOTIFY message from them. [RT #25960]



See also: What does named log message "deleted from unreachable cache" mean?.)

Note also that when attempting a zone refresh there is a fallback from UDP to TCP on the SOA query.  If there is no response to the UDP SOA queries from any server, the final attempt is to retrieve the SOA via TCP.  It's TCP connection failures that cause the server to be added to the unreachable cache - either the last attempt at an SOA query, or, if the SOA query was successful via UDP,  the first attempt at a zone transfer.

17-Nov-2011 20:46:51.807 general: info: zone testzone.com/IN: refresh: skipping zone transfer as master 
192.0.2.1#53 (source 192.0.2.4#0) is unreachable (cached)

Here we have an indication that a zone refresh attempted earlier has failed and that this information was cached.  There are several similar error messages, but when the reason for the zone refresh not taking place is that the primary's IP address is already in the unreachable cache, then the text 'unreachable (cached)' is always included.

17-Nov-2011 21:50:14.762 xfer-in: info: transfer of 'testzone.com/IN' from 192.0.2.1#53: connected using 192.0.2.4#47296
17-Nov-2011 21:50:14.762 xfer-in: error: transfer of 'testzone.com/IN' from 192.0.2.1#53: failed while receiving responses: connection reset
17-Nov-2011 21:50:14.762 xfer-in: info: transfer of 'testzone.com/IN' from 192.0.2.1#53: Transfer completed: 0 messages, 0 records, 0 bytes, 0.001 secs (0 bytes/sec)

Above we can see that a zone transfer was started, but failed due to the TCP connection to the primary server being reset.  Typically this will be due to the tcp-listen-queue not being increased in line with other tuning options.

• Note that this error has been logged from category xfer-in - this is an example of more detailed information relating to an error logged in the general logging category (as seen above).
• Since the zone transfer operation was started, it also logs at completion, but you will observe that the statistics indicate that no records were transferred .
• There are some variations that may be seen for a connection reset, depending at which point in the life of the TCP connection that the reset was received.  Some variations are:

17-Nov-2011 21:50:15.712 xfer-in: error: transfer of 'testzone.com/IN' from 192.0.2.1#53: failed sending request length prefix: connection reset

17-Nov-2011 21:50:28.394 xfer-in: error: transfer of 'testzone.com/IN' from 192.0.2.1#53: failed sending request data: connection reset

The significant portion of the message from each of the above is "connection reset."

13-Jan-2012 18:45:51.036 client: warning: client 192.0.2.4#42229: no more TCP clients: quota reached.

This is another problem relating to TCP tuning - in this case a warning logged on the master server which should be providing the zone update.  There is a tunable limit (tcp-clients ) on the number of TCP 'clients' that can be handled concurrently - this is shared between client queries that are received over TCP and zone transfers.  This limit has been reached - which can be another reason for a zone transfer to see the a connection reset error logged from category xfer-in .

13-Jan-2012 18:45:01.871 general: info: zone testzone.com/IN: notify from 192.0.2.1#62160: refresh in progress, refresh check queued

A notify was received, but the zone being notified was already in the process of being refreshed or is waiting to be refreshed, so the check is queued and will be processed later.

14-Jan-2012 10:34:06.054 general: info: zone testzone.com/IN: notify from 192.0.2.1#20016: zone is up to date

A notify was received but upon checking, the secondary zone SOA serial number is already the same as on the primary it checked, therefore no zone transfer is required.

13-Jan-2012 18:45:38.702 xfer-in: info: zone testzone.com/IN: zone transfer deferred due to quota

This error is logged on a secondary server and it indicates that it has reached one of its own configured limits on concurrent inbound transfers (transfers-in or transfers-per-ns set globally, or for a specific server statement, thetransfers option) - which is not necessarily a bad thing depending on your bandwidth and chosen tuning options. The transfer is deferred (queued to be retried later).

02-Aug-2012 07:58:20.601 general: info: master 192.0.2.4#53 (source 192.0.2.8#0) deleted from unreachable cache

BIND maintains a cache of unreachable primaries to which it refers when handling a zone refresh.  If a zone refresh fails with a specific primary (either during the query for the SOA or after querying and while attempting a subsequent zone transfer), then this primary is cached as 'unreachable' for 10 minutes.  As of versions
9.6-ESV-R6, 9.7.5, 9.8.2 and 9.9.0 onwards BIND implements an earlier removal of a master server from the unreachable cache if a notify is received from it. (Note that receipt of a notify (which is a UDP packet travelling from primary to secondary) doesn't guarantee that the primary will be reachable from the secondary, but it does ensure quicker recovery in the situation where a primary was temporarily unavailable, for example for a reboot.)

Options for tuning masters

When tuning the behavior of the primary, there are several factors that you can control:

• The rate of notifications of changes to secondary servers (serial-query-rate and notify-delay)
• Limits on concurrent zone transfers (transfers-out, tcp-clients, tcp-listen-queue, reserved-sockets)
• Efficiency/management options (max-transfer-time-out, max-transfer-idle-out, transfer-format)

The most important options to focus on are transfers-out, serial-query-rate, tcp-clients and tcp-listen-queue.

transfers-out is the maximum number of concurrent zone transfers outbound that will be permitted - the default is 10.  If you make this value too large on a primary server with many zones that are frequently updated, you may find that your server is too busy handling zone transfers to handle queries effectively.  It will also need to be able to handle the volume of concurrent TCP client connections (see tcp-clients) for zone transfers.

tcp-clients controls the maximum number of inbound TCP connections to this server.  Inbound TCP connections includes both zone transfers and TCP-based queries.  The default is 100.  You can monitor the count of tcp clients using rndc status (provides a one-time snapshot of the counters - you can repeat the command, collect the statistics and look at trends).

tcp-listen-queue is a subtle control setting (although not applicable to all OS environments).  When there is a high rate of inbound TCP connections, it controls how many connections can be queued before they are accepted by the application.  If named has already reached both the limit on concurrent zone transfers, and the limit specified by tcp-listen-queue, then any new inbound TCP connections will be dropped.  If you're expecting a high rate of zone transfers or that zone transfer requests will be competing for primary server resources, then you should increase this configuration option, whose default is 3 (increased to 10 from BIND 9.10, 9.9.4, 9.8.6 and 9.6-ESV-R10).

serial-query-rate (default 20) is a rate-limiter, that is used to control (depending on which version of BIND is running) both the rate of notifies and of zone refresh (SOA queries).  Although the limit is expressed as a per-second rate, it is the actions that are being limited, not the network packets needed to complete an action.  So for example, in the case of a zone refresh, there may multiple queries sent as named retries if no response is received, or tries different authoritative servers.  You are therefore recommended to increase the setting gradually to a level that meets your production environment's requirements.  The effects of this setting are usually most apparently immediately following a restart of server that is authoritative for a large number of zones.

Note that older versions of BIND managed notifies and SOA refresh queries in a single queue (which sometimes caused problems for slaves that are also masters for other servers). To ensure that notifies and refreshes were not competing with each other, in BIND versions 9.6-ESV-R11, 9.8.7, 9.9.5 and 9.10.0 we introduced independent queues - both still with serial-query-rate controlling them.

From BIND 9.9.7-S1 (and this change will also be found in BIND 9.11.1) there are three separate rate-limiting controls: serial-query-rate ; notify-rate and startup-notify-rate .

For more information on rate-limiting notifications and SOA refresh queries, please read: serial-query-rate, notify-rate and startup-notify-rate: how they impact zone transfers in different versions of BIND.

notify-delay goes hand-in-hand with overall rate-limiting but is used to configure what happens for notifications for a single zone.  The default is 5 seconds - which means that even if the zone is being updated constantly, there will not be a storm of notifications being sent to secondary servers.  Instead, the notifies are deferred until 5 seconds has elapsed.   It would be unusual to change this configuration setting, but it might be appropriate in some situations.

max-transfer-time-out is used to terminate outbound zone transfers that have not completed by a set time limit.  The default is 2 hours, which should be adequate for most scenarios.

max-transfer-idle-out is used to terminate outbound zone transfers that are no longer making any progress.  The default is 1 hour.

If your network connectivity is unreliable and you are observing high numbers of 'stalled' zone transfers, then consider reducing both of the above max-transfer-*-out settings.  You might also want to tune some of your kernel parameters for TCP too.

transfer-format is used either globally or per server (using the server statement) to control whether zone transfers send one record (one-answer ) or many records (many-answers ) per DNS message.  Clearly many-answers is more efficient, but at the time that many-answers format was introduced, adoption was not universal and some older servers could not handle this type of zone transfer.  Nowadays it would be unusual to find any servers that don't support many-answers format, but the option may have persisted from old configuration files.  If zone transfers are slow, it would be worth checking that your primary server is not still using one-answer format.

Options for tuning secondary servers

When tuning the behavior of a secondary server, you can control:

• The rate of inbound zone transfers on a per server basis, as well as in aggregate (transfers-in, transfers-per-ns, server option 'transfers')
• The rate at which refresh (SOA) queries are sent (serial-query-rate, min-refresh-time, max-refresh-time, min-retry-time, max-retry-time)
• Efficiency/management options: (try-tcp-refresh, max-transfer-time-in, max-transfer-idle-in)

The most important options on which to focus are transfers-in and transfers-per-ns.

transfers-in is the maximum number of concurrent zone transfers inbound that will be permitted - the default is 10.  If you make this value too large on a secondary  server with many zones that are frequently updated, you may find that your server is too busy handling zone transfers to handle queries effectively.  Depending also on your secondary zone configuration and zone data propagation strategy, each inbound zone transfer completion may cause onward notifications to other servers along with inbound SOA queries that also increase the workload.

transfers-per-ns is a per-server limit on concurrent zone transfers inbound.  It's a 'good behavior' configuration setting, so that even though the overall maximum number of inbound zone transfers is larger, named is preventing itself from hitting any one primary server too hard.

The server phrase transfers can be used to override the global transfers-per-ns setting for a specific server.  If the per-server transfer rate is too high for your primary servers, you may find that attempted zone transfers are rejected and the server is added to the unreachable primaries list (for up to 10 minutes).

serial-query-rate is used to tune the rate at which SOA refresh queries (for secondary zones) are sent to the listed primaries.  Depending on the version of BIND being used it may also be controlling the rate at which notifies are sent.  Although the limit is expressed as a per-second rate, it is the actions that are being limited, not the network packets needed to complete an action.  So for example, in the case of a zone refresh, there may multiple queries sent as named retries if no response is received, or tries different authoritative servers.   You are therefore recommended to increase the setting gradually to a level that meets your production environment's requirements.  The effects of this setting are usually most apparently immediately following the restart of secondary server since this sets effectively sets the refresh time to 'now' (with some jitter) on all zones, even if they are initially loaded and being served from disk backups of the zone data.

Note that older versions of BIND managed notifies and SOA refresh queries in a single queue (which sometimes caused problems for secondaries that are also primaries for other servers). To ensure that notifies and refreshes were not
competing with each other, in BIND versions 9.6-ESV-R11, 9.8.7, 9.9.5 and 9.10.0 we introduced independent queues - both still with serial-query-rate controlling them.

From BIND 9.9.7-S1 (and this change will also be found in BIND 9.11.1) there are three separate rate-limiting controls: serial-query-rate ; notify-rate and startup-notify-rate .

For more information on rate-limiting notifications and SOA refresh queries, please read: serial-query-rate, notify-rate and startup-notify-rate: how they impact zone transfers in different versions of BIND.

Zone refresh times are usually controlled by the primary server for a zone which provides them to its secondaries in the SOA record as refresh and retry intervals and zone expire interval (see above). Busy secondary servers may want to have some control over their own behavior however, so the following settings can be used to set boundaries on the refresh and retry settings received in SOA records. The primary server's expire time however is always observed. It would be unusual, although not unreasonable to reconfigure these limits, although bear in mind that frequent refreshes will use both network bandwidth and server processing resources.

min-refresh-time is used to set a lower boundary on the rate at which a zone refresh is scheduled for any one secondary zone.  The default is 300 seconds (5 minutes).  Even if the primary server requests a more frequent refresh interval in the SOA record, named will override this.

max-refresh-time (default 4 weeks) is used to ensure that copied zones are periodically refreshed from the primary, even if the primary server has tried to specify otherwise.

min-retry-time (500 seconds) and max-retry-time (2 weeks) are used similarly to limit the range of retry timer values that may be received in an SOA record for a zone.

max-transfer-time-in is used to terminate inbound zone transfers that have not completed by a set time limit.  The default is 2 hours, which should be adequate for
most scenarios.

max-transfer-idle-in is used to terminate inbound zone transfers that are no longer making any progress.  The default is 1 hour.

try-tcp-refresh is a compatibility setting that defaults to yes.  When set, if all prior attempts to obtain the SOA record over UDP have failed, then before giving up, there will be one more try, this time using TCP.  It would be unusual to change this setting.