Contents x
Recursive Client Rate limiting
  • 07 Oct 2021
  • 8 Minutes to read
  • Contributors
  • Print
  • Dark
    Light
  • PDF
Contents

Recursive Client Rate limiting

  • Updated on 07 Oct 2021
  • 8 Minutes to read
  • Contributors
  • Print
  • Dark
    Light
  • PDF
Article Summary

The Recursive Client Rate Limiting feature was introduced in BIND 9.11 (and was also made available as a build-time option in BIND 9.9.8 and 9.10.3)

This feature is intended to optimize recursive server behavior in favor of good client queries, whilst at the same time limiting the impact of bad client queries (e.g. queries which cannot be resolved, or which take too long to resolve) on local recursive server resource use.

Rate-limiting Fetches Per Server

The fetches-per-server option sets a hard upper limit to the number of outstanding fetches allowed for a single server.  The lower limit is 2% of fetches-per-server, but never below 1.  It also allows you to select what to do with the queries that are being limited - either drop them, or send back a SERVFAIL response.

Based on a moving average of the timeout ratio for each server, the server's individual quota will be periodically adjusted up or down.  The adjustments up and down are not linear; instead they follow a curve that is initially aggressive but which has a long tail.

The default value for fetches-per-server is 0, which disables this feature.  When fetches-per-server is enabled, the default behaviour when rate-limiting is active is to SERVFAIL queries that exceed the limit.

The fetch-quota-params option specifies four parameters that control how the per-server fetch limit is calculated.

For example:

fetches-per-server 200 fail;
fetch-quota-params 100 0.1 0.3 0.7;

The first number in fetch-quota-params specifies how often, in number of queries to the server, to recalculate its fetch quota.  The default is to recalculate every 100 queries sent.

The second number specifies the threshold timeout ratio below which the server will be considered to be "good" and will have its fetch quota raised if it is below the maximum.  The default is 0.1, or 10%.

The third number specifies the threshold timeout ratio above which the server will be considered to be "bad" and will have its fetch quota lowered if it is above the minimum.  The default is 0.3, or 30%.

The fourth number specifies the weight given to the most recent counting period when averaging it with the previously held timeout ratio.  The default is 0.7, or 70%.

It would be unusual for there to be a genuine operational need to change the default fetch-quota-param settings and we do not recommend doing so without undertaking your own extensive testing of different scenarios that might be encountered by your resolver.

By design, this per-server quota should have little impact on lightly-used servers no matter how responsive (or not) they are, whilst heavily-used servers will have enough traffic to keep the moving average of their timeout ratio "fresh" even when they are deeply penalized for not responding.

Rate-limiting Fetches Per Zone

BIND already had an option that limits how many identical client queries (that cannot be answered directly from cache or authoritative zone data) it will accept.  When many clients simultaneously query for the same name and type, the clients will all be attached to the same fetch, up to the max-clients-per-query limit, and only one iterative query will be sent. This doesn't help however in the situation where client queries are for the same domain, but the hostname portion of the query is unique for each.

To help with this, we've introduced logic to rate-limit by zone instead.  This is configured using a new option fetches-per-zone which defines the maximum number of simultaneous iterative queries to any one domain that the server will permit before blocking new queries for data in or beneath that zone.  If fetches-per-zone is set to zero, then there is no limit on the number of fetches per query and no queries will be dropped.  Similar to fetches-per-server, fetches-per-zone also offers the choice of whether to drop or send back a SERVFAIL response when queries are being limited.

The default value for fetches-per-zone is 0, which disables this feature.  When fetches-per-zone is enabled, the default behaviour when rate-limiting is active is to drop queries that exceed the limit (this is not the same as the default for fetches-per server) .

When a fetch context is created to carry out an iterative query, it gets initialized with the closest known zone cut, and named adds both a cap (the value of which is configured by fetches-per-zone) on the number of fetches are allowed to be querying for that same zone cut at a time, and a counter for those that are currently outstanding (waiting for responses from authoritative servers).

Counters on fetches per zone don't persist!
The counters maintained on fetches per zone are reset when there are no outstanding fetches for that zone.  This is because the structure that was holding them doesn't persist once the last fetch for that zone has completed.  The periodic logging of the impact of fetches-per-zone on named's performance will therefore produce unreliable results for monitoring purposes - we recommend using the new counters added to BIND statistics instead.
Do not disable clients-per-query when enabling Recursive Client Rate Limiting
Recursive Client Rate Limiting is applied to fetches. If many clients make identical queries, they will generate between them a single fetch, but all of these clients are added to the list of recursive clients. If you disable clients-per-query then there will be no limit on the number of recursive clients being accepted for a single popular query. Fetch-limits are unlikely to provide much protection for your server resources in this situation. For more information see How does clients-per-query work?

Statistics and logging

rndc recursing now reports the list of current fetches, with statistics on how many are active, how many have been allowed, and how many have been dropped due to exceeding the fetches-per-server and fetches-per-zone quotas.

You can also monitor the BIND statistics - two new counters have been added:

  • ZoneQuota counts the number of client queries that are dropped or sent SERVFAIL due to the fetches-per-zone limit being reached.
  • ServerQuota counts the number of client queries that are dropped or sent SERVFAIL due to the fetches-per-server limit

Logging
Here are two examples of Recursive Client Rate limiting log messages, and how to interpret them.

05-Nov-2018 22:28:12.339 database: info: adb: quota 192.0.2.24 (1/2414): atr 0.99, quota decreased to 2414

05-Nov-2018 22:32:45.155 database: info: adb: quota 192.0.2.35 (1/667): atr 0.00, quota increased to 667

This first example (above) is emitted by the code that his handling fetches-per-server. Logging occurs only when the quota is increased or decreased.

The first log entry shows a server that doesn't receive many queries (there is only one onward fetch outstanding, against a newly-adjusted quota of 2414), but the Adjusted Timeout Rate (atr) is very high (.99, or 99% of all completed fetches were timeouts).

The second log entry shows a server that is responding (its atr is 0.00), therefore its quota is being increased. Unsurprisingly, since it is now responsive, it too only has one fetch outstanding.

05-Nov-2018 22:15:49.244 spill: info: too many simultaneous fetches for troubled-zone.example.com (allowed 200 spilled 1)

This second logging example is emitted by the code that is handling fetches-per-zone. Here is it is noting (per logging period) how many onward fetches for the zone troubled-zone.example.com were dropped.

Relying on logging for statistical purposes will produce inconsistent results and may not highlight all ongoing problems accurately

Recursive Client Rate limiting logging is useful as an indication that it is active during a time period, and to what extent client queries are being dropped, but BIND's statistics provide a much more accurate set of counters for graphing and statistics.

When applying fetches-per-zone, logging is emitted at intervals, but the logging of per-zone statistics may sporadically reset back to the original value (when the structure that was capturing the values is released.
Logging of per-server quota changes takes place only as the quota is adjusted. If there is no change to the current quota when it is tested at a calculation interval, nothing will be logged. This effectively hides from ongoing logging that a server has been penalised in the past and is continuing to be rate-limited.

Recursive Client Contexts Soft Quota

Strictly speaking, this is not part of the Recursive Client Rate-limiting functionality, but it was included with and tested at the same time as the other mitigation techniques we were developing, so is noted here for completeness.

In the traditional recursive clients context model, we have both a soft and a hard limit to the number of recursive clients.  When reached, the soft limit acts by dropping a pending request for each new incoming request that it starts to process.  When named reaches the hard limit, it drops both a pending request, and the new inbound client query.  So ideally we want named to be managing its backlog of recursive clients before reaching the hard limit - i.e., reaching a soft limit is the preferred mode of operation when under pressure.

In versions of BIND earlier than 9.9.8 and 9.10.3, there was no soft quota at all when recursive-clients <= 1000.  For recursive-clients > 1000, the soft quota defaulted to hard-quota minus 100.

This was a particular problem for DNS administrators who used the default recursive-clients (1000) because under high rates of client query traffic, it could happen that legitimate client queries that should be handled (and which would resolve quickly) were being rejected because there was only a hard quota.

From BIND 9.9.8 and 9.10.3 (now obsolete) and in all currently supported versions of BIND, when recursive-clients <= 1000 the soft quota is 90% of recursive-clients.  When
recursive-clients > 1000, the soft quota will the equal to the hard quota minus either 100 or the number of worker threads, whichever is greater.

SERVFAIL cache

SERVFAIL cache (also newly introduced in BIND 9.11.0) may help mitigate server loads where clients are repeated sending the same failing queries.

By default this is enabled at 1 seconds - equivalent to:

servfail-ttl 1;

You can disable the servfail cache by setting the ttl to zero.  The maximum is 30s but we do not recommend increasing the value beyond 1 or 2 seconds without operational testing as you may find that transient failures will be cached and persist for longer than you would like them to.  Most clients will retry a failed query and for a transient problem it is better that the query is retried versus the server responding with a second SERVFAIL from cache.

Please refer to the the Administrator Reference Manual for more details.

Other Resources

In addition to the information provided in this article, you may also be interested in:

What's Next

Cookie consent

We use our own and third-party cookies to understand how you interact with our Knowledgebase. This helps us show you more relevant content and ads based on your browsing and navigation history.