How can I protect my DHCP server from a badly behaved spamming client?

ISC DHCP

There's no rate-limiting built into the DHCP server, but you may be able to use iptables or other software or middleware solutions to provide that kind of protection for your server.

However, in dhcpd.conf you can block a client by using deny booting; or ignore booting;.

For example:

class "badguys" {
    match hardware;
    ignore booting;
}

subclass "badguys" 1:00:01:02:03:04:05;

host "bad1" {
    hardware ethernet 00:01:02:03:04:05;
    ignore booting;
}

What is the difference between deny and ignore?

The difference between deny and ignore is that deny logs a message; ignore is silent.

Kea

As of March 2017, Kea allows selective client drops by using hooks mechanism (see pkt4_receive and pkt6_receive hook points).