Introduction
Operators should strive to implement general best practices for the servers hosting their critical infrastructure. Software is only as good as the platform it runs on.
System administration is well beyond the scope and remit of ISC Support. However, we can offer a very few suggestions, based on the most common problems we see.
Keep current
- Use an operating system which is still being maintained
- Keep current with security and stability updates (patches/fixes)
- ISC gets many reports for problems already fixed in newer releases
- Do not wait until the last minute to upgrade (for both OS and ISC software)
- Waiting creates unnecessary time pressure
- Any problems encountered at the last minute further compound the pressure
- Operating systems approaching end-of-life often have compatibility issues with newer software
Backups and history
- Make backups of all important files on a regular basis
- Keep multiple backups in rotation
- Keep some backups off-site/off-line
- Keep copies of old versions of config files for historical reference
Deployment
- Read the release notes, going back to the version you are upgrading from
- Test changes in a lab before deployment in production
- Have a plan to roll-back production changes if problems are discovered
- When practical, stagger changes across servers, so not everything changes at once
- Keep a record of when changes are made, by who, and why
- When practical, dedicate a server to each service (e.g., do not run a public web server and your DNS server on the same OS instance)
Monitoring
- Watch logs for errors, warnings, and unusual messages
- Do not ignore problem indications just because they are not critical (yet)
- Monitor key metrics for system resources
- Disk space - Full disks are a surprisingly common problem
- CPU consumption
- Network utilization
- Monitoring these over time can suggest when growth is exceeding current capacity
- Act before a shortage becomes exhaustion