Upgrading Stork

Each Stork release includes both major pieces of the Stork architecture: Stork server and Stork agent. This brings the problem of the order of deployed components to be upgraded. There is no one single correct approach, as there may be deployment specific factors involved. As such, this article discusses several strategies.

Overall, it is not recommended to mix different server and agent versions.

Approach 1: Both at the same time (recommended)

The most straighforward approach is to shut down both server and agents, upgrade the server, upgrade the agents, start server, start agents.

This approach eliminates any potential of running mismatched versions (old agents vs new server or new agents vs old server). The disadvantage is that the downtime is longest compared to other methods.

Approach 2: Agent first

Second best approach is to upgrade agents first: shut down agent, install new agent version, start agent. Repeat for all agents. Then shutdown the server, upgrade server, start server. Once completed, ensure that all agents are connected. If not, restart agents.

This approach has the benefit of shorter downtime, however the chances for temporary errors are increased. Not all agents and servers are compatible between releases. It is possible that during certain upgrades, the agents will refuse to connect to old server. Usually this is a temporary problem that will go away once the server is also upgraded.

Approach 3: Server first

The least preferred way is to upgrade the server first, while the old agents are still running. The chances are this approach will work well if there are no significant changes in the communication between versions. Due to known issues when upgrading from earlier versions to 1.15.1, this is not recommended. However, since the issues are temporary, there are no long term consequences and an easy to perform workaround is available, this approach can still be be used successfully if it's preferred for some reason.

Dealing with known issues during upgrade

Upgrading from 1.15.0 to later versions

The 1.15.1 release introduced changes in the TLS certificates handling. If you upgrade server first and let old agents connect, they will store incorrect server certificate fingerprint (all zeros). Once you later upgrade agents to 1.15.1, they will attempt to verify server's certificate againts invalid all zeros fingerprint and will refuse to connect as a result. The recovery for such a case is to delete server certificates on the agent:

sudo rm /var/lib/stork-agent/certs/ca.pem
sudo rm /var/lib/stork-agent/certs/cert.pem
sudo rm /var/lib/stork-agent/certs/key.pem
sudo rm /var/lib/stork-agent/tokens/server-cert.sha256

And restart the agent. The agent will then proceed with a registration and will retrieve the server's certificates as part of the registration process. The agents will have to be authorized on the server using the Stork server web interface.

To avoid aforementioned problem, the recommended upgrade procedure is to upgrade both server and agent at the same time or upgrade agents first.

sudo su stork-agent -s /bin/sh -c 'stork-agent register --server-url http://stork.example.org:8080'