Each Stork release includes both major pieces of the Stork architecture: Stork server and Stork agent. This brings the problem of the order of deployed components to be upgraded. There is no one single correct approach, as there may be deployment specific factors involved. As such, this article discusses several strategies.
Overall, it is not recommended to mix different server and agent versions.
Tip: Before upgrading, check the Stork Known Issues to see if anything affects you.
Approach 1: Both at the same time (recommended)
The most straightforward approach is:
- Shut down both the Stork server, and all Stork agents
- Upgrade the Stork server
- Upgrade the Stork agents
- Start the Stork server
- Start the Stork agents
This approach eliminates any potential of running mismatched versions (old agents vs new server or new agents vs old server). The disadvantage is that the downtime is longest compared to other methods.
Approach 2: Agent first
the second best approach is to upgrade agents first: shut down agent, install new agent version, start agent. Repeat for all agents. Then shutdown the server, upgrade server, start server. Once completed, ensure that all agents are connected. If not, restart agents.
This approach has the benefit of shorter downtime, however the chances for temporary errors are increased. Not all agents and servers are compatible between releases. It is possible that during certain upgrades, the agents will refuse to connect to old server. Usually this is a temporary problem that will go away once the server is also upgraded.
Approach 3: Server first
The least preferred way is to upgrade the server first, while the old agents are still running. The chances are this approach will work well if there are no significant changes in the communication between versions. Due to known issues when upgrading from earlier versions to 1.15.1, this is not recommended. However, since the issues are temporary, there are no long term consequences and an easy to perform workaround is available, this approach can still be be used successfully if it's preferred for some reason.
Dealing with known issues during upgrade
Stork 2.6.0 certificate file paths
The directory and file names the Stork agent uses for TLS certificates was changed in 2.6.0, without handling existing installations. This results in the Stork agent failing to start after an upgrade. A workaround is to copy the files from the old location to the new, and rename the files in the new location.
| Old | New |
|---|---|
/var/lib/stork-agent/certs/ca.pem |
/var/lib/stork-agent/root-cert.pem |
/var/lib/stork-agent/certs/cert.pem |
/var/lib/stork-agent/server-cert.pem |
/var/lib/stork-agent/certs/key.pem |
/var/lib/stork-agent/server-key.pem |
/var/lib/stork-agent/tokens/agent-token.txt |
/var/lib/stork-agent/token.txt |
/var/lib/stork-agent/tokens/server-cert.sha256 |
/var/lib/stork-agent/server-cert.sha256 |
Upgrading from 1.15.0 to later versions
The 1.15.1 release introduced changes in the TLS certificates handling. If you upgrade the server first and let old agents connect, they will store an incorrect server certificate fingerprint (all zeros). Once you later upgrade agents to 1.15.1, they will attempt to verify server's certificate against that invalid all-zeros fingerprint and will refuse to connect as a result. The recovery for such a case is to delete server certificates on the agent:
sudo rm /var/lib/stork-agent/certs/ca.pem
sudo rm /var/lib/stork-agent/certs/cert.pem
sudo rm /var/lib/stork-agent/certs/key.pem
sudo rm /var/lib/stork-agent/tokens/server-cert.sha256
Then restart the agent. The agent will then proceed with a registration and will retrieve the server's certificates as part of the registration process. The agents will have to be authorized on the server using the Stork server web interface.
To avoid aforementioned problem, the recommended upgrade procedure is to upgrade both server and agent at the same time, or upgrade agents first.
Manual re-registration
If you followed the manual registration process for the Stork agent(s) as outlined in the Stork Quickstart guide when originally setting them up, then additional steps are required.
Re-register the stork agent as described in the aforementioned Stork Quickstart guide by running this command on the server that contains the stork-agent(s):
sudo su stork-agent -s /bin/sh -c 'stork-agent register --server-url http://stork.example.org:8080'
Be sure and replace the stork.example.org:8080 with the correct URL of your Stork server. Now restart your Stork agent(s). Finally, you should find the Stork agent(s) in "Unauthorized" under Services -> Machines in the Stork server UI. Authorize these machines and they should start working again.