Firmware change management

Consider the following points when performing firmware upgrades, not only in FortiOS but as general rules for any change you have to make in a production environment.

Understanding the new version

Before attempting any changes in production, first make sure you set up a laboratory where you can freely play with the new features and understand them without pressure and time constraints. Read the release notes, manuals, and other documentation, such as presentations, videos, or podcasts about the new version.

You are ready to explain the need for an upgrade once you understand:

The differences and enhancements between the new version and the previous versions.
The impact of the upgrade on customers and the users of the operating platform.
The known limitations that might affect your environment.
The potential risks when performing the upgrade.
The licensing changes that may apply.

Never attempt to upgrade to a version that you do not fully understand, in terms of both features and known limitations, and on which you have no operational experience.

Reasons to upgrade

You should have a valid reason for upgrading the firmware. The reason cannot be only because you want the latest version. The reason has to be explained in terms of business, technical, or operational improvement.

Affirmative answers to the following questions are valid reasons to upgrade:

Does the new version have a feature that helps to ensure compliance?
Does the new version have an enhancement that allows a 40% decrease on the time it takes to perform a certain operation?
Does a new feature correct a known defect or bug found on a previous version that affects the company business or operations?
Will the new version allow your organization to deploy new services that will help to gain new customers or increase loyalty of existing customers?
Is the vendor cutting support for the version your organization is currently using?

If the best reason to upgrade is because the new features seem to be cool or because you want the latest version, some more understanding and planning may be necessary.

Preparing an upgrade plan

If you choose to upgrade for a valid reason, make sure you create a plan that covers business, technical, and operational aspects of the upgrade.

Business aspects of the upgrade

Proper planning and justification for an upgrade should be proportional to how critical the system is to the business.

Make sure you can clearly articulate the benefits of the upgrade in business terms, such as time, money, and efficiency.
Understand the business processes that will be affected by the change.
Make sure the upgrade maintenance window is not close to a business-critical process, such as quarterly or monthly business closure.
Obtain executive and operational approval for the maintenance window. The approval must come from the owners of all the system and information affected by the upgrade, not only from those that own the system being upgraded. The approval must be done formally through written statement or e-mail.

Technical and operational aspects of the upgrade

A plan must be created to account for technical and operational inputs.

Re-read the release notes for the technology that you are upgrading. Supported hardware models, upgrade paths, and known limitations should be clearly understood.
Make sure your upgrade maintenance window does not overlap with any other maintenance window on your infrastructure.
If you have any premium support offer, such as TAM Premium Support, do a capacity planning exercise to ensure the new firmware or software version does not take more hardware resources than you currently have.
Create a backup, whether or not you already have scheduled backups.
Obtain offline copies of both the currently installed firmware and the new version.
Create a list of systems with inter-dependencies to the system you are upgrading. For example, if you are upgrading a FortiGate, understand the impact on any FortiAP, FortiAuthenticator, FortiToken, FortiManager, or FortiAnalyzer you have on your environment.
If your FortiGate is part of a Security Fabric, understand any firmware dependencies between fabric devices and plan to upgrade fabric devices as necessary.
Ensure you have a list of adjacent devices to the upgrading platform and have administrative access to them, in case you need to do some troubleshooting. For example, if you are upgrading FortiWeb, make sure you can administratively access the web applications. Likewise, if you are upgrading FortiGate, make sure you can administratively access the surrounding switches and routers.
Have a step-by-step plan on how to perform and test the upgrade. You want to make sure you think of the worst situation before it happens, and have predefined courses of action, instead of thinking under pressure when something has already gone wrong.
Define a set of tests (that include critical business applications that should be working) to make sure the upgrade was successful. If any test does not go well, define which ones mandate a rollback and which ones can be tolerated for further troubleshooting. This set of tests should be run before and after the upgrade to compare results. The tests performed before and after the upgrade should be the same.
Define a clear rollback plan. If something goes wrong with the upgrade or the tests, the rollback plan will help you get your environment back to a known and operational status. The plan must clearly state the conditions under which the rollback will be started.
Declare configuration freezes shortly before and after the upgrade. This reduces the amount of variables to take into consideration if something goes wrong.
Perform a quality assurance upgrade. Load a copy of the production configuration on a non-production box and execute the upgrade to see if there are any issues on the process. Adjust your plan according to the results you obtain.
Have a list of information elements to be gathered if something goes wrong. This ensures that, even if the upgrade fails, you will collect enough information to troubleshoot the issue without needing to repeat the problem. Get help from Fortinet Support if you need to confirm what could be missing from your list.
Define a test monitoring period after the change was completed. Even if the upgrade went smoothly, something could still go wrong. Make sure that you monitor the upgraded system for at least one business cycle. Business cycles may be a week, month, or quarter depending on your organization's business priorities.

Executing the upgrade plan

Execution of an upgrade is just as key as planning. Once you are performing the upgrade, the pressure will rise and stress might peak. This is why you should stick to the plan you created with a cool head.

Resist the temptation to make decisions while performing the upgrade, as your judgment will be clouded by the stress of the moment, even if a new decision seems to be an obvious improvement in the moment. If your plan says you should rollback, then execute the rollback despite the potential quick fix mentality.

While performing the upgrade, make sure all the involved components are permanently monitored before, during, and after the upgrade, either through monitoring systems, SNMP alerts, or with a ping. Critical resources like CPU, memory, network, and disk utilization must also be constantly monitored.

To avoid misunderstandings, when performing the tests for each critical application defined in the planning, make sure there are formal notifications on the results for each user area, service, system, and application tested.

Regardless if you have to rollback or not, if a problem occurs, make sure you gather as much information about the problem as possible, so you can later place a support ticket to find a solution.

Finally, document the upgrade:

Enable your terminal emulation program to leave trace of all the commands executed and all the output generated. If you are performing steps through the GUI, consider using a video capture tool to document it.
Document any command or change performed over the adjacent and interdependent systems. Make sure they are acknowledged by the relevant administrators.
Document any deviations performed over the upgrade plan. This is the planned-versus-actual.

Learning more about change management

Change management and change control are huge knowledge areas in the fields of Information Systems, and Computer and Network Security.

This document is by no means a comprehensive list on what you should do when performing an upgrade, with either Fortinet or any other technology. It is merely a list of important things you should take into consideration when performing upgrades. It is the result of years of experience dealing with changes on critical environments, as it is common that security devices are protecting critical applications and processes.

There are vast resources on the topic of change management and change control, including books, public whitepapers, blog entries, and so on. If you search the internet for the "Change Control Best Practices" or "Change Management Best Practices," you will find many helpful results.

Changes on production IT infrastructure are critical to the business. Make sure they play in your favor and not against you.

For details on upgrading and downgrading your device firmware, see the Firmware & Registration section of the FortiOS Administration Guide.