Or at least not entirely useless

Maintenance Windows: The Never Never Land of Patching

Maintenance Windows are a concept that can be tricky to fully grasp at first. I’m too lazy to even try to help you there so if you need some ‘splainin then I suggest you first head over to the docs and Jason Sandy’s excellent rundown.

Before we get into it though I want to quickly rant about one thing first. Maintenance windows can only decrease your patch compliance. Use them because there’s a real life and/or business threatening reason to do so. If you just need to make sure that patches/applications/task sequences install at a specific date or time … that’s what deployment deadlines are for.

Ok, off the soap-box. There’s two requirements I’ve seen frequently asked about that I use maintenance windows to solve. This week we’re just going to tackle the first one:

I Never, Ever, Want You To Patch This Box

If you’ve followed along with my blog or talked with me for even the briefest of time you will recognize how painful that heading was to type. It burns. Patching is too important to leave to us meat-bags and should be automated as much as possible. However, there are certain situations where a legitimate reason exists not to automate patching and rebooting (which are the same thing). Those situations tend to be on the server side of things where workloads need to be moved around (ex. SQL, Exchange) but there are some workstation use cases as well. If you have workstations in an operating room you get a pass on automatically updating those. At least while I’m on the table please. Your Windows 2000 box running a nuclear reactor is probably fine too. Please don’t power cycle that thing.

There’s a variety of solutions to this problem but to be perfectly honest the only good one doesn’t exist: Support Available Deployments for ADRs. Until then the solution I like best and in my opinion is the most reliable is to create what I call a ‘Never’ maintenance window. Create a non-recurring maintenance window that occurred in the past. It should end up looking something like this:

This works because once a device has a single maintenance window it will not act outside of it. If that single maintenance window is in the past and doesn’t repeat itself then nothing will ever automatically install on that device again. Instead, when a deployment’s deadline hits it will try to install, detect that it is currently not within a maintenance window, and permanently report itself as ‘Past Due’ waiting for a maintenance window that will never come. Poor little updates … so lonely. They will forever stay that way until someone opens Software Center and manually initiates the install. If you wanted to get all fancy-shmancy you could trigger the manual installs remotely using a variety of techniques including <full shill mode> something like the Right Click Tool’s Install Missing Software Updates tool </full shill mode>.

There’s two assumptions being made here. First: that you don’t deploy something that overrides maintenance windows. There’s no fixing stupid so don’t be stupid. Second: that you do not apply another maintenance window to the devices in the Never collection. This one is a little trickier and is part of why it is almost universally agreed upon that you should create separate collections that exist only for maintenance window purposes. Further, I highly recommend making it easy to identify your maintenance windows by putting them in a folder and/or prefixing the collections with something meaningful. For example: ‘MW – Never’. This way you know where your maintenance windows are and you can make sure that they all exclude your Never maintenance window. If you make sure that every maintenance window collection excludes your Never maintenance window then simply adding devices to your Never maintenance window guarantees that you do not cross the streams:

A Maintenance Window With Benefits!

There’s some fringe benefits of using a Never maintenance window that you don’t get trying to solve this problem other ways.

First, using a Never maintenance window is a great way to ease into fully automated patching. Have some team members that believe their devices are special little snowflakes instead of cattle ready for slaughter? Put their devices in a Never maintenance window and deploy patches to them normally. You get control and reporting while the end user gets to avoid having to wait for Windows Update scans and updates to download. My server admins loved the fact that they could open Software Center during the day to validate what patches were waiting to be applied before they logged in during the evening to apply them. They also loved the ability to script the process so it could be done manually but without having to log into each machine. Crucially, after a while many asked themselves why they were getting up at ass o’clock to push a single button. It made automated patching converts for the good of the computing race.

Second, your Never maintenance window is an agreed upon black-list of devices that are dead to you. My reporting dashboard was written specifically to report on servers globally, servers with active maintenance windows, and servers with the Never maintenance window. This allowed me to show management how awful some of my fellow admins were at manually patching their devices while also excluding them from the compliance numbers I cared about. When security came knocking about a particular box step one was to review my Never maintenance window membership for the device(s) in question. If they were there then I happily told them to take it up with the app owners who held on to manual patching their own devices. This too helped create automated patching converts … albeit less willing ones.

Lastly, related to the above, the Never maintenance window was an outlet for for dealing with cumulative updates that impacted only a small subset of applications. With cumulative updates, just blocking this month’s patch means they’ll just break again when next month’s patches come out unless the underlying problem is resolved. If an application owner told me to not patch their box I directed them to our CIO and required the CIO’s written approval to place their boxes in the Never maintenance window and … this is the crucial part … never patch those boxes again. Roughly 50% of cases never got past that part and the ones that did were no longer my problem because they were excluded from the reporting that matters to me.

So there you have it. As I said, there’s more than one way to address this requirement but for the reasons above the concept of a Never maintenance window is one I constantly fall back on as the best solution until the product team gives us available deployments in ADRs. Though even then, if you’re concerned about applications and task sequences accidentally being deployed as required instead of available then only a Never maintenance window can guarantee that.

7 Comments

  1. Ian

    So, how does one control for updates that are past a deadline but did not install for some reason. For example, a server with a non-functional SCCM client that got fixed and suddenly had a bunch of updates to do as they are now post-deadline. Without a maintenance window, the server in question may get updated and rebooted immediately.
    This is the scenario that happened to us recently, resulting in an uncontrolled reboot during the business day.

    I get the idea that a deadline is all that is needed to set typical updating schedule, but I am still at a loss of how to deal with the not so typical if it is not a maintenance window.

    • Bryan Dam

      Right, which is why if you have servers that provide mission critical services they should have maintenance windows to control when they be offline. Those would be the ‘business threatening’ kind of thing I talked about in the preamble.

  2. Paul

    I’ve been frustrated that you can’t make ADRs with Available deployments, and it seems like having a non-recurring maintenance window in the past can be an excellent solution to that problem! I never would of thought of that, thanks for the great tip!!

  3. Ken

    I’ve used this technique since SMS 2007 was released, and it works very well, but isn’t perfect.

    I was always irritated that there was no good method to have a recurring maintenance window that would be something like, From the 2nd Tuesday +/- X number of days, or the 1st Saturday after the 15th of the month. We have a “floating” maintenance window that is different each month because it’s based on the weekend following the 15th of the month for test systems, then two weeks later for production. The Gregorian calendar makes automating that impossible. We also have other app owners who want test servers on the Xth Thursday, and then production the next day, which again, isn’t possible to automate. Maybe some day!

    Side question for the masses: I’ve been looking for a good way to use PowerShell to report a month’s worth of devices’ maintenance windows, and in fact, I’ve not found a good PowerShell method to calculate the “next maintenance window” of a device that isn’t broken when it crosses into the next month. I haven’t found any existing code to point me in the right direction. Any suggestions? I really don’t want to reinvent the wheel if there’s available code I just haven’t seen.

  4. david zemdegs

    I though maintenance windows are used to prevent servers restarting in the middle of the day? We have a 2-6am maint window on all servers for that reason.

Leave a Reply

© 2024 Dam Good Admin

Theme by Anders NorenUp ↑