Servicing Stack Updates: What Is This Madness?

5/14/2019 Update: They did it AGAIN! In May 2019 the Cumulative Updates for Win 10 1607 (LTSC), Win 10 1803, Server 2016, and Server 2019 all require the Servicing Stack Updates released in the same month.

1/9/19 Update: The most recently released SSU (KB4486458) was given a distinctive title (2019-01 Servicing Stack Update for Windows 10 Version 1703). If Microsoft is reliable in this (to be determined) you could create an specific ADR for them and deploy ahead of your regular scheduled patch cycle. Still seems kinda janky to me though.

Recently there was a thread on Twitter discussing servicing stack updates and just how great and swell they are.  There was some slight confusion about the lay of the land so it’s time we had a talk.

First, what the heck is the servicing stack?  The servicing stack is the set of components that update the operating system.  There are other components but the two most recognizable ones are the Windows Update Agent (WUA) and Component Based Servicing (CBS).   If you want the full rundown, go read the docs since they’re actually really good. For the purposes of this discussion it’s the thing that installs updates.

What then is a servicing stack update (SSU)? Well, you see, when two operating systems love each other … oh wait, wrong talk. The SSU is an update to the set of components that install updates. It’s an update for the update mechanism. That update mechanism is just one part of the overall operating system. SSUs aren’t really some special snowflake that you need to think of drastically differently. I know what you’re thinking, ‘then why the heck am I reading this?’. Wait for it, we’ll get there. If it helps to get your mind around it just equate the servicing stack to the WUA. That makes SSUs simply new versions of the WUA that you were familiar with on Windows 7. Technically they are not the exact same thing but from a practical point of view they are.

Historically, like the WUA, the servicing stack isn’t updated all that often.  Why should it be?  It’s the kind of thing you’d think would be static.  There’s no super exciting feature that’s going to come via a new servicing stack.  And yet, for reasons I don’t fully understand, Microsoft has been handing these things out like candy recently.

It’s important to understand this part so it bears repeating.  The servicing stack update (SSU) patches the patcher.  It’s not fixing or securing the operating system itself.  That’s what Cumulative Updates (CUs) do.  It’s also why, I assume, it would be non-trivial to combine the two.  Patcher patch thyself!

Simple enough, what’s all the fuss about?

Because of their nature SSUs are separate from the CUs.  That’s not a problem of course, it just means you might have an additional update to apply apart from the CU on any given month.  The problem is, Microsoft had to go and make life difficult.

In May our beloved Patch Tuesday came and went on the 8th.  I’ve said it before and I’ll say it again: Patch Tuesday Is A Lie.  Microsoft proved that by releasing a servicing stack update (KB4132216) on the 17th.  In and of itself, that’s not a problem.  Most orgs have a monthly cadence so they just picked it up and deployed it in June.  What Microsoft did was make May’s post-Patch Tuesday SSU a pre-requisite for installing some of June’s CUs.  That, ladies and gentlemen, is pure bedlam. 

Bedlam You Say? Go On ….

See, the problem with what they did in May/June is that they made two patches that were effectively released in the same month depend on each other.  What that meant is that the CU was not applicable to the device until the SSU update is installed.  That doesn’t sound bad until you realize that it’s not applicable in the same way that an update for Windows 7 isn’t applicable to Server 2016 and therefore isn’t even offered for install.  You will not see the CU at all until the SSU is applied and, this is the crucial part here, a software update evaluation is ran.  

This creates a catch-22 situation: 

• The CU isn’t applicable until the SSU is installed and a full scan is ran. 
• A full scan won’t run outside of its normal schedule unless you reboot. 
• The SSU doesn’t trigger a reboot. 

Even if you changed your client settings down to 24 hours it’s still going to take a day or so after the SSU installs for that to run and the CU to appear for install.  On devices that have maintenance windows that’s not gonna fly.

In Current Branch 1606 the product team added a deployment option to perform a full scan (evaluation) after the computer is rebooted to complete a software update installation.  That’s great but it doesn’t help here because SSUs do not require reboots.  Which is probably as it should be; I imagine the relevant files are not in-use most of the time.

Microsoft Tries a Non-Existent Middle Ground

Microsoft took a bunch of flack for doing what they did in June, and rightfully so.  They’ve done it one other time as well but it’s late and the wine says I’m too lazy to go look up the details.  The point is, they have released more SSUs since then but they have specifically not made that month’s CU dependent upon it.  Instead they’ve started adding the following note in the KB articles (ex. 4471324):

Microsoft strongly recommends you install the latest servicing stack update (SSU) for your operating system before installing the latest cumulative update (LCU)

See, that’s just kinda bullshit right there.  The CUs either require a specific SSU or they do not.  There is no room for ‘well … you don’t have to but it’d be great if you did anyways’.  Understand that pre-requisites are specified in the metadata of a given update.  That makes it a very binary kind of thing: the SSU is either a pre-requisite or it’s not a pre-requisite.  There is no try.

Also of note, Microsoft has started classifying the SSUs as Critical Updates. Though it stands to reason that there could be other non-SSU Critical Updates it’s unlikely in the new cumulative update world.

Awwwww Crap.  Now What?

Hopefully you now know why everyone is all atizzy about SSUs. There’s a bunch of admins trying to figure out how to orchestrate this. Before you get too worked up about it though ask yourself this first: am I seeing any actual impact of installing the SSU and CU in a random order? I have yet to see any hard evidence of this being the case. So the simplest solution to all of this is to simply not care and move along. Microsoft can recommend whatever they want but I wouldn’t change a single thing unless you actually experience installation issues solved by the latest SSU. That’s right, the whole thing is probably a tempest in a teapot so go back to living your life to the fullest.

But What If I Really Want to Care About This?

Well my dear completist … honestly … there’s really no good solution here.  There’s nothing currently in ConfigMgr that’s going to fix this problem for us.  The only suggestion I have is to deploy the SSUs ‘out of band’ ahead of the CUs.  To be clear though, that should only be required/necessary when MS releases a SSU and CU in the same month and they depend on each other.  Microsoft hasn’t been doing that so really, you shouldn’t need to do anything.  If you do though, the SSU is relatively safe.  It’s not touching the core OS itself and doesn’t require a reboot. Heck, go wild and suppress reboots on thesSSU-only deployment to guarantee no outages. Pilot and test it of course but it’s a pretty safe thing to roll out.

Note: In January 2019 Microsoft started releasing SSUs with distinct titles: ‘Servicing Stack Updates for…’. This allows you to create an ADR for just SSUs using a title filter and deploy them ahead of your normal roll-out. Just make sure that they deadline early enough to guarantee that a software update evaluation runs to detect the CU before its deadline.

Unacceptable.  Try Harder.  And With More Talent.

If you find that solution unacceptable … I don’t exactly blame you.  We should push for better solutions and ideas.  That’s what moves the product forward.

Part of what triggered this post was this User Voice item: Install Servicing Stack Updates Before Other Updates.  Although I agree in spirit I don’t think this is the right or likely solution. 

First, I have reason to be really familiar with the metadata used by Windows Update.  I’m not aware of any piece of metadata that would flag an update as a Servicing Stack Update.  Even the update titles are of no use here.  Here’s the latest cumulative update for Windows 10 1809: 2018-11 Update for Windows 10 Version 1809 for x64-based Systems (KB4470788). There’s nothing there to go on.  If the product team wanted to implement this functionality I’m not sure how they could do it properly without getting the Windows Update and possibly WSUS teams to update the metadata schema.

Second, SSUs are not the only kind of pre-requisite that exists.  Microsoft has and will likely release future updates with other kinds of pre-requisites.  When that happens whatever is cooked up for the UVI above won’t help.

Similar to the above, classifying the updates as Critical doesn’t really solve anything either. There’s no way to make sure Critical updates get installed ahead of others. You could create a separate ADR to deploy them ahead of other updates but that could quickly get complicated by maintenance windows and all sort of other scheduling shenanigans.

What I believe is the correct solution is that ConfigMgr triggers a full scan after all updates have attempted to install, including failed installs and regardless of reboot.  This would solve the whole pre-requisite problem regardless of what kind it is.  The scan would kick off, any newly applicable updates would appear and install based on deployment and remaining maintenance windows.  If that sounds about right to you go vote it up: Run Software Update Evaluation After Updates Have Installed.  If the Windows team made an SSU, or any update at all, a pre-requisite it just wouldn’t matter and everything would get installed in the correct order in a reasonable amount of time.

Note: It looks like the product team merged my UserVoice Item with the first one. The technical details were different and in this case important to get this right. When the CU requires the SSU then installing in order doesn’t matter … you’re only going to see the SSU anyways. We need to re-evaluate without a reboot to detect the CU after the SSU installs.

Unified Update Platform: The Patching Boogeyman

I’ve heard it said now by multiple program managers that the Unified Update Platform announced back in 2016 will fix all of this. Apart from a follow up post in 2017 I can’t really find any technical details about what the UUP is. It seems to use binary deltas to limit the amount of data the client needs to download. I can’t find any mention of it being the holy grail for SSUs or really any meaningful deep-dive into UUP at all. If you know of anything, I’d love to hear it.

Welp … dammit … that went long as usual.  I really do like the sound of myself typing apparently.  Remember, keep your stick on the ice.  We’re all in this together.