Dam Good Admin

Or at least not entirely useless

Servicing Stack Updates: What Is This Madness?

5/14/2019 Update: They did it AGAIN! In May 2019 the Cumulative Updates for Win 10 1607 (LTSC), Win 10 1803, Server 2016, and Server 2019 all require the Servicing Stack Updates released in the same month.

1/9/19 Update: The most recently released SSU (KB4486458) was given a distinctive title (2019-01 Servicing Stack Update for Windows 10 Version 1703). If Microsoft is reliable in this (to be determined) you could create an specific ADR for them and deploy ahead of your regular scheduled patch cycle. Still seems kinda janky to me though.

Recently there was a thread on Twitter discussing servicing stack updates and just how great and swell they are.  There was some slight confusion about the lay of the land so it’s time we had a talk.

First, what are these mysterious servicing stack updates and where do they come from?  Well, you see, when two operating systems love each other … oh wait, wrong talk.  It’s a gross over-generalization but the servicing stack is the set of components that update the operating system.  It’s not exactly the same as the Windows Update Agent but for now let’s equate the two.  So for all the same reasons everyone has told you to update the WUA you need to update the servicing stack.  If you want the full rundown, go read the docs since they’re actually really good.

Historically, like the WUA, the servicing stack isn’t updated all that often.  Why should it be?  It’s the kind of thing you’d think would be static.  There’s no super exciting feature that’s going to come via a new servicing stack.  And yet, for reasons I don’t fully understand, Microsoft has been handing these things out like candy recently.

It’s important to understand this part so it bears repeating.  The servicing stack update (SSU) patches the patcher.  It’s not fixing or securing the operating system itself.  That’s what Cumulative Updates (CUs) do.  It’s also why, I assume, it would be non-trivial to combine the two.  Patcher patch thyself!

Simple enough, what’s all the fuss about?

Because of their nature SSUs are separate from the CUs.  That’s not a problem of course, it just means you might have an additional update to apply apart from the CU on any given month.  The problem is, Microsoft had to go and make life difficult.

In May our beloved Patch Tuesday came and went on the 8th.  I’ve said it before and I’ll say it again: Patch Tuesday Is A Lie.  Microsoft proved that by releasing a servicing stack update (KB4132216) on the 17th.  In and of itself, that’s not a problem.  Most orgs have a monthly cadence so they just picked it up and deployed it in June.  What Microsoft did was make Mays’s post-Patch Tuesday SSU a pre-requisite for installing some of June’s CUs.  That, ladies and gentlemen, is pure bedlam. 

Bedlam You Say? Go On ….

See, the problem with what they did in May/June is that they made two patches that were effectively released in the same month depend on each other.  What that meant is that the CU was not applicable to the device until the SSU update is installed.  That doesn’t sound bad until you realize that it’s not applicable in the same way that an update for Windows 7 isn’t applicable to Server 2016 and therefore isn’t even offered for install.  You will not see the CU at all until the SSU is applied and, this is the crucial part here, a software update evaluation is ran.  

This creates a catch-22 situation: 

  • The CU isn’t applicable until the SSU is installed and a full scan is ran. 
  • A full scan won’t run outside of its normal schedule unless you reboot. 
  • The SSU doesn’t trigger a reboot. 

Even if you changed your client settings down to 24 hours it’s still going to take a day or so after the SSU installs for that to run and the CU to appear for install.  On devices that have maintenance windows that’s not gonna fly.

In Current Branch 1606 the product team added a deployment option to perform a full scan (evaluation) after the computer is rebooted to complete a software update installation.  That’s great but it doesn’t help here because SSUs do not require reboots.  Which is probably as it should be; I imagine the relevant files are not in-use most of the time.

Microsoft Tries a Non-Existent Middle Ground

Microsoft took a bunch of flack for doing what they did in June, and rightfully so.  They’ve done it one other time as well but it’s late and the wine says I’m too lazy to go look up the details.  The point is, they have released more SSUs since then but they have specifically not made that month’s CU dependent upon it.  Instead they’ve started adding the following note in the KB articles (ex. 4471324):

Microsoft strongly recommends you install the latest servicing stack update (SSU) for your operating system before installing the latest cumulative update (LCU)

See, that’s just kinda bullshit right there.  The CUs either require a specific SSU or they do not.  There is no room for ‘well … you don’t have to but it’d be great if you did anyways’.  Understand that pre-requisites are specified in the metadata of a given update.  That makes it a very binary kind of thing: the SSU is either a pre-requisite or it’s not a pre-requisite.  There is no try.

Also of note, Microsoft has started classifying the SSUs as Critical Updates. Though it stands to reason that there could be other non-SSU Critical Updates it’s unlikely in the new cumulative update world.

Awwwww Crap.  Now What?

Hopefully you now know why everyone is all atizzy about SSUs. There’s a bunch of admins trying to figure out how to orchestrate this. Before you get too worked up about it though ask yourself this first: am I seeing any actual impact of installing the SSU and CU in a random order? I have yet to see any hard evidence of this being the case. So the simplest solution to all of this is to simply not care and move along. Microsoft can recommend whatever they want but I wouldn’t change a single thing unless you actually experience installation issues solved by the latest SSU. That’s right, the whole thing is probably a tempest in a teapot so go back to living your life to the fullest.

But What If I Really Want to Care About This?

Well my dear completist … honestly … there’s really no good solution here.  There’s nothing currently in ConfigMgr that’s going to fix this problem for us.  The only suggestion I have is to deploy the SSUs ‘out of band’ ahead of the CUs.  To be clear though, that should only be required/necessary when MS releases a SSU and CU in the same month and they depend on each other.  Microsoft hasn’t been doing that so really, you shouldn’t need to do anything.  If you do though, the SSU is relatively safe.  It’s not touching the core OS itself and doesn’t require a reboot. Heck, go wild and suppress reboots on thesSSU-only deployment to guarantee no outages. Pilot and test it of course but it’s a pretty safe thing to roll out.

Note: In January 2019 Microsoft started releasing SSUs with distinct titles: ‘Servicing Stack Updates for…’. This allows you to create an ADR for just SSUs using a title filter and deploy them ahead of your normal roll-out. Just make sure that they deadline early enough to guarantee that a software update evaluation runs to detect the CU before its deadline.

Unacceptable.  Try Harder.  And With More Talent.

If you find that solution unacceptable … I don’t exactly blame you.  We should push for better solutions and ideas.  That’s what moves the product forward.

Part of what triggered this post was this User Voice item: Install Servicing Stack Updates Before Other Updates.  Although I agree in spirit I don’t think this is the right or likely solution. 

First, I have reason to be really familiar with the metadata used by Windows Update.  I’m not aware of any piece of metadata that would flag an update as a Servicing Stack Update.  Even the update titles are of no use here.  Here’s the latest cumulative update for Windows 10 1809: 2018-11 Update for Windows 10 Version 1809 for x64-based Systems (KB4470788). There’s nothing there to go on.  If the product team wanted to implement this functionality I’m not sure how they could do it properly without getting the Windows Update and possibly WSUS teams to update the metadata schema.

Second, SSUs are not the only kind of pre-requisite that exists.  Microsoft has and will likely release future updates with other kinds of pre-requisites.  When that happens whatever is cooked up for the UVI above won’t help.

Similar to the above, classifying the updates as Critical doesn’t really solve anything either. There’s no way to make sure Critical updates get installed ahead of others. You could create a separate ADR to deploy them ahead of other updates but that could quickly get complicated by maintenance windows and all sort of other scheduling shenanigans.

What I believe is the correct solution is that ConfigMgr triggers a full scan after all updates have attempted to install, including failed installs and regardless of reboot.  This would solve the whole pre-requisite problem regardless of what kind it is.  The scan would kick off, any newly applicable updates would appear and install based on deployment and remaining maintenance windows.  If that sounds about right to you go vote it up: Run Software Update Evaluation After Updates Have Installed.  If the Windows team made an SSU, or any update at all, a pre-requisite it just wouldn’t matter and everything would get installed in the correct order in a reasonable amount of time.

Note: It looks like the product team merged my UserVoice Item with the first one. The technical details were different and in this case important to get this right. When the CU requires the SSU then installing in order doesn’t matter … you’re only going to see the SSU anyways. We need to re-evaluate without a reboot to detect the CU after the SSU installs.

Welp … dammit … that went long as usual.  I really do like the sound of myself typing apparently.  Remember, keep your stick on the ice.  We’re all in this together.

8 Comments

  1. Do you know if the SSUs are cumulative? I ask because KB4456655 for Win10 1803 still shows as not superseded in my console, whereas I know it’s latest is KB4477137

    • Bryan Dam

      January 4, 2019 at 2:04 pm

      I’ve tried reaching out on Twitter to one of the members of the Windows Update team but haven’t heard back. It’s an interesting question. The current latest SSU (KB4477137) supersedes some but not all of the previous SSU (KB4456655). My intuition is that the SSUs are cumulative and that’s just an oversight in the metadata. I have an idea to test and will try and take a stab at it soon.

  2. Thinking out loud here… I’m pretty certain a workaround could be built until Microsoft fixes the whole issue. Create a PowerShell script that will check the WMI class to determine if any updates are still pending, if so, do nothing. If there are no updates pending for a full scan and optionally a deployment cycle. Add the script to a Scheduled Task running as SYSTEM that triggers on Event ID 2,4 for Source “Servicing”. Thus anytime a patch is installed the script would run and if no more patches were in queue to install a scan cycle would be kicked off. This is going around the normal process, but I can’t see it causing any harm. Thoughts?

    • Bryan Dam

      December 20, 2018 at 10:23 am

      That’s a super interesting idea. I always forget that you can trigger a scheduled task based on an event like that. That’s a super interesting idea. Main downside is it means doing something like that on every single endpoint. But you’re right, it’s not like it would hurt anything.

      The other scripting solution is to just wrap the SSU into an app or package and trigger the scan that way.

      • Wrapping the MSU and scan in a script would certainly work, but then you have to maintain and deploy both a Software Update and a Package/Application.

        … why am I so squeamish about brainstorming in public?

        • Yea, neither are particularly appealing to me.

          Protip: Lose all self-respect. I’ve found it to be rather liberating. Hard to recommend mind you, but it’s worked for me.

Leave a Reply

© 2019 Dam Good Admin

Theme by Anders NorenUp ↑