Automating SCCM patching process using ADRs

Let me start with a term that has often been referred by SCCM users & admins for a long time: ‘Best Practice‘. Personally, I feel there is no Best Practice for doing one thing in a certain way to achieve the end goal. With or without SCCM/MECM, one can attack the problem and navigate issues in so many different combinations.

For the purpose of writing this article on ADR, I am documenting a method that has worked for me in multiple enterprise design solutions.

As I work for a Healthcare system, the ADR solution is a little bit different from traditional design to ensure critical systems are not impacted by automated deployment & restart scenarios. It goes without saying that the process is not a representation of my organization’s policies but a general guideline on how to do things.

Let’s talk about deployment packages in SCCM, I would create four deployment packages to control them more granularly:

1. Windows 10 Microsoft office updates
2. Windows 10 windows update
3. Adobe Reader updates
4. Dell updates or any other supported third-party updates (For unsupported updates, I highly recommend PatchMyPC)

Each deployment package consists of distribution settings with medium distribution priority and content locations selecting all distribution points. You can choose the right configurations based on your SCCM layout. Also, one thing to keep in mind is to keep Adobe Reader ADR separate from other ADRs. One of the reasons to do so is because it helps prevent scheduling & content overlap with Microsoft’s Patch Tuesday.

ADR Planning

There are two school of thoughts when it comes to ADR planning :

1. Create New SUG every time an ADR runs
2. Add to existing SUG

The importance of SUG comes in to play as a SUG is used for grouping the software updates into one batch. For many organizations and SCCM architects have created SUG on a monthly basis which is a good practice but leads to a lot of overhead in terms of managing individual SUGs. The result is that you will end up with 12 months of SUG times two. One for Office & the other for Windows patch. Total: 24 SUGs, not to forget Adobe Reader, Dell Updates etc also need to be taken into account. I am in no way saying this is not good but it did not work for me while managing so many other stuff in SCCM. In a typical environment, you may want to spread the ADRs as Dev-QA-PROD or Lab-QA, PROD environments. This makes the total counts of SUGs 24×3= 72! on an annual basis. Too much work in my opinion. If you are able to track them, good for you!

However, adding to existing SUG is a much simpler option if you are mindful of not going beyond 2000 software updates per SUG. If you do go beyond 2000 software updates for SUG, you will need to clean it up. In my experience. for just window patches for windows 10 environment, a once an year cleanup is a must. This is a good practice to ensure that obsolete software updates are purged and also helps keep inventory clean. The total count here for SUG is just 2X3=6 which is reasonable amount of SUG to manage. I do use scripts to clean up the obsolete updates in SCCM’s SQL server every 6 months so it helps keep things in check.

Software Update Search criteria for Windows 10 & later versions:

ADR: Automatic Dancing Routine

This is where things get interesting. Reddit is filled with examples of how easy it is to time the ADR run to match up with Microsoft Patch Tuesday. There are also accounts of how hard it is to match up with a Patch-for-Patch Tuesday that often is released by Microsoft on Fridays! As mentioned earlier, there is no right or wrong approach. In fact, I have seen my management & security teams demand change in the policy that was set in stone couple of years ago. The art of getting this right is to find a balance between what Microsoft suggests & how your end user experience gets impacted. Strictly speaking in terms of Windows 10 Managed PC environment & not servers here. The server side of story is way more complex.

I call the ADR scheduling as a dance routine due to following three factors:

1. Microsoft’s recommendation to stay approx. 1 month behind for general patching.
2. Enterprise User experience aka business
3. Enterprise security compliance & directive

If one of the stakeholders demands an explanation, SCCM admins & architects alike are dancing LOL.

Assumptions

1. Change process approvals accounted for Monday, Tuesday, or Wednesday of each week *
2. Patch-For-Patch is the security updates re-released with modifications, mostly occurring on Fridays after Patch Tuesday. It could happen any time later but the general trend I have observed is the Fridays.

ADR Schedules & Target audience

I find the following three schedules that have worked for me in various companies where I worked as SCCM engineer:

1. DEV-Lab :
   • Schedule: Runs everyday. When Patch Tuesday occurs, the updates automatically deployed and systems rebooted per client settings.
   • Audience: Keep a good set of VDIs, Laptops, HyperVMs, PCs in this group to monitor impact.
2. QA:
   • Schedule: Runs every Third Wednesday at around 3 pm.
     • WHY Third Wednesday: As it helps me put a method to madness. Generally, it will come one week after the Patch Tuesday (which is second Tuesday of the month). If a patch for a patch is released on Patch Friday (if that’s a term you want to call), then I would not like to update systems over the weekend to prevent potential outages. Sometimes, we run into cycles where patch Tuesday may come after second Wednesday so it helps me overcome that problem too.
     • Example: For May 2024 calendar, I don’t need to tweak my design as Patch Tuesday occurs on 14th & third Wednesday occurs on 15th. DEV/LAB to QA movement account for In this unique situation, Third Wednesday falls just one day after patch Tuesday. Had the rule been set for second Wednesday, there would be no patching activity. Patch for Patch release can be individually monitored in this scenario without disrupting ADR ruleset.
   • Audience: These are technology savvy folks i.e. IT teams etc who are well aware of the patching schedule & what needs to be done. This group also consists of critical app users with their systems in QA to report any potential issue.
3. PROD:
   1. Schedule: 2-3 weeks after Patch Tuesday. Manually move patches from QA SUG to PROD SUG by using “Edit Membership” option. This goes in-line with change management process. I suggest always use manual process for patch movement to PROD as it gives a lot of control & fallback in case approvals are not granted due to reasons not limited to outages, application issues, network problem or simply lack of approvals.
   2. Audience: The whole village + distant hamlets.

DEV/LAB to QA movement of patches for regular patch cycle

Unique scenario: DEV/LAB to QA movement accounts for Third Wednesday which falls just one day after patch Tuesday. Had the rule been set for the second Wednesday, there would be no patching activity. Patch-for-Patch release can be individually monitored in this scenario without disrupting the ADR ruleset. This is once or twice per year scenario.

Classic CMG failed during conversion to VSS-Recovery steps

Recently I attempted to convert CMG classic to a Virtual Scale Set using the native SCCM conversion option. The conversion failed and I had to reach out to Microsoft to help and support.

Per Microsoft, I was not the only one facing this challenge. I am sharing the actions undertaken to recover from failed conversion:

Problem:  CMG went down after trying to upgrade from classic cloud to VSS to support the 2203 upgrade.

Resolution:

1. CMG showed error state and CMG connection point was disconnected in SCCM Console.

2. An attempt was made to remove the role from the site server and we reconfigured the CMG but it still failed as well. 

3. It was observed that the Classic cloud upgraded to VSS on SCCM side but failed to provision the resources in Azure as there were only classic cloud and storage in the resource group.

4. For recovery of CMG, we reverted the SCCM to classic cloud deployment using the following query:

Update azure_service set Fqdn = ‘<CMG Service Name>’ , DeploymentModel = ‘1’

6. CMG role was reinstalled again on the server to take the changes into effect and it worked after 5 minutes.