PENDING SCCM Crash caused server time change to a future date causing loads of problems

[Shaka420]

Hi there,

Our company has been dealing with a lot of weird SCCM issues since mid September and couldn't really pin point the cause of all the drama.
- Applications stopped appearing in Software Center on deployment.
- Client Operations were just expiring and not doing anything, summarization didn't work on this page either anymore
- Monitoring Deployments and Software Updates was not reporting anything anymore,
every new app remained 0% compliance even though I knew for a fact that applications had
installed themselves through SC, when they finally did appear in the application list, on some clients.
- Summarization only worked if you manually ran it, nothing is summarizing automatically anymore, or data isn't updated.

Finally, back from vacation last week I got around to troubleshooting the issue and came across an app crash that happened on the 10th
of September in the AM hours which we never noticed.

This smsexec.exe app crashed on our primary SCCM MP site and at the same time this triggered a time change on the server for about 20 minutes before it corrected itself. (We are still looking into how this could have happened). It was the only server in our domain that experienced a time change. A time change you say, what's the big deal?!
Wel this lovely crash on the server caused the time to change to a future date which am assuming is what brought on the parade of issues
we are experiencing now.

This is the date I am seeing in SCCM now on certain overview and status pages.





The icing on the cake, colleagues updated the servers to 2207 during my vacation not aware of the crash that had happened a few weeks before.
So now we also have no idea what other issues this might have brought on.
All components are green and deployments do still happen, but we have no control over when and if they appear in SC.
Machine Policy triggering and Evaluating Deployments seems to have no effect. Reinstalling the client agent does trigger an immediate install
of new application, but if I decide to deploy an extra application to that same client, it's a no go. Have to wait again till SCCM decides it wants
to push the application or reinstall the client agent again.

Would a Site reset correct this ? Is there a component I could trigger or PS script I could run to pin point where these values are coming from ?

 

[Garth]

A site reset can only help this situation. You might not be able to do anything about this until AFTER Jan 13 2023.

 

[Shaka420]

A site reset didn't help unfortunately.

Used CEViewer after reset to check evaluation cycles and All Systems is indeed still stuck with a pending full evaluation cycle to run next year. I managed to update evaluation dates on some collections to current date, just by changing the limiting collection and returning it to what it was before. This was enough to trigger an evaluation, but All Systems is a built-in collection and the root collection of most other collections.

Is there any way to force an evaluation trigger on the All Systems collection? This in turn would trigger on all children collections I assume. For built-in collections the checkmarks for incremental and full updates are greyed out so am hoping there is a way to manually queue up a full evaluation for All Systems.

Haven't been able to find a PowerShell script online so far for this purpose. Came across some scripts that change the RefreshType but guess they don't work with flagged 4106 collections. Invoking CMCollectionUpdate also doesn't seem to work. Starting to fear a manual evaluation is not possible for this collection. Guess that's what Garth meant with having to wait till after the 2023 date for anything to happen.

 

[Shaka420]

Issue resolved itself this morning...Triggers started running again and the backlog from my other post also resolved itself...
Message Processing Engine big backlog troubleshooting ? How ?