SOLVED SCCM Hotfix KB14244456 - Notification Manager Errors

[mikey090tx]

Hello All,
I recently installed KB14244456 on our SCCM instance running ver 2203 and after the installation completed, I noticed an issue with our clients showing no an X (offline) status under the Icon column in the Devices view. We do have Client Activity showing "Active" but the icon indicator was working properly prior to the KB installation. I reviewed the component status under the Monitoring tab and noticed that the notification server indicated ID 4951. We do have .Net and the Configuration manager instance running on separate drives and attempted to run the fix provided by Prajwal via the edit of the two .net config files. This has not resolved the issue and we continue to experience the same problem with devices not showing their true status.

We run 1 management point with 4 other (desktop) DP's at a few sites. While reviewing the logs to identify the error, it appears that the issue is with disk space and also possibly with the original version of the msi file. I can confirm that there is enough disk space available on the drives, this is a false positive. Any help in identifying the root cause and a resolution would be appreciated.

Bgbisapimsilog-----------------------
Action start 8:51:39: CcmCheckFreeDiskSpace.
MSI (s) (D0:38) [08:51:39:761]: Product: BGB http proxy -- There is not enough available disk space on to complete this operation. Installation requires at least 10MB free disk space.

There is not enough available disk space on to complete this operation. Installation requires at least 10MB free disk space.
Action ended 8:51:39: CcmCheckFreeDiskSpace. Return value 3.
Action ended 8:51:39: INSTALL. Return value 3.

BgbSetup.log-------------------------
<Tue Jul 5 08:14:30 2022> Installing the SMSBGB
<Tue Jul 5 08:14:30 2022> Passed OS version check.
<Tue Jul 5 08:14:31 2022> IIS Service is installed.
<Tue Jul 5 08:14:31 2022> SMSBGB already installed (Product Code: {26131850-4466-4E51-9AF2-F504261B9349}). Upgrading/Reinstalling SMSBGB
<Tue Jul 5 08:14:31 2022> New SMSBGB is a new product code {{54B8832C-885C-465A-8FD1-4A7F01D57637}}. This is a major upgrade.
<Tue Jul 5 08:14:31 2022> Enabling MSI logging. bgbisapi.msi will log to D:\Program Files\Microsoft Configuration Manager\logs\bgbisapiMSI.log
<Tue Jul 5 08:14:31 2022> Installing D:\Program Files\Microsoft Configuration Manager\bin\x64\bgbisapi.msi REINSTALLMODE=vmaus CCMINSTALLDIR="D:\Program Files\SMS_CCM" CCMSERVERDATAROOT="D:\Program Files\Microsoft Configuration Manager" USESMSPORTS=TRUE SMSPORTS=80 USESMSSSLPORTS=TRUE SMSSSLPORTS=443 USESMSSSL=TRUE SMSSSLSTATE=0 CCMENABLELOGGING=TRUE CCMLOGLEVEL=1 CCMLOGMAXSIZE=1000000 CCMLOGMAXHISTORY=1
<Tue Jul 5 08:14:31 2022> bgbisapi.msi exited with return code: 1603
<Tue Jul 5 08:14:31 2022> Backing up D:\Program Files\Microsoft Configuration Manager\logs\bgbisapiMSI.log to D:\Program Files\Microsoft Configuration Manager\logs\bgbisapiMSI.log.LastError
<Tue Jul 5 08:14:31 2022> Fatal MSI Error - bgbisapi.msi could not be installed.
<Tue Jul 5 08:14:31 2022> ~RoleSetup().
<Tue Jul 5 08:51:16 2022> ====================================================================
<Tue Jul 5 08:51:16 2022> SMSBGB Setup Started....
<Tue Jul 5 08:51:16 2022> Parameters: D:\Program Files\Microsoft Configuration Manager\bin\x64\rolesetup.exe /deinstall /siteserver:SVRCOSCCM01 SMSBGB 0
<Tue Jul 5 08:51:16 2022> Deinstalling the SMSBGB
<Tue Jul 5 08:51:16 2022> CTool::RegisterComPlusService: run command line: "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\RegSvcs.exe" /u "D:\Program Files\Microsoft Configuration Manager\bin\x64\microsoft.configurationmanager.bgbserverchannel.dll"
<Tue Jul 5 08:51:20 2022> CTool::RegisterComPlusService: Failed to unregister D:\Program Files\Microsoft Configuration Manager\bin\x64\microsoft.configurationmanager.bgbserverchannel.dll with .Net Fx 4.0
<Tue Jul 5 08:51:20 2022> Failed to unregister BGB server channel DLL D:\Program Files\Microsoft Configuration Manager\bin\x64\microsoft.configurationmanager.bgbserverchannel.dll. Error = 0x80004005.
<Tue Jul 5 08:51:20 2022> CTool::RegisterComPlusService: run command line: "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\RegSvcs.exe" /u "D:\Program Files\Microsoft Configuration Manager\bin\x64\BGBServer\microsoft.configurationmanager.bgbserverchannel.dll"
<Tue Jul 5 08:51:23 2022> CTool::RegisterComPlusService: Failed to unregister D:\Program Files\Microsoft Configuration Manager\bin\x64\BGBServer\microsoft.configurationmanager.bgbserverchannel.dll with .Net Fx 4.0
<Tue Jul 5 08:51:23 2022> Failed to unregister BGB server channel DLL D:\Program Files\Microsoft Configuration Manager\bin\x64\BGBServer\microsoft.configurationmanager.bgbserverchannel.dll. Error = 0x80004005.
<Tue Jul 5 08:51:23 2022> DeleteBgbServerApplication: failed to find the application
<Tue Jul 5 08:51:23 2022> CTool::InstallManagedAssembly: run command line: "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\InstallUtil.exe" /u "D:\Program Files\Microsoft Configuration Manager\bin\x64\BGBServer\microsoft.configurationmanager.bgbserverchannel.dll"
<Tue Jul 5 08:51:24 2022> Successfully uninstall BGB server channel DLL D:\Program Files\Microsoft Configuration Manager\bin\x64\BGBServer\microsoft.configurationmanager.bgbserverchannel.dll
<Tue Jul 5 08:51:24 2022> Enabling MSI logging. bgbisapi.msi will log to D:\Program Files\Microsoft Configuration Manager\logs\bgbisapiMSI.log
<Tue Jul 5 08:51:24 2022> Deinstalling SMSBGB, with product code {26131850-4466-4E51-9AF2-F504261B9349}
<Tue Jul 5 08:51:24 2022> SMSBGB deinstall exited with return code: 1605
<Tue Jul 5 08:51:24 2022> Backing up D:\Program Files\Microsoft Configuration Manager\logs\bgbisapiMSI.log to D:\Program Files\Microsoft Configuration Manager\logs\bgbisapiMSI.log.LastError
<Tue Jul 5 08:51:24 2022> Product version is no longer installed. Returning success.
<Tue Jul 5 08:51:24 2022> Removing BGBHttp Registry.
<Tue Jul 5 08:51:24 2022> ~RoleSetup().
<Tue Jul 5 08:51:39 2022> ====================================================================
<Tue Jul 5 08:51:39 2022> SMSBGB Setup Started....
<Tue Jul 5 08:51:39 2022> Parameters: D:\Program Files\Microsoft Configuration Manager\bin\x64\rolesetup.exe /install /siteserver:SVRCOSCCM01 SMSBGB 0
<Tue Jul 5 08:51:39 2022> Installing Pre Reqs for SMSBGB

 

[Prajwal Desai]

Are the client agents updated to latest version 5.00.9078.1025?

 

[mikey090tx]

Are the client agents updated to latest version 5.00.9078.1025?

Prajwal- Yes, the majority of the clients are reporting the correct version and all of those are still showing an offline status as well as the others. I also do want to note that I did update config file as described in your other support doc and did pay close attention to the quotes to use the proper ones with the statement.

<runtime>
<loadFromRemoteSources enabled="true"/>
</runtime>

 

[mikey090tx]

I continued to dig through the logs and removed and re-added the MP role to the site and the issue continued. Part of the issue was due to a false positive error with the low disk space when installing the bgbisapi.msi, once I identified a work around for that problem I continued to see errors with removing the notificationserver dll file as it would error out and could not deregister. I ran the deregister command manually and it noted that it could not find the application id.
Today I decided to set the role on another system that was previously just used as a DP but can handle the workload etc and that install completed and now clients have started to report as online. We are running server 2012 on both of these instances and I will just plan to spin up a newer os server and set a new MP/CAS to migrate everything to and decomission the trouble server.