PENDING MP has rejected a message... because the signature could not be validated. How do I resolve this?

[MicrosoftCloudRegKey]

We upgraded our SCCM server from 2211 to 2309 and now 2 of our servers are getting that message:

MP has rejected a message... because the signature could not be validated.

The servers go offline in SCCM because MP is rejecting the message. However what I have noticed is that the servers do come online maybe 10~ minutes later and they will accept requests / successfully deliver messages, etc. And maybe 10-20 minutes later go offline and repeat the above.

I looked at the logs on the machine itself and the only error I can find specifically is in the CCMNotificationAgent log:

[CCMHTTP] ERROR: URL=https://... url to our server TEXT=CCM_E_BAD_HTTP_STATUS_CODE
[CCMHTTP] ERROR INFO: StatusCode=990 StatusText=BGB Session Ended

These appear in the logs multiple times during the time the MP rejects the message, and after 10 minutes, everything is working again and the CCMNotificationAgentlogs show that it was able to connect and no issues..

I restarted CccmExec on each server and verified the PKI cert it is pointing to in the logs is the correct cert. Which makes sense because these machines CAN communicate on and off and if the cert itself was bad, I imagine there would be zero communication.

This happened right after the upgrade of the server, no changes on these machines itself were done... thoughts?

 

[MicrosoftCloudRegKey]

I noticed on the SCCM server itself if I look at the bgbserver.log I see...
failed to decode message body with message header...
error: the message timestamp is older or newer than 1 hour
error: the message body is invalid

the server has the same time as the SCCM server, so I'm not sure how the message is older or newer than 1 hour

 

[Garth]

I noticed on the SCCM server itself if I look at the bgbserver.log I see...
failed to decode message body with message header...
error: the message timestamp is older or newer than 1 hour
error: the message body is invalid

the server has the same time as the SCCM server, so I'm not sure how the message is older or newer than 1 hour

It sound like you mp is unhealth. Try reinstalling it. Ultimately you might need to contact ms support for help.

 

[MicrosoftCloudRegKey]

Would you say the MP is unhealthy if it's only doing this to 2 servers and every other endpoint / server is completely fine and doesn't have issues?
Another thing I noticed is that for these 2 servers having issues. the Heartbeat DDR does indeed show that they are several hours in the future. For example it could be 6AM but the Heartbeat DDR will say 10AM. But if I look at the logs or go on the server itself, there isn't any jump in time anywhere so I'm not sure why the heartbeat DDR is the only thing thinking the two servers are in the future.

I did contact MS Support but the response I got was that SCCM support isn't free and that we would need to open a pay per incident ticket. Is that normal?

 

[Garth]

Is their time zone 4 hours ahead of the MP? What does Offline mean? What network troubleshooting have you done? Yes that is normal that MS support has a cost. if you company is not paying for support then I would start looking for a new job elsewhere and it means they don't believe that IT is valuable to the bottom line.