SOLVED SCCM - IIS Log files exceed 7Gb per day - full of 401 errors

[PILAWebber]

Hi,
Our SCCM server started showing that items deployed via software centre, were stuck at 0%. I have been troubleshooting this on and off for a few weeks but not really getting anywhere. SCCM is used mainly for OS deployment, but Software Centre is getting more use these days and has just become essential for customising Music dept PC's post OS deployment.
Today the server ran out of space on the C: drive. I found over 70gb of logs in IIS for the last week. Here's a sample line - from one of the logs that exceeded 7gb:
2024-10-06 22:39:11 172.16.x.yyy PROPFIND /CCMTOKENAUTH_SMS_DP_SMSPKG$/Content_a6da6f98-81bd-44ff-8613-c4110acaa6be.1 - 443 - 172.17.x.yyy SMS+CCM+5.0 - 401 1 0 72
I can browse to http://servername.domain.local/SMS_DP_SMSPKG$/DataLib/ from a client PC successfully - not not sure where to look next for a cause.
Does anyone have any suggestions to point me the right way??
Many thanks in advance.

 

[vshewale]

Hi,
Our SCCM server started showing that items deployed via software centre, were stuck at 0%. I have been troubleshooting this on and off for a few weeks but not really getting anywhere. SCCM is used mainly for OS deployment, but Software Centre is getting more use these days and has just become essential for customising Music dept PC's post OS deployment.
Today the server ran out of space on the C: drive. I found over 70gb of logs in IIS for the last week. Here's a sample line - from one of the logs that exceeded 7gb:
2024-10-06 22:39:11 172.16.x.yyy PROPFIND /CCMTOKENAUTH_SMS_DP_SMSPKG$/Content_a6da6f98-81bd-44ff-8613-c4110acaa6be.1 - 443 - 172.17.x.yyy SMS+CCM+5.0 - 401 1 0 72
I can browse to http://servername.domain.local/SMS_DP_SMSPKG$/DataLib/ from a client PC successfully - not not sure where to look next for a cause.
Does anyone have any suggestions to point me the right way??
Many thanks in advance.

This could happen due to incorrect DP configuration. Distribution point prerequisites not met or IIS permission are reset. Compare the permissions of SCCMContentLibrary folder with the one which is working file. Check antivirus or anti-malware running on server is not blocking the content.
Add and Grant Read Permissions to "Users" group on HKLM\SOFTWARE\Microsoft\SMS registry and rest IIS

Delete any log files over a week or 10 days. I have automated this for all our servers in our environment using simple task scheduler job that run every week on Sunday morning. Please make sure the account that run the task scheduler has rights on all the servers that you mentioned in clean up.

<#
.NOTES
===========================================================================
Created on: 10/18/2024
Created by: Vilas Shewale
Filename: Clean 10 days or older IIS logs remotely
===========================================================================
.DESCRIPTION
Clean up IIS logs remotely.
IISLogCleanupServerList.txt would look like this
ServerName-A
ServerName-B
..
ServerName-X
#>

#Folder Path for reading Server List
$FolderPath = <#FolderPath#>
$IISLogCleanupServers = Get-content $FolderPath + "\IISLogCleanupServerList.txt"

foreach ($IISLogCleanupServer in $IISLogCleanupServers)
{
if (Test-Connection -ComputerName $IISLogCleanupServer) #DP Online
{
#Write-Host $IISLogCleanupServer is online
$LogFolder = "\\" + $IISLogCleanupServer + "\C$\inetpub\logs\LogFiles"
Get-ChildItem $LogFolder -Recurse -Force | where { $_.LastWriteTime -lt (Get-Date).AddDays(-24) -and -not $_.psicontainer } | % { Remove-Item $_.fullname -Force}
#$LogFileList = Get-ChildItem $LogFolder -Recurse -Force | where { $_.LastWriteTime -lt (Get-Date).AddDays(-10) -and -not $_.psicontainer }
#Write-Host $LogFileList
}
else
{
#Write-Host $IISLogCleanupServer is offline
}
}

 

[PILAWebber]

This could happen due to incorrect DP configuration. Distribution point prerequisites not met or IIS permission are reset. Compare the permissions of SCCMContentLibrary folder with the one which is working file. Check antivirus or anti-malware running on server is not blocking the content.
Add and Grant Read Permissions to "Users" group on HKLM\SOFTWARE\Microsoft\SMS registry and rest IIS

Delete any log files over a week or 10 days. I have automated this for all our servers in our environment using simple task scheduler job that run every week on Sunday morning. Please make sure the account that run the task scheduler has rights on all the servers that you mentioned in clean up.

<#
.NOTES
===========================================================================
Created on: 10/18/2024
Created by: Vilas Shewale
Filename: Clean 10 days or older IIS logs remotely
===========================================================================
.DESCRIPTION
Clean up IIS logs remotely.
IISLogCleanupServerList.txt would look like this
ServerName-A
ServerName-B
..
ServerName-X
#>

#Folder Path for reading Server List
$FolderPath = <#FolderPath#>
$IISLogCleanupServers = Get-content $FolderPath + "\IISLogCleanupServerList.txt"

foreach ($IISLogCleanupServer in $IISLogCleanupServers)
{
if (Test-Connection -ComputerName $IISLogCleanupServer) #DP Online
{
#Write-Host $IISLogCleanupServer is online
$LogFolder = "\\" + $IISLogCleanupServer + "\C$\inetpub\logs\LogFiles"
Get-ChildItem $LogFolder -Recurse -Force | where { $_.LastWriteTime -lt (Get-Date).AddDays(-24) -and -not $_.psicontainer } | % { Remove-Item $_.fullname -Force}
#$LogFileList = Get-ChildItem $LogFolder -Recurse -Force | where { $_.LastWriteTime -lt (Get-Date).AddDays(-10) -and -not $_.psicontainer }
#Write-Host $LogFileList
}
else
{
#Write-Host $IISLogCleanupServer is offline
}
}

This could happen due to incorrect DP configuration. Distribution point prerequisites not met or IIS permission are reset. Compare the permissions of SCCMContentLibrary folder with the one which is working file. Check antivirus or anti-malware running on server is not blocking the content.
Add and Grant Read Permissions to "Users" group on HKLM\SOFTWARE\Microsoft\SMS registry and rest IIS

Delete any log files over a week or 10 days. I have automated this for all our servers in our environment using simple task scheduler job that run every week on Sunday morning. Please make sure the account that run the task scheduler has rights on all the servers that you mentioned in clean up.

<#
.NOTES
===========================================================================
Created on: 10/18/2024
Created by: Vilas Shewale
Filename: Clean 10 days or older IIS logs remotely
===========================================================================
.DESCRIPTION
Clean up IIS logs remotely.
IISLogCleanupServerList.txt would look like this
ServerName-A
ServerName-B
..
ServerName-X
#>

#Folder Path for reading Server List
$FolderPath = <#FolderPath#>
$IISLogCleanupServers = Get-content $FolderPath + "\IISLogCleanupServerList.txt"

foreach ($IISLogCleanupServer in $IISLogCleanupServers)
{
if (Test-Connection -ComputerName $IISLogCleanupServer) #DP Online
{
#Write-Host $IISLogCleanupServer is online
$LogFolder = "\\" + $IISLogCleanupServer + "\C$\inetpub\logs\LogFiles"
Get-ChildItem $LogFolder -Recurse -Force | where { $_.LastWriteTime -lt (Get-Date).AddDays(-24) -and -not $_.psicontainer } | % { Remove-Item $_.fullname -Force}
#$LogFileList = Get-ChildItem $LogFolder -Recurse -Force | where { $_.LastWriteTime -lt (Get-Date).AddDays(-10) -and -not $_.psicontainer }
#Write-Host $LogFileList
}
else
{
#Write-Host $IISLogCleanupServer is offline
}
}

Hi,
Thanks for getting back to me.
Our configuration is straightforward - a single SCCM server which has all the default roles (MP/DP/etc) as one installation.
I have been trying to dig a bit further into this and have noticed if I use IIS to browse the SMS_DP_SMSPKG$, then I can see many subfolders, but not the ones getting 401 errors.
I seem to be able to get software to deploy, by removing it from SCCM and re-adding it back. This will take weeks to do with more than 70 software titles. It did seem that going to a "Deployment type" and right clicking "Update Content" would help to get a deployment moving, but the 401 errors remain in IIS and compared to historic logs of approx 200 to 300mb I am still getting 7gb a day now!!.
It's odd - as everything in the OSD task sequence works - it's just software centre (which deploys software only used by specific departments).
I did check the registry setting and no change was needed.
I'll keep investigating...
Many thanks for your help,

 

[vshewale]

Hi,
Thanks for getting back to me.
Our configuration is straightforward - a single SCCM server which has all the default roles (MP/DP/etc) as one installation.
I have been trying to dig a bit further into this and have noticed if I use IIS to browse the SMS_DP_SMSPKG$, then I can see many subfolders, but not the ones getting 401 errors.
I seem to be able to get software to deploy, by removing it from SCCM and re-adding it back. This will take weeks to do with more than 70 software titles. It did seem that going to a "Deployment type" and right clicking "Update Content" would help to get a deployment moving, but the 401 errors remain in IIS and compared to historic logs of approx 200 to 300mb I am still getting 7gb a day now!!.
It's odd - as everything in the OSD task sequence works - it's just software centre (which deploys software only used by specific departments).
I did check the registry setting and no change was needed.
I'll keep investigating...
Many thanks for your help,

Anytime. Please keep us posted.

 

[PILAWebber]

Anytime. Please keep us posted.

It appears that in our case - the solution to this was:
In Software Library > Application Management > Applications
Click "Deployment Types" tab
Right click and "Update Content".
Whats a little frustrating is not knowing why the server got into the state it did - and not completely understanding why this appears to work.
Logs are now about 1 to 1.5gb a day which is much more manageable.
Now I have a new issue, but will create a new ticket for that one!!!
Thanks for your help.

 

[vshewale]

It appears that in our case - the solution to this was:
In Software Library > Application Management > Applications
Click "Deployment Types" tab
Right click and "Update Content".
Whats a little frustrating is not knowing why the server got into the state it did - and not completely understanding why this appears to work.
Logs are now about 1 to 1.5gb a day which is much more manageable.
Now I have a new issue, but will create a new ticket for that one!!!
Thanks for your help.

Update all applicable packages and application content or something specific? Please share the specifics.

 

[PILAWebber]

Update all applicable packages and application content or something specific? Please share the specifics.

I had to update all applications using the process above in each application subfolder in turn. That was the only way I could get Software Centre working again and it was far quicker than removing and re-adding each application.

 

[vshewale]

I had to update all applications using the process above in each application subfolder in turn. That was the only way I could get Software Centre working again and it was far quicker than removing and re-adding each application.

Thanks for sharing the information.