SOLVED Unable to distribute content to main site DP Expand failed 0x80070003

[AnScott]

System: Windows Server 2008 R2 64 Bit
SCCM: 1511

A few weeks ago, we had an issue with the storage array that hold the SCCM content and DP for our main site. After getting the array back up and running with the files recovered, we were unable to do any imaging. After looking at the logs, I decided to completely remove the DP and then set it back up. I did the following:

• Deleted content off the DP
• Remove the DP in SCCM console
• Uninstalled WDS service on server
• Place a no_sms_on_drive.sms on a secondary drive that previously contained some DP files in the server so that all DP files were on one drive
• Restarted Server
• Created the DP in SCCM console

When trying to then distribute our two boot images to the DP, they fail. Below is the relevant log entries form distmgr.log:

7/7/2017 9:57:37 AM Started package processing thread for package 'IVC00002', thread ID = 0x369C (13980)
7/7/2017 9:57:37 AM Sleep 3600 seconds...
7/7/2017 9:57:37 AM STATMSG: ID=2300 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=IVCCD-SCCM-01.ivd.iavalley.cc.ia.us SITE=IVC PID=24008 TID=13980 GMTDATE=Fri Jul 07 14:57:37.582 2017 ISTR0="Boot image (x64)" ISTR1="IVC00002" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 AID0=400 AVAL0="IVC00002"
7/7/2017 9:57:37 AM No action specified for the package IVC00002, however there may be package server changes for this package.
7/7/2017 9:57:37 AM Start adding package IVC00002 to server ["Display=\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\"]MSWNET:["SMS_SITE=IVC"]\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\...
7/7/2017 9:57:37 AM Created DP processing thread 23980 for addition or update of package IVC00002 on server ["Display=\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\"]MSWNET:["SMS_SITE=IVC"]\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\
7/7/2017 9:57:37 AM Waiting for all DP threads to complete for package IVC00002 processing thread.
7/7/2017 9:57:37 AM DP Thread: Attempting to add or update package IVC00002 on DP ["Display=\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\"]MSWNET:["SMS_SITE=IVC"]\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\
7/7/2017 9:57:37 AM The distribution point is on the siteserver and the package is a content type package. There is nothing to be copied over.
7/7/2017 9:57:37 AM STATMSG: ID=2342 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=IVCCD-SCCM-01.ivd.iavalley.cc.ia.us SITE=IVC PID=24008 TID=23980 GMTDATE=Fri Jul 07 14:57:37.756 2017 ISTR0="Boot image (x64)" ISTR1="["Display=\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\"]MSWNET:["SMS_SITE=IVC"]\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=2 AID0=400 AVAL0="IVC00002" AID1=404 AVAL1="["Display=\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\"]MSWNET:["SMS_SITE=IVC"]\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\"
7/7/2017 9:57:38 AM The current user context will be used for connecting to ["Display=\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\"]MSWNET:["SMS_SITE=IVC"]\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\.
7/7/2017 9:57:38 AM No network connection is needed to ["Display=\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\"]MSWNET:["SMS_SITE=IVC"]\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\ as this is the local machine.
7/7/2017 9:57:38 AM Ignoring drive C:\. File C:\NO_SMS_ON_DRIVE.SMS exists.
7/7/2017 9:57:38 AM Ignoring drive D:\. File D:\NO_SMS_ON_DRIVE.SMS exists.
7/7/2017 9:57:38 AM Ignoring drive E:\. File E:\NO_SMS_ON_DRIVE.SMS exists.
7/7/2017 9:57:38 AM Ignoring drive I:\. File I:\NO_SMS_ON_DRIVE.SMS exists.
7/7/2017 9:57:38 AM user(NT AUTHORITY\SYSTEM) runing application(SMS_DISTRIBUTION_MANAGER) from machine (IVCCD-SCCM-01.ivd.iavalley.cc.ia.us) is submitting SDK changes from site(IVC)
7/7/2017 9:57:38 AM CreateSignatureShare, connecting to DP
7/7/2017 9:57:38 AM Signature share exists on distribution point path \\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\SMSSIG$
7/7/2017 9:57:38 AM Ignoring drive C:\. File C:\NO_SMS_ON_DRIVE.SMS exists.
7/7/2017 9:57:38 AM Ignoring drive D:\. File D:\NO_SMS_ON_DRIVE.SMS exists.
7/7/2017 9:57:38 AM Ignoring drive E:\. File E:\NO_SMS_ON_DRIVE.SMS exists.
7/7/2017 9:57:38 AM Ignoring drive I:\. File I:\NO_SMS_ON_DRIVE.SMS exists.
7/7/2017 9:57:38 AM Share SMSPKGM$ exists on distribution point \\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\SMSPKGM$
7/7/2017 9:57:38 AM user(NT AUTHORITY\SYSTEM) runing application(SMS_DISTRIBUTION_MANAGER) from machine (IVCCD-SCCM-01.ivd.iavalley.cc.ia.us) is submitting SDK changes from site(IVC)
7/7/2017 9:57:39 AM Successfully created directory \\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\SMSPKGM$\IVC00002 for package IVC00002
7/7/2017 9:57:39 AM user(NT AUTHORITY\SYSTEM) runing application(SMS_DISTRIBUTION_MANAGER) from machine (IVCCD-SCCM-01.ivd.iavalley.cc.ia.us) is submitting SDK changes from site(IVC)
7/7/2017 9:57:39 AM \\?\M:\SMSPKGSIG\IVC00002.20 could not be located
7/7/2017 9:57:39 AM Failed to get RDC signature path for for package IVC00002 version 20. Error = 2
7/7/2017 9:57:39 AM Copying package IVC00002 to the package share location \\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\SMSPKGM$\IVC00002 on the server ["Display=\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\"]MSWNET:["SMS_SITE=IVC"]\\IVCCD-SCCM-01.ivd.iavalley.cc.ia.us\
7/7/2017 9:57:39 AM CContentDefinition::ExpandContentDefinitionItems failed; 0x80070003
7/7/2017 9:57:39 AM CContentDefinition::Expand failed; 0x80070003
7/7/2017 9:57:39 AM Failed to expand package IVC00002 to \\?\M:\SMSPKGM$\IVC00002. Error code = 0x80070003
7/7/2017 9:57:39 AM Error occurred. Performing error cleanup prior to returning.
7/7/2017 9:57:42 AM DP thread for package IVC00002 with thread handle 0000000000001594 and thread ID 23980 ended.
7/7/2017 9:57:47 AM All DP threads have completed for package IVC00002 processing thread.
7/7/2017 9:57:48 AM Package IVC00002 does not have a preferred sender.
7/7/2017 9:57:48 AM The package and/or program properties for package IVC00002 have not changed, need to determine which site(s) need updated package info.
7/7/2017 9:57:48 AM CDistributionSrcSQL::UpdateAvailableVersion PackageID=IVC00002, Version=20, Status=2301
7/7/2017 9:57:48 AM StoredPkgVersion (20) of package IVC00002. StoredPkgVersion in database is 20.
7/7/2017 9:57:48 AM SourceVersion (20) of package IVC00002. SourceVersion in database is 20.
7/7/2017 9:57:48 AM STATMSG: ID=2302 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=IVCCD-SCCM-01.ivd.iavalley.cc.ia.us SITE=IVC PID=24008 TID=13980 GMTDATE=Fri Jul 07 14:57:48.328 2017 ISTR0="Boot image (x64)" ISTR1="IVC00002" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 AID0=400 AVAL0="IVC00002"
7/7/2017 9:57:48 AM Failed to process package IVC00002 after 0 retries, will retry 100 more times
7/7/2017 9:57:48 AM Exiting package processing thread for package IVC00002.

I believe the main point of failure is the error at 9:57:39 AM Failed to get RDC signature path for for package. From what I've found, that can be fixed by updating the content on the DP but that doesn't work for our boot images.

 

[Prajwal Desai]

7/7/2017 9:57:39 AM CContentDefinition::ExpandContentDefinitionItems failed; 0x80070003
7/7/2017 9:57:39 AM CContentDefinition::Expand failed; 0x80070003

quick check - Have you placed a NO_SMS_ON_DRIVE.SMS file on the same partition on which the Content Library is located ??.

 

[AnScott]

No, I haven't. I believe I have found a possible solution to the issue. I've moved any files I can find relating to the package from one of our secondary sites and put them in the corresponding places on the DP on the main site. I then updated the content on the main site and it completed successfully. I've only done this for our x64 boot image and will try in on the x86 boot image and update the thread.

 

[AnScott]

Alright, two year update time. I ended solving this issue (and then having to re-solve it after a bad site update, hence the update). There were two parts: one a permissions debacle and one a distribution issue.

The permissions problem was that I had not given the correct permissions on the network share and at the file level. To ensure access, I gave full access to the network access account, the computer account of the server and the SYSTEM account of the server to both the network share and file level permissions.

The distribution issue was the lack of content in the package on the DP. To solve this, distribute the content to the DP (which fails) then update distribution points. This copies the content to the SCCM internal package, then distributes it to the DPs. You have to do this to all content that was originally on the DP before loss. New packages created after the recovery shouldn't be effected but if they are, the same process should fix them.

After that, just wait the distmgr.log for any packages that fail and go from there. Hopes this helps someone else and not just me again in two years.