Crawling SiteWeb fails with “System.ArguementException *** Message: An item with the same key has already been added.”

• When we see this error, generally on the mssdmn.exe process with a EventID = dm1k
  

  System.ArgumentException *** Message : An item with the same key has already been added.
  

• This means it has probably failed in the “ParseGroupCollection” method within the SiteData call for the “Site“, or SiteCollection, object
  
• In the ULS, if you were to filter where you see that error, by the ThreadID, you would see something like this:
  
  

  06/07/2017 09:40:56.23 mssdmn.exe (0x3124) 0x2F28 SharePoint Server Search Connectors:SharePoint ac3jn Verbose GetContent CorrelationID 393ef99d-174e-b0b4-a06f-97dc29dd8957 ObjectType SiteCollection Url https://foo/_vti_bin/sitedata.asmx Search RequestDuration 449 SPIISLatency 0 SP RequestDuration 339 57ec8c92-d753-42c2-b636-a1280b8b9e6c
  

  06/07/2017 09:40:56.37 mssdmn.exe (0x3124) 0x2F28 SharePoint Server Search Connectors:SharePoint dm1k Medium Exception Type: System.ArgumentException *** Message : An item with the same key has already been added. *** StackTrace: at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add) at Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.ParseGroupCollection(XmlReader groupReader) at Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.InitSite(String strSiteUrl) at Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.GetSite(String strSiteUrl, sSite3& site) 57ec8c92-d753-42c2-b636-a1280b8b9e6c
  

• We can utilize PowerShell to emulate this sitedata.asmx call on the SiteCollection Object.
  
• I suggest doing this, so we can investigate the XML andor use it for reference later.
  
• You can see in the ULS sample above that we called the “SiteCollection” object on https://foo/_vti_bin/sitedata.asmx ( if it were another SiteCollection, you would see the URL in this ULS entry )
  
• In order to get this XML, you can run this ( when prompted for creds, you want to use the Default Content Access Account credentials )
  

  ########################################################################################
  

  ## Set SiteData Web Service Object ### Use the Content Access Account for the $creds variable
  

  ########################################################################################
  

  $creds = Get-Credential
  $url = “https://foo”
  $wsUrl = $url + “/_vti_bin/sitedata.asmx?wsdl”
  $sitedataWS = New-WebServiceProxy -Uri
  $wsUrl -Credential $creds
  

  ########################################################################################
  

  ## SiteCollection Object – ObjectType = 2 ##
  

  ########################################################################################
  

  $int = [int]2
  $xmlinput =
  “<GetContent><ObjectType>2</ObjectType><ObjectId /><FolderUrl /><ItemId /><RetrieveChildItems>False</RetrieveChildItems><SecurityOnly>False</SecurityOnly><LastItemIdOnPage /><AllowRichText>False</AllowRichText><RequestLoad>100</RequestLoad><RemoveInvalidXmlChars>True</RemoveInvalidXmlChars></GetContent>” 

  $xml = $sitedataWS.GetContentEx($int,$xmlinput)
  

  $xml | Out-File “C:TempSiteCollection.xml”
  

• Once you collect the XML, we can parse it through additional PowerShell
  
• Some time back, we created some PowerShell that would try to mimic this “ParseGroupCollection” and try to identify duplicate “Groups” within the returned XML.
  

#the attributes that are in the group element/node

$attributes = “ID”,“Name”,“OwnerID”,“OwnerIsUser”,“Description”

#the object array of custom objects we will use to return data

$results = @()

#setup the xml reader and get the data

$xmlReader = [System.Xml.XmlReader]::Create(‘C:TempSiteCollection.xml’)

#read dat data

while($xmlReader.Read())

{

#Set things we want to check ahead of the compare operation

$isElementNodeType = $xmlReader.NodeType -eq [System.Xml.XmlNodeType]::Element 

$isGroupNode = $xmlReader.Name -eq ‘Group’

#Compare ALL THE THINGS!
if($isElementNodeType -and $xmlReader.HasAttributes -and $isGroupNode)

{
#create object to store the current group element 
$currentGroup = “” | Select “ID”,“Name”,“OwnerID”,“OwnerIsUser”,“Description”

#loop through attribute array set above.
foreach($attribute in $attributes)
{
#start populating data in $currentGroup by reading that attribute from current point in reader

$currentGroup.$attribute = $xmlReader.GetAttribute($attribute) 

}
#add current group to results
$results += $currentGroup
}
}
$resultcount = $results.Count
Write-Host “There are $resultcount group elements”
##see the items easily
#$results | Select Id, Name | Out-GridView
#group items and then look for items when 

$results | group -Property name | ? {$_.count -gt1} | select count, name
$results | group -Property Id | ? {$_.count -gt1} | select count, name

• If there are any Duplicates, on the Name or ID, we would dump out the Name or ID with a count and the Name of the duplicate Group
  
• If you get any returns on these, where count > 1, then you can go find those groups in the Site Permissions of your Site and remove one of the Groups.
  
• There are times that parsing through this XML doesn’t yield any Duplicate NamesIDs so what do you do in that case?
  
• This is where a Process Dump would be handy
  
• We have a pre-created rule that willshould dump on the following:

            tag_dm1k AND message contains “An item with the same key has already been added.”

• Download “Debug Diag 2.0” from here:

https://www.microsoft.com/en-us/download/details.aspx?id=49924

• Install it on your Crawl Server(s)
  
• Once you install it, open “DebugDiag 2.0 Collection”
  
• When the crashhang wizard comes up, click on “Close” or Cancel
  
• Then in the Lower right corner, click on the “Import” button
  
• Select that “tag_dm1k_partial.ddconfig” file you saved to your Crawl Server(s)
  
• Click OK
  
• Once the rule loads up, right click the Rule > Activate Rule
  
• Trigger a crawl..
  

• This should monitor mssdmn.exe process and capture a DMP file when that EventId = dm1k is thrown, with that message in the ULS
  

• looking at the DMP file.
  

• i see this
  

  0:106> !threads
  

  ThreadCount: 143
  

  UnstartedThread: 0
  

  BackgroundThread: 143
  

  PendingThread: 0
  

  DeadThread: 0
  

  Hosted Runtime: no
  

  Lock
  

  ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception
  

  0 1 1a60 000000338a9306f0 20220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA
  

  4 2 3138 000000338a93b3e0 2b220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA (Finalizer)
  

  9 3 3628 00000033a48469a0 20220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA
  

  13 4 24ec 00000033ad63a0e0 28220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA
  

  18 5 1be8 00000033ad63e890 28220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA
  

  <truncated>
  

  12 71 24a4 00000033b02fa880 20220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA
  

  104 72 8cc 00000033b7c2b8b0 20220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA
  

  106 73 188c 00000033b0336fd0 20220 Preemptive 000000338DE59980:000000338DE5AFC0 000000338a923120 0 MTA System.ArgumentException 000000338de50b08
  

• next i dumped the Stack out
  

0:106> ~k

# Child-SP RetAddr Call Site

00 00000033`ab3f5ef8 00007ff7`e637d4f7 Microsoft_Office_Server_Native!ULSSendFormattedTrace [n:srcutilulsapinativetrace.cpp @ 404]

01 00000033`ab3f5f00 00007ff7`e637c265 ConnectorPH!DomainBoundILStubClass.IL_STUB_PInvoke(UInt32, UInt32, Microsoft.Office.Server.Diagnostics.ULSTraceLevel, System.String, Boolean)+0x157

02 00000033`ab3f6020 00007ff7`e637b095 Microsoft_Office_Server!Microsoft.Office.Server.Diagnostics.ULS.SendTraceImpl(UInt32, Microsoft.Office.Server.Diagnostics.ULSCatBase, Microsoft.Office.Server.Diagnostics.ULSTraceLevel, System.String, System.Object[])+0x125 [n:srcotoolsincosrvDiagnosticsULS.cs @ 2334]

03 00000033`ab3f6080 00007ff7`e7392bd1 Microsoft_Office_Server!Microsoft.Office.Server.Diagnostics.ULS.SendTraceTag(UInt32, Microsoft.Office.Server.Diagnostics.ULSCatBase, Microsoft.Office.Server.Diagnostics.ULSTraceLevel, System.String, System.Object[])+0x155 [n:srcotoolsincosrvDiagnosticsULS.cs @ 2303]

04 00000033`ab3f6120 00007ff7`e737627e Microsoft_Office_Server_Search!Microsoft.Office.Server.Search.Internal.Protocols.CProtocolHelper.HandleException(System.Exception, UInt32)+0xf1 [n:srcsearchmanagedlibOMProtocolsutil.cs @ 545]

05 00000033`ab3f61b0 00007ff8`45ac59c5 Microsoft_Office_Server_Search!Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.GetSite(System.String, Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.sSite3 ByRef)+0x42e [n:srcsearchmanagedlibOMProtocolsstsv3ph.cs @ 2732]

06 00000033`ab3f6220 00007ff8`45ac58c3 clr!ExceptionTracker::CallHandler+0xc5 [f:ddndpclrsrcvmexceptionhandling.cpp @ 3341]

07 00000033`ab3f62c0 00007ff8`45ac42af clr!ExceptionTracker::CallCatchHandler+0x7f [f:ddndpclrsrcvmexceptionhandling.cpp @ 535]

08 00000033`ab3f6350 00007ff8`4e53347d clr!ProcessCLRException+0x2e6 [f:ddndpclrsrcvmexceptionhandling.cpp @ 1067]

09 00000033`ab3f6430 00007ff8`4e4f5405 ntdll!RtlpExecuteHandlerForUnwind+0xd [d:blueminkernelntosrtlamd64xcptmisc.asm @ 252]

0a 00000033`ab3f6460 00007ff8`45ac4348 ntdll!RtlUnwindEx+0x385 [d:blueminkernelntosrtlamd64exdsptch.c @ 1006]

0b 00000033`ab3f6b40 00007ff8`45ac4304 clr!ClrUnwindEx+0x40 [f:ddndpclrsrcvmexceptionhandling.cpp @ 4328]

0c 00000033`ab3f7060 00007ff8`4e5333fd clr!ProcessCLRException+0x2b2 [f:ddndpclrsrcvmexceptionhandling.cpp @ 1032]

0d 00000033`ab3f7140 00007ff8`4e4f4847 ntdll!RtlpExecuteHandlerForException+0xd [d:blueminkernelntosrtlamd64xcptmisc.asm @ 131]

0e 00000033`ab3f7170 00007ff8`4e53258a ntdll!RtlDispatchException+0x197 [d:blueminkernelntosrtlamd64exdsptch.c @ 535]

0f 00000033`ab3f7840 00007ff8`4b6b871c ntdll!KiUserExceptionDispatch+0x3a [d:blueminkernelntosrtlamd64trampoln.asm @ 715]

10 00000033`ab3f7f50 00007ff8`45ac5294 KERNELBASE!RaiseException+0x68 [d:blueminkernelkernelbasexcpt.c @ 828]

11 00000033`ab3f8030 00007ff8`45ac508e clr!RaiseTheExceptionInternalOnly+0x2fe [f:ddndpclrsrcvmexcep.cpp @ 3134]

12 00000033`ab3f8130 00007ff8`453daacf clr!IL_Throw+0x11b [f:ddndpclrsrcvmjithelpers.cpp @ 5017]

13 00000033`ab3f8300 00007ff7`e737cdc2 mscorlib_ni!System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Insert(System.__Canon, System.__Canon, Boolean)+0xdd274f [f:ddndpclrsrcBCLSystemCollectionsGenericDictionary.cs @ 314]

14 00000033`ab3f8390 00007ff7`e737660e Microsoft_Office_Server_Search!Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.ParseGroupCollection(System.Xml.XmlReader)+0x192 [n:srcsearchmanagedlibOMProtocolsstsv3ph.cs @ 2642]

15 00000033`ab3f8410 00007ff7`e7375ef2 Microsoft_Office_Server_Search!Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.InitSite(System.String)+0x2ae [n:srcsearchmanagedlibOMProtocolsstsv3ph.cs @ 2628]

16 00000033`ab3f84b0 00007ff7`e7375af9 Microsoft_Office_Server_Search!Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.GetSite(System.String, Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.sSite3 ByRef)+0xa2 [n:srcsearchmanagedlibOMProtocolsstsv3ph.cs @ 2704]

<truncated>

• We can see our function in here…
  

ParseGroupCollection(System.Xml.XmlReader)

• the line above is the hashtablecollection
  
• so i dumped that Stack Object out:
  

0:106> !dso 00000033`ab3f8300

OS Thread Id: 0x188c (106)

RSP/REG Object Name

rbx 000000338de594e8 System.String Exception Type: System.ArgumentException *** Message : An item with the same key has already been added.
*** StackTrace: at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)

at Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.ParseGroupCollection(XmlReader groupReader)

at Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.InitSite(String strSiteUrl)

at Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.GetSite(String strSiteUrl, sSite3& site)

r12 000000338c634130 Microsoft.Office.Server.Diagnostics.ULSCat

r13 000000338c73add8 System.String Exception Type: {0} *** Message : {1} *** StackTrace: {2}

r14 000000338de50e08 System.Object[] (System.Object[])

00000033AB3F8300 000000338d241d58 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[Microsoft.Office.Server.Search.Internal.Protocols.sSecurityUser[], Microsoft.Office.Server.Search]]

00000033AB3F8320 000000338d241d58 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[Microsoft.Office.Server.Search.Internal.Protocols.sSecurityUser[], Microsoft.Office.Server.Search]]

00000033AB3F8330 000000338de50970 Microsoft.Office.Server.Search.Internal.Protocols.sSecurityUser[]

00000033AB3F8338 000000338de11038 System.String 1031479 Precision breeding for piglet survival PhD studentship Owners

00000033AB3F8358 000000338d242000 System.String https://foo

00000033AB3F8370 000000338c7694c0 System.Xml.XmlSubtreeReader

00000033AB3F8378 000000338d241cd0 Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper

00000033AB3F8380 000000338de4fe80 System.Xml.XmlSubtreeReader

00000033AB3F8390 000000338de4fe80 System.Xml.XmlSubtreeReader

00000033AB3F8398 000000338be97090 System.Globalization.CultureInfo

00000033AB3F83B0 000000338de4fdb0 System.String 1031479 Precision breeding for piglet survival PhD studentship Owners

00000033AB3F83B8 000000338de4fe58 System.String 10058

00000033AB3F83C8 000000338de50970 Microsoft.Office.Server.Search.Internal.Protocols.sSecurityUser[]

00000033AB3F83D0 000000338de4fdb0 System.String 1031479 Precision breeding for piglet survival PhD studentship Owners

00000033AB3F83D8 000000338de4fe58 System.String 10058

00000033AB3F83E8 000000338c7660a8 System.Xml.XmlTextReaderImpl

00000033AB3F83F8 000000338d241cd0 Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper

00000033AB3F8400 000000338c7694c0 System.Xml.XmlSubtreeReader

00000033AB3F8548 000000338de50b08 System.ArgumentException

00000033AB3F85B0 000000338d241cd0 Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper

00000033AB3F85B8 000000338d242000 System.String https://foo

00000033AB3F85E0 000000338d242000 System.String https://foo

00000033AB3F8780 000000338d241cd0 Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper

00000033AB3F96B0 000000338be91420 System.String

00000033AB3F96C0 000000338be91420 System.String

00000033AB3F9740 000000338be91420 System.String

00000033AB3F9770 000000338be91420 System.String

00000033AB3F9780 000000338be91420 System.String

00000033AB3F9810 000000338be91420 System.String

00000033AB3F9820 000000338be91420 System.String

00000033AB3F9848 000000338be91420 System.String

00000033AB3F9878 000000338be91420 System.String

00000033AB3F9880 000000338be91420 System.String

00000033AB3F9890 000000338be91420 System.String

• this group is of interest to me:
  

1031479 Precision breeding for piglet survival PhD studentship Owners

10058

• so i went back to my SiteCollection.xml we had collected..
  
• i changed the PS a bit to dump the $results object from the Previous PS
  

              $results | Select Id, Name | Out-GridView

• and then I filtered where Name contains “Precision”
  
• i see these entries here:
  

• those with IDs = 9972, 9973, 9974 appear to be the Culprit here. ( somehow these entries made it into the Groups table with that funky encoded value )
  
• so we used the following PowerShell “SiteGroups” removed:
  

$ids = “9972”, “9973”, “9974”
$w = Get-SPWeb https://foo

foreach($sg in $w.SiteGroups | ?{$_.Name -match “Precision breeding for piglet survival PhD Student”})
{
 foreach($id in $ids)
{
if($sg.Id -eq $id)
{
“Removing Group ” + $sg.Name + ” With ID: ” + $sg.Id

    $w.SiteGroups.RemoveByID($id)
}
}
}

• this should get these groups out and allow the site to be crawled..