This was a special situation where most of the site appeared to work, but certain links under Site Settings would fail with 403 Forbidden. For example:
- Themes
- Master Pages
- Solutions
- Composed looks
- List Templates
- Most document libraries
Actually, in some cases, the page request would result in Access Denied, and redirect the user to the Access Denied page (_layouts/15/AccessDenied.aspx), and that request would fail with 403 Forbidden.
This would occur for users with full control permission on the site, even site collection administrators.
Cause:
This turned out to be a configuration problem with Host-Named Site Collections (HNSC).
Due to a host header / IIS binding configuration issue, the request was actually being served by a different SharePoint web app / IIS site than the one we expected. That other IIS site was running in a different application pool, which was running as a different service account. That service account did not have enough permission to the SharePoint content to access the site. In that case, most requests were working, but any request that called into the "SPSecurity.RunWithElevatedPrivileges" method would fail because "RunWithElevatedPrivileges" runs as the application pool account instead of the logged-on user.
Resolution:
Configure your Web Apps and IIS bindings properly for HNSCs.
Typically, host-named site collections do not use host header bindings in IIS as described here: https://docs.microsoft.com/en-us/SharePoint/administration/host-named-site-collection-architecture-and-deployment#use-host-headers-and-host-named-site-collections
Host-named site collections should ideally all be hosted under the same SharePoint web application, in which case, host header bindings in IIS are unnecessary. Or if there are multiple HNSC web apps, they all should run as the same service account, which would also avoid this problem.
Here's a sample of the error sequence seen in the SharePoint ULS logs.
The giveaway is that every access denied stack contains a call to SPSecurity.RunWithElevatedPrivileges:
10/29/2018 10:16:01.47 w3wp.exe (0x51F4) 0x41D0 SharePoint Foundation Authentication Authorization ajmmv VerboseEx VsecurableObject: EffectiveRightsMask=0x00000000, OriginalEffectiveRightsMask=0x00000000 922c9d9e-f771-9026-a024-8163eefb3d05
10/29/2018 10:16:01.47 w3wp.exe (0x51F4) 0x41D0 SharePoint Foundation General 8xfr Verbose PermissionMask check failed. asking for 0x08000000, have 0x00000000 922c9d9e-f771-9026-a024-8163eefb3d05
10/29/2018 10:16:01.47 w3wp.exe (0x51F4) 0x41D0 SharePoint Foundation General 8e2s Medium Unknown SPRequest error occurred. More information: 0x80070005 922c9d9e-f771-9026-a024-8163eefb3d05
10/29/2018 10:16:01.47 w3wp.exe (0x51F4) 0x41D0 SharePoint Foundation General aix9j High SPRequest.GetUsersDataAsSafeArray: UserPrincipalName=, AppPrincipalName= ,bstrUrl=https://teams.contoso.com ,dwUsersScope=5 ,dwUserCollectionFlags=0 ,bstrValue= ,dwValue=1073741823 922c9d9e-f771-9026-a024-8163eefb3d05
10/29/2018 10:16:01.47 w3wp.exe (0x51F4) 0x41D0 SharePoint Foundation General ai1wu Medium System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)), StackTrace:
at Microsoft.SharePoint.SPUser.InitMember()
at Microsoft.SharePoint.SPUser..ctor(SPWeb web, SPSecurableObject scope, String strIdentifier, Object[,] arrUsersData, UInt32 index, Int32 iByParamId, String strByParamSID, String strByParamEmail, SPUserCollectionType userCollectionType, SPUserCollectionFlags ucf, Boolean isSiteAuditor)
at Microsoft.SharePoint.SPUserCollection.GetByIDNoThrow(Int32 id)
at SharePointBoost.SharePoint.Security.SecurityUtility. . ()
at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3()
at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)
at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)
at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)
<stack truncated>
10/29/2018 10:16:01.47 w3wp.exe (0x51F4) 0x41D0 SharePoint Foundation General aep94 High UnauthorizedAccessException for the request. 403 Forbidden will be returned. Error=Exception of type 'System.Web.HttpUnhandledException' was thrown.
at System.Web.UI.Page.HandleError(Exception e)
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
at System.Web.UI.Page.ProcessRequest()
at System.Web.UI.Page.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
10/29/2018 10:16:01.47 w3wp.exe (0x51F4) 0x41D0 SharePoint Foundation General b6p2 VerboseEx Sending HTTP response 403 - text/plain:403 FORBIDDEN. 922c9d9e-f771-9026-a024-8163eefb3d05