Recently, I was working on a case with pool-pairing with a unique twist. The import status for Conferencing Module was working, but for User Module was failing . Upon looking further, I noticed EVENT ID 4060 with the following text
Log Name: Lync Server
Source: LS Backup Service
Date: 5/16/2017 11:25:48 AM
Event ID: 4060
Task Category: (4000)
Level: Error
Keywords: Classic
User: N/A
Computer: skype-pool1-FE2.contoso.com
Description:
Skype for Business Server 2015, Backup Service user store backup module failed to complete import operation.
Configurations:
Backup Module Identity:UserServices.PresenceFocus
Working Directory path:\contoso.comSFBShare1-BackupService-2BackupStoreTemp
Local File Store Unc path:\contoso.comSFBShare1-BackupService-2BackupStore
Remote File Store Unc path:\contoso.comSFBShare2-BackupService-1BackupStore
Additional Message:
Exception: Microsoft.Rtc.BackupService.ImportOperationException: Import operation (from zip archive ) is failed due to: Failed to execute stored procedure XdsQueryReplicaStatus. Native Error: 916, Exception: The server principal “CONTOSOskype-pool1-FE2$” is not able to access the database “msdb” under the current security context.. Retriable: False. Cookie: . —> System.Data.SqlClient.SqlException: The server principal “CONTOSOskype-pool1-FE2$” is not able to access the database “msdb” under the current security context.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows)
at System.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more)
at System.Data.SqlClient.SqlDataReader.TryNextResult(Boolean& more)
at System.Data.SqlClient.SqlDataReader.NextResult()
at Microsoft.Rtc.Common.Data.DBCore.ParseResults(SprocContext sprocContext, SqlDataReader sqlReader)
at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)
— End of inner exception stack trace —
at Microsoft.Rtc.BackupService.BackupModules.XdsBackupModuleBase.QueryBackupStatus()
at Microsoft.Rtc.BackupService.BackupModules.UserStoreBackupModule.GetBackupCookie()
at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.GetBackupCookie(Boolean& isModuleInitialized)
at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.InternalExecute()
at Microsoft.Rtc.Common.TaskManager`1.ExecuteTask(Object state)
Cause: Either network or permission issues. Please look through the exception details for more information.
Resolution:
Resolution
So I decided to collect logs the built-in scenario HADR. The scenario HADR has the following components
- BackupService
- PowerShell
- RtcDbSyncAgent
- UserServices
Since the scenario included both BackupService and UserServices, I was optimistic that I would certainly nail-down the issue. However, the logs were not leading me to any solution. It was only providing the name of the sproc XdsQueryReplicaStatus. However, this sproc exists on the Front-End Server and the Back-End Servers, so I needed more information on which database to troubleshoot.
Next I decided to collect a memory dump using the command
ProcDump.exe -ma -e System.Data.SqlClient.SqlException LyncBackupService.exe
Upon investigation, I finally found that the sproc XdsQueryReplicaStatus was connecting to the Lync Back-end Server. Now that I know the issue was with SQL, it certainly seemed like a Permissions issue, and so I double-checked the permissions with permission in a working environment. But was unable to find anything within the permissions of the databases used by Skype for Business Server 2015.
Finally, I decided to check the permissions for System Databases, and found that in the failing scenario, GUEST login was removed from MSDB (system database). So I decided to grant GUEST user the Connect permissions for MSDB database by running
USE msdb;
GRANT connect TO guest;
GO
Next, since this was an update to the System Database, we restarted the SQL Services, and the issue was resolved.
Resource:
You should not disable the guest user in the msdb database in SQL Server
What does the status reported by Get-CsBackupServiceStatus mean?