I had an interesting issue where a Skype for Business admin site would sit at the spinning wheel at 100%. This environment had two Enterprise pools so I checked the other site to find the same thing. At this stage I was fairly convinced that it was bigger than just a bad server.
I then opened up powershell which connected fine. Great!!
Next I ran a command after much thought or more to the point after typing get-cs<couple of tabs><enter> which happened to end up on Get-CSADDomain.
So this returned LC_DOMAINSETTINGS_STATE_FAILED. Urgh!
That looks pretty average for what, at this point, is an operational environment.
So next I ran get-CSUser, and we waited. Yeah there are a few users in the environment so that’s the be expected but after a couple of minutes I knew that this wasn’t going to end.
I checked the event log and found the following error in the Lync Server Log
Source: LS Remote PowerShell
Event ID: 35009
Remote PowerShell cannot create InitialSessionState.
Remote PowerShell cannot create InitialSessionState for user: S-1-5-21-XXXXXXXXX-XXXXXXXXX-XXXXXXXXX-XXXXX. Cause of failure: Thread was being aborted.. Stacktrace: System.Threading.ThreadAbortException: Thread was being aborted.
at System.Threading.WaitHandle.WaitOneNative(SafeHandle waitableSafeHandle, UInt32 millisecondsTimeout, Boolean hasThreadAffinity, Boolean exitContext)
at System.Threading.WaitHandle.InternalWaitOne(SafeHandle waitableSafeHandle, Int64 millisecondsTimeout, Boolean hasThreadAffinity, Boolean exitContext)
at System.Threading.WaitHandle.WaitOne(Int32 millisecondsTimeout, Boolean exitContext)
at Microsoft.Rtc.Management.Store.Sql.ClientDBAccess.OnBeforeSprocExecution(SprocContext sprocContext)
at Microsoft.Rtc.Common.Data.DBCore.ExecuteSprocContext(SprocContext sprocContext)
at Microsoft.Rtc.Management.Store.Sql.XdsSqlConnection.ReadDocItems(ICollection`1 key)
at Microsoft.Rtc.Management.ScopeFramework.AnchoredXmlReader.Read(ICollection`1 key)
at Microsoft.Rtc.Management.ServiceConsumer.CachedAnchoredXmlReader.Read(ICollection`1 key)
at Microsoft.Rtc.Management.ServiceConsumer.TypedXmlReader.Read(SchemaId schemaId, IList`1 scopeContextList, Boolean useDefaultIfNoneExists)
at Microsoft.Rtc.Management.Authorization.OcsRunspaceConfiguration.ConstructCmdletsAndScopesMap(List`1 tokenSIDs)
at Microsoft.Rtc.Management.Authorization.OcsRunspaceConfiguration..ctor(IIdentity logonIdentity, IRoleStoreAccessor roleAccessor, List`1 tokenGroups)
at Microsoft.Rtc.Management.Authorization.OcsAuthorizationPlugin.CreateInitialSessionState(IIdentity identity, Boolean insertFormats, Boolean insertTypes, Boolean addServiceCmdlets)
Cause: Remote PowerShell can fail to create InitialSessionState for varied number of reasons. Please look for other events that can give some specific information.
Follow the resolution on the corresponding failure events.
Well that doesn’t look so good. Reading this it looked like it might be a database issue. This would make sense since the CMS database is in a single location with all servers accessing it. Even if an object is in AD, Skype for Business will get information about it from a single place, the CMS.
If you have multiple pools including fail-over pools then there is still just one CMS service.
The database server was busier than expected but nothing was standing out as really bad. (60% average CPU for the SQL process and a few deadlocked processes reported in the SQL log) but it did seem responsive.
It was at this point that other services using the same SQL server also were reported as being down and the SQL admin made the call to restart the SQL Service.
Once restarted everything became responsive again.
Unfortunately I never got to the bottom of what was wrong in the SQL server, but I think it’s still good to remember the heavy reliance on the database service in Skype for Business. Yes there is a SQL service on each Skype for Business Server, but this isn’t used for all processes.