Search Broken II

I had an issue with configuring the Windows SharePoint Services Search Service. There is nothing in the SharePoint logs which suggest any problem. A few lines in the log does not provide any help.

01/08/2009 18:52:53.19 
w3wp.exe (0x2C6C)                       
0x1AE8 Search Server Common          
MS Search Administration      
86z4
High    
Exception caught in Search Admin web-service (server). Microsoft.Office.Server.Search.Administration.DeletedConcurrencyException: Object 3e41e0c9-53d5-4349-ae92-8c8dc488164c not found. ---> System.Collections.Generic.KeyNotFoundException: Object 3e41e0c9-53d5-4349-ae92-8c8dc488164c not found.     at Microsoft.Office.Server.Search.Administration.SearchApi.get_App()     at Microsoft.Office.Server.Search.Administration.SearchApi.get_Project()     at Microsoft.Office.Server.Search.Administration.SearchApi.b__d5()     at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Boolean useCurrentSecurityContext, Int32 versionIn)     --- End of inner exception stack trace ---     at Microsoft.Office.Server.Search.Admi... 
01/08/2009 18:52:53.19* w3wp.exe (0x2C6C)                        0x1AE8 Search Server Common           MS Search Administration       86z4 High     ...nistration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Boolean useCurrentSecurityContext, Int32 versionIn)     at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Int32 versionIn)     at Microsoft.Office.Server.Search.Administration.SearchApi.GetImportAccounts(Int32 versionIn)     at Microsoft.Office.Server.Search.Administration.SearchWebService.RunWithSoapExceptionHandling[T](CodeRequiresSoapExceptionHandling`1 webMethodCode, Boolean impersonateLocalAdmin)
Event Viewer generates the following errors at regular intervals:
Event Type: Error
Event Source: Office Server Search
Event Category: Gatherer
Event ID: 10034
Date:  1/8/2009
Time:  7:29:03 PM
User:  N/A
Computer: IndexServer1
Description:
Could not access the Search service configuration database.
Context: Application '3e41e0c9-53d5-4349-ae92-8c8dc488164c'
Details: One or more errors occurred during processing of command.   (0x80040e14)
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type: Error
Event Source: Office Server Search
Event Category: Gatherer
Event ID: 10036
Date:  1/8/2009
Time:  7:29:03 PM
User:  N/A
Computer: IndexServer1
Description:
A database error occurred. Source: Microsoft OLE DB Provider for SQL Server Code: 102 occurred 1 time(s) Description: Incorrect syntax near 'OPTIMIZE'.
Context: Application '3e41e0c9-53d5-4349-ae92-8c8dc488164c'
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type: Error
Event Source: Office Server Search
Event Category: Gatherer
Event ID: 10036
Date:  1/8/2009
Time:  7:29:03 PM
User:  N/A
Computer: IndexServer1
Description:
A database error occurred. Source: Microsoft OLE DB Provider for SQL Server Code: 257 occurred 1 time(s) Description: Implicit conversion from data type sql_variant to nvarchar is not allowed. Use the CONVERT function to run this query.
Context: Application '3e41e0c9-53d5-4349-ae92-8c8dc488164c'
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type: Error
Event Source: Office SharePoint Server
Event Category: Office Server Shared Services
Event ID: 6482
Date:  1/8/2009
Time:  7:28:03 PM
User:  N/A
Computer: IndexServer1
Description:
Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (e5c2e518-3711-40a3-8860-ae5ee787307b).
Reason: Could not access the Search service configuration database 
Techinal Support Details:
Microsoft.Office.Server.Search.Administration.DatabaseException: Could not access the Search service configuration database   ---> System.Runtime.InteropServices.COMException (0xC004122D): Could not access the Search service configuration database 
   at Microsoft.Office.Server.Search.Administration.MSSITLB.IGatherApplications2.Mount(String bstrName)
   at Microsoft.Office.Server.Search.Administration.GathererApplication.ProvisionGathererApplication()
   at Microsoft.Office.Server.Search.Administration.GathererApplication.Provision()
   at Microsoft.Office.Server.Search.Administration.OSSIndexGathererApplication.Provision()
   at Microsoft.Office.Server.Search.Administration.SearchApi.<>c__DisplayClass20.<AddApp>b__1f()
   at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Boolean useCurrentSecurityContext, Int32 versionIn)
   --- End of inner exception stack trace ---
   at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Boolean useCurrentSecurityContext, Int32 versionIn)
   at Microsoft.Office.Server.Search.Administration.SearchApi.AddApp(Role role)
   at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.InstallGathererApplicationIf()
   at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize()
   at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Solution:
The event id 10036 was the key here. It was actually a database issue. We figured out that the two databases of the cluster (active and passive) have two levels of patches. That is why when it failed over to the other node today morning, it stopped working. We have switched it back to the main node and installed the required patches on the passive node. The search service came back as soon as we moved to the primary node

If this post helped you, please visit the sponsors from SharePointAds.com to check the latest offerings