When you mention that the search results of one of the IC components
aren’t up to date anymore it could be that there is corrupt data
inside one of the components.
You will notice this when you try to search recent articles and they don’t
show up in the search results even if the “Search index was last updated”
status mentions that is was updated very recently ( /search component ).
You will find the CLFRT0066E error message if you try to retreive the
seedlist of the IC component that has no recent search results.
In the few cases I heard about the CLFRT0066E error it was caused
by corrupt data in the Blogs component. So I in this example i will go with Blogs as well.
CLFRT0066E: A backend internal error occurred – Java Exception Message : java.lang.NullPointerException
The next challenge is to find the corrupt data in the Blogs database. When
searching inside the scope of the Blogs component you should roughly be
able to find out since when the search results weren’t updated anymore.
I didn’t try it but deciding to remove the index and start rebuilding it all
will not likely help to fix the issue as the corrupt data inside the component
will continue to cause the retrieval of the seedlist to fail.
If you have some good log rotation in place for WebSphere you should
also be able to pinpoint since when the indexing process for Blogs started
to fail from the SystemOut.log of the JVM where search is running on.
To find out the specific time when the corrupt data was occured
you can use the start and range seedlist arguments.
Just begin with start=0 and range=1000 then try to get as close to the point
when you receive the CLFRT0066E error. The content of the seedlist is
chronological starting with start=0 will probaby give you as the first result
the creation of the first Blog.
When you have a good idea around what time the corrupt data started to
occur you will have to try to find the corrupt data inside the Blogs database.
My best guess is that it is about a corrupt Blog, a Blog entry or a comment.
For Oracle you will have to use the to_timestamp function to search in the
columns with the type of timestamp.
select * from BLOGS.WEBSITE
where LASTMODIFIED >= totimestamp(’14-05-2012 21:24:00′, ‘dd-mm-yyyy hh24:mi:ss’)
('14-05-2012 21:25:33', 'dd-mm-yyyy hh24:mi:ss')
//LASTMODIFIED or DBMODTIME just play around with both.
The next steps are bit depending on where you find the corrupt data and if
you are able to remove it in the Connections interface itself.
If you cannot do it anymore in the interface what will be very likely you will
have to try and clean it up in with SQL queries in the database itself.
Cleaning this up with SQL will require some db expertise as you will bump
into table relationships, foreigns keys etc.
After you got rid of the corrupt data and you no longer get the CLFRT0066E
error when retrieving the seedlist the next step is rebuilding the complete index.
( Not entirely sure about this step but I had to do it to fix the index of the Blogs component ).
When running in a multi node setup you can use the startBackgroundIndex to recreate
the index without taking Connections offline.
Start the background indexing process for one of the nodes.
SearchService.startBackgroundIndex(“/opt/IBM/LotusConnections/backgroundIndex”, “activities, blogs, communities, dogear, files, forums, profiles, wikis”)
When this is finished stop the JVM which hosts the Search application on that node.
Rename the old index directory as specified in the SEARCHINDEXDIR
WebSphere environment variable and replace it with the new created directory and
start up the JVM again. Repeat this step for all your nodes.
As file content indexing is done per node and stuff like recommendations, content
in the Do-You-Know widget are all retreived from the search index it will take
some time before all this information is rebuild again.
Thanks to Sjaak Ursinus who pointed me in the right direction.