We've confirmed that the "flakiness" some people reported this morning with Illumina was related to new indexing procedures CSA is implementing. The new indexing tools work in "real-time" against databases — that is, instead of updating a copy of the database and then copying it over the production database, the online version of the database is now being updated as it is being accessed by searchers. This means that while a database is being updated, searches against that database will be slower. This morning, while MEDLINE and PsycINFO were being updated, some searches were timing out and returning zero hits.
The new indexing method will allow us to build much larger indexes, so this will be of particular value for Scholars Portal as we continue to push the envelope on content. What we need to do, however, is adjust the timeout values on database searches whenever updates are being run so that we avoid the "zero hit" problems of this morning. The new indexing method will also allow us to schedule indexing jobs in off hours, since the procedures will be less labour intensive, at least in the long run.
