RECOLL indexing speed and index sizes

The time needed to index a given set of documents, and the resulting index size depend of many factors.

The index size depends almost only on the size of the uncompressed input text, and you can expect it to be roughly of the same order of magnitude. Depending on the type of file, the proportion of text to file size varies very widely, going from close to 1 for pure text files to a very small factor for, e.g., metadata tags in mp3 files.

Estimating indexing time is a much more complicated issue, depending on the type and size of input and on system performance. There is no general way to determine what part of the hardware should be optimized. Depending on the type of input, performance may be bound by I/O read or write performance, CPU single-processing speed, or combined multi-processing speed.

It should be noted that Recoll performance will not be an issue for most people. The indexer can process 1000 typical PDF files per minute, or 500 Wikipedia HTML pages per second on medium-range hardware, meaning that the initial indexing of a typical dataset will need a few dozen minutes at most. Further incremental index updates will be much faster because most files will not need to be processed again.

However, there are Recoll installations with terabyte-sized datasets, on which indexing can take days. For such operations (or even much smaller ones), it is very important to know what kind of performance can be expected, and what aspects of the hardware should be optimized.

In order to provide some reference points, I have run a number of benchs on medium-sized datasets, using typical mid-range desktop hardware, and varying the indexing configuration parameters to show how they affect the results.

The following may help you check that you are getting typical performance for your indexing, and give some indications about what to adjust to improve it.

From time to time, the is a report about a system becoming unusable during indexing. As far as I know, with the default Recoll configuration, and barring an exceptional issue (bug), this is always due to a system problem (typically bad hardware such as a disk doing retries). The tests below were mostly run while I was using the desktop, which never became unusable. However, some tests rendered it less responsive and this is noted with the results.

The following text refers to the indexing parameters without further explanation. Here follow links to more explanation about the processing model and configuration parameters.

All tests were run without generating the stemming database or aspell dictionary. These phases are relatively short and there is nothing which can be optimized about them.

The tests were run on what could be considered a mid-range desktop PC (around 2016):

This is usually a fanless PC, but I did run a fan on the external case fins during some of the tests (esp. PDF indexing), because the CPU was running a bit too hot.

The tests were run on 18000 random PDFs harvested on Google, with a total size of around 30 GB, using Recoll 1.22.3 and Xapian 1.2.22. The resulting index size was 1.2 GB.

The tests were run on an (old) French Wikipedia dump: 2.9 million HTML files stored in 42000 directories, for an approximate total size of 41 GB (average file size 14 KB).

The files are stored on a local SSD. Just reading them with find+cpio takes close to 8 mn.

The resulting index has a size of around 30 GB.

I was too lazy to extract 3 million entries tar file on a spinning disk, so all tests were performed with the data stored on a local SSD.

For this test, the indexing time is strongly dominated by the Xapian index updates. As these are single threaded, only the flush interval has a real influence.

The indexing process becomes quite big (resident size around 4GB), and the combination of high I/O load and high memory usage makes the system less responsive at times (but not unusable). As this happens principally when switching applications, my guess would be that some program pages (e.g. from the window manager and X) get flushed out, and take time being read in, during which time the display appears frozen.

For this kind of data, single-threaded CPU performance and storage write speed can make a difference. Multithreading does not help significantly because the indexing thread is such a dominant bottleneck.

The indexing time can only be improved by partitionning the data set and indexing into multiple Xapian indexes in parallel (the indexes can then be joined for querying).

I think that the following multi-step approach has a good chance to improve performance:

At some point, the index updating and writing may become the bottleneck (this depends on the data mix, very quickly with HTML or text files). As far as I can think, the only possible approach is then to partition the index. You can query the multiple Xapian indices either by using the Recoll external index capability, or by actually merging the results with xapian-compact.

Kept here to provide a point of comparison for the evolution of hardware and software, but these are obsolete.

The following very old data was obtained (around 2007?) on a machine with a 1800 Mhz AMD Duron CPU, 768Mb of Ram, and a 7200 RPM 160 GBytes IDE disk, running Suse 10.1.

recollindex (version 1.8.2 with xapian 1.0.0) is executed with the default flush threshold value. The process memory usage is the one given by *ps.

Notice how the index size for the mail archive is bigger than the data size. Myriads of small pure text documents will do this. The factor of expansion would be even much worse with compressed folders of course (the test was on uncompressed data).

The last test was performed with Recoll 1.9.0 which has an ajustable flush threshold (<em>idxflushmb</em> parameter), here set to 10 MB. Notice the much lower peak memory usage, with no performance degradation. The resulting index is bigger though, the exact reason is not known to me, possibly because of additional fragmentation