SOLR large data indexing help
Closed - This job posting has been filled and work has been completed.
We deal with huge solr data (huge rows - 10 billion, but only small size - 200GB) We are trying to index it to solr in fastest way from mysql DB.
The solr importer is custom and works very well, and mysql works very well and is not the bottleneck. The issue is is solr does not commit the documents fast enough. it hangs and then stops after 100 million rows.
We can easily get 100,000 rows per second from all parts, apart from the actual submit / index to solr. More information can be found here:
We need urgent help and advice from an expert who has used solr to index these sort of speeds / data before.