solr - Run incremental crawl using NCrawler -
When we use NCRLL with SOR, is there any way to run incremental crawling and indexing? I crawl my crawler every time, so do not want to get complete data. Is there a way to make crawl intentment?
Thanks in advance.
Nothing has been built in this for NCRWaller, you will need to make your processing to handle it. . However, the Extensible IPipelineStep mechanism will allow you to make any process around your crawling which you want. For example, when pages change and only the converted pages can be stored on the index, they can store each URL in the database with the content of the page's content.
Comments
Post a Comment