Request for Granular Control of Web Page Crawling and Storage

·Jul 30, 2025 01:05 PM·

Can we get more granular control of the pages that are crawled and stored when a website is used as a source? I would appreciate the ability to delete certain subpages that the crawler found but are wholly irrelevant. Cheers!

💡1

1 comment

· Sorted by Oldest

Tomas
·
·
Also, it seems that currently it's not possible to let the crawler start at a certain depth of a page - is that so? So for example I would like to navigate to a certain sub-section of a website and only load all the sources in that subsection and up to 2 levels deeper.

Tomas
·
·
Also, it seems that currently it's not possible to let the crawler start at a certain depth of a page - is that so? So for example I would like to navigate to a certain sub-section of a website and only load all the sources in that subsection and up to 2 levels deeper.