Understanding Page Too Large Error: Size Limits and Solutions

·

Hello, when I try to add a website as data source, I get a "page too large" error. What's the size limit? Would it be possible to increase the limit a bit? Thanks!

8 comments

Remi
·
·
Thanks for the question! Would you agree to share your page? And maybe a screenshot of the error message?
Damien Laborie
·
·
Damien Laborie
·
·
https://xrp-ultimate.cegid.cloud/90565515dev/hlp/qualiac/fr/sac/html5/parametres_val_sac.html
Remi
·
·
Thanks ! I forwarded to the team. Someone will get back to you on Monday 😊
Jules Belveze (Dust)
·
·
Hello Damien Laborie, we indeed have a heuristic to prevent from crawling webpages that are too big. I manually inspected the page and indeed it is very long. Unfortunately we cannot easily bypass this rule 😕
Remi
·
·
If it is just this one page Damien Laborie , have you consider extracting it in other ways and pushing it to Dust in a data folder?
➕1
Damien Laborie
·
·
I have a couple of pages... I guess I will scrap this site to create my own site and then plug it to Dust...
Remi
·
·
I understand this is not optimal :s I'll keep an eye open in case this is a feedback that comes often 👀