Dust Community Icon
Join the Community
Help

Understanding Page Too Large Error: Size Limits and Solutions

Avatar of Damien LaborieDamien Laborie
·
·

Hello, when I try to add a website as data source, I get a "page too large" error. What's the size limit? Would it be possible to increase the limit a bit? Thanks!

8 comments

  • Avatar of Remi
    Remi
    ·
    ·

    Thanks for the question! Would you agree to share your page? And maybe a screenshot of the error message?

  • Avatar of Damien Laborie
    Damien Laborie
    ·
    ·
  • Avatar of Remi
    Remi
    ·
    ·

    Thanks ! I forwarded to the team. Someone will get back to you on Monday 😊

  • Avatar of Jules Belveze (Dust)
    Jules Belveze (Dust)
    ·
    ·

    Hello Damien Laborie, we indeed have a heuristic to prevent from crawling webpages that are too big. I manually inspected the page and indeed it is very long. Unfortunately we cannot easily bypass this rule 😕

  • Avatar of Remi
    Remi
    ·
    ·

    If it is just this one page Damien Laborie , have you consider extracting it in other ways and pushing it to Dust in a data folder?

    ➕1
  • Avatar of Damien Laborie
    Damien Laborie
    ·
    ·

    I have a couple of pages... I guess I will scrap this site to create my own site and then plug it to Dust...

  • Avatar of Remi
    Remi
    ·
    ·

    I understand this is not optimal :s I'll keep an eye open in case this is a feedback that comes often 👀