Indexing a Single Language Version of Your Multilingual Website

·Aug 04, 2025 10:00 AM·

If I have a website in multiple languages but I would only like to index one of them as a source for Dust, how can I do this? Our base website is in English and all paths are just "website.com/x/y" the translations are in the format "website.com/nl/x/y"

3 comments

· Sorted by Oldest

Remi
·
·
Tomas You can add the sub domain as starting point to the "website" connection and it will use extract only sub urls
Tomas
·
·
Yes, so that works for when there is a language subdomain, but unfortunately by default our website is in English and all English pages come without a sub-domain. So just "website.com/x/y". The translated versions always include a sub-domain, e.g. "website.com/nl/x/y" for Dutch. Hence it's impossible to only index the English pages because the crawler will always start to include the other languages as well. Anyway, for now I finally managed to just index the whole website, including all the languages, but it's close to the limit of 1024 pages so if we keep adding blog-posts it will soon run out of space
Remi
·
·
Unfortunately, I don't think there is a workaround in this case today

Remi
·
·
Tomas You can add the sub domain as starting point to the "website" connection and it will use extract only sub urls
Tomas
·
·
Yes, so that works for when there is a language subdomain, but unfortunately by default our website is in English and all English pages come without a sub-domain. So just "website.com/x/y". The translated versions always include a sub-domain, e.g. "website.com/nl/x/y" for Dutch. Hence it's impossible to only index the English pages because the crawler will always start to include the other languages as well. Anyway, for now I finally managed to just index the whole website, including all the languages, but it's close to the limit of 1024 pages so if we keep adding blog-posts it will soon run out of space
Remi
·
·
Unfortunately, I don't think there is a workaround in this case today