Clarifying Query Structure for Effective Data Source in App Development

Hi there, I’ve built an app that I use in one of my bots. The first block of the app is a data_source block, which I use to find related discussions (from call transcript files). From what I understand, the query section in the block specifies what the semantic search should target. However, no matter what I input in the query, I consistently get the same 24 chunks returned—even though there are over 100 documents with many more chunks available. I might be misunderstanding how to structure the query. I’ve gone through the documentation, but it’s still unclear. Could you provide a clear example of a properly written query so I can ensure I’m using it correctly? Thanks in advance!

16 comments

Nathys Marchal
·
·
Also cannot access this section of the doc: https://docs.dust.tt/docs/core-blocks Getting a blank page
Remi
·
·
Hey Nathys Marchal 👋 Thanks for you question ! Does the following answer from @help help? Happy to forward this to someone from the team if needed.
Remi
·
·
Also cannot access this section of the doc:
Not sure what you mean here. Could you clarify?
Nathys Marchal
·
·
The page just do not charge, it stay full white
Nathys Marchal
·
·
Yes I tried to tell him find discussions about... but whatever i say after about it returns me the same chuncks and same number of chunks, so doesnt look like working
Nathys Marchal
·
·
Even worst when I say "find discussions about companyX" 10% of the time I get 26 items return and 90% of the time I get nothing return. query isn't working well and isn"t constant. Do you have similar issues on your side ?
Remi
·
·
Can you share a screenshot of your Dust app? 🙏
Nathys Marchal
·
·
Getting 24 chuncks (all from the same document) while when I remove the query, I am getting the 800 chunks and while manually checking, most are mentionning and speaking about the query made.
Remi
·
·
Thanks ! Just to make sure I understand, why did you decide to create an app instead of an assistant with a "search" tool plugged to your data source?
Nathys Marchal
·
·
cause I have thousands of files and each file got many pages. I already try to use the search fonction directly and results are very poor. Purpose of the app is to give in sources only related topic in the file instead of full file
Nathys Marchal
·
·
which is working pretty good in front of normal search but the semantic search isn't working well at all
Nathys Marchal
·
·
The app also lets me use scripts to remove specific elements, such as salesperson speeches, which tend to confuse the results.
Remi
·
·
Ok ! I forwarded the thread to the team. Someone will jump in soon 😊
Nathys Marchal
·
·
Appreciate 🙂
Guillaume Sondag
·
·
Hello Remi, I'm looking to do more or less the same thing as Nathys Marchal I have a folder with one file per transcript. And I want to search by ID to return only the data in this file so that I can work with this information only. What did you end up doing?
Remi
·
·
Guillaume Sondag I think the best way to go about doing this is add the ID as tags on the doc using the API, then, to use a "data source" block and to add a "tag" filter with a variable for the ID. Let me know if you need more specific guidance, I can ask someone else in the team. 😊

Clarifying Query Structure for Effective Data Source in App Development | Dust Community