Building a Voice Agent with Dust: Advice and Documentation Needed | Dust Community

Building a Voice Agent with Dust: Advice and Documentation Needed

Hi team! Any advice/documentation for building a voice agent with Dust? I'd like to use VAPI or Eleven Labs and n8n/Pipedream to really 'chat' with our knowledge base. I've seen people on YouTube get creative with how they handle the loading step so it makes sense in a voice context (e.g. "Ok, I understand. Let me think about it... I'm currently blah blah— ready, here's my answer XYZ").

👀1

7 comments

Remi
·
·
Hello Juan Felipe Campos 👋 It is not currently on the roadmap to add voice input in Dust. The team expects some release from OpenAI and others soon-ish which would make the integration much simpler. I tried elevenlab voice agent feature and it's pretty interesting. They also give you the ability to add some docs and manage RAG on their side. It won't give you Dust-level of accuracy & depth of search but it will make a good POC until voice is available in Dust 😊
Juan Felipe Campos
·
·
Hi Remi, sorry rereading my question I see it wasn't well phrased: I want to use n8n/Pipedream to connect Dust to VAPI. This way I can have a "call" with my knowledgebase. Any advice on how to do this? I can think of a few solutions (e.g. using the Dust n8n community node or a webhook to Pipedream or a Dust app) but am not sure if you have any thoughts on what would be "best" from an experience perspective. For example, I don't think the Dust n8n community node reveals the agent's thought process but it's possible that the Dust app does. Between the two, I would choose the option that reveals the thought process as it would make a more natural voice chat experience. Does that make sense?
Juan Felipe Campos
·
·
So these are my options:
1.
n8n Dust/VAPI - Pros: fast to set up Cons: slow to get a response so that would mean waiting some time for the voice to respond
2.
Pipdream Dust/VAPI - Pros and Cons: same as above (I think but maybe not as it's possible that the webhook does reveal the thought process?)
3.
Dust app - Pros and Cons: Unknown
My question I suppose would be— any thoughts on my thought process above? Your context on each of the three capabilities would save me a lot of time of setting all three options myself to compare them.
Remi
·
·
The tricky thing is that to have a smooth experience, you'll probably want to have something that looks like the "real time" API from openAI. It isn't the case for the Dust API, I expect there will be quite a bit of lag between you talking and your agent answering - but from what you mention, it looks like you already saw those limitations. I don't have any "quick fix" unfortunately. Maybe there are tools that can act as the main agent and to which you could plug your Dust agent as one of the "tool" that it can call and access. It would call Dust only when needed and have a more fluid convo experience for the rest. But I don't know if it's possible in any of the tool you mentioned 😊
Juan Felipe Campos
·
·
This is super helpful. Exactly what you're mentioning is possible and I'll build it like that - I'm sure it will have it's hiccups and disadvantages but this is already directionally the way to do it. Thanks so much!
❤️1
Juan Felipe Campos
·
·
Remi I did it and it works well! Wanted to update you on how it turned out. It's basically a VAPI voice bot running Claude as the main AI for having quick conversations. It has an n8n webhook set up as one of its tools, so whenever something knowledge base-related comes up, the VAPI bot calls this webhook which is then connected to Dust. I got a VAPI phone number for this bot which means I can call it any time and chat with my Dust data in a really user-friendly way. Thanks again for your help and hope this use case is at least interesting to know about.
Remi
·
·
Wow that looks great! thanks for the update 🙏 If you have a chance to share a quick video in 🔒[private feed] that would be awesome. I am sure people would be really interested to learn more.