# How to use multiple languages in VAPI

## Metadata

- **Published:** 6/3/2024
- **Duration:** 17 minutes
- **YouTube URL:** https://youtube.com/watch?v=009YgFNkM5w
- **Channel:** nerding.io

## Description

Continuing from our previous video, we will delve into multilingual support for your AI voice bots. 🌐✨

In this part, you'll learn how to:
- Enable multilingual capabilities in your VAPI voice app
- Configure language detection and translation features
- Enhance user experience with seamless language switching

Stay tuned as we explore how Vapi and Next.js can help you build a truly global application, reaching users in their preferred languages. Don't miss out on this exciting journey into the future of AI-driven communication! 🚀

🔔 Make sure to like, comment, and subscribe for more tutorials and updates!

Course: https://forms.gle/PN3RpKD8eHsicL9Z7

📰 News & Resources: https://sendfox.com/nerdingio
📞 Book a Call: https://calendar.app.google/M1iU6X2x18metzDeA

🎥 Chapters
00:00 Introduction
00:37 Vapi Documentation
03:26 Vapi Dashboard
06:46 11labs Setup
07:56 Demo
09:19 Local code
14:11 Dynamic Language Locale
15:14 Twilio
16:25 Bonus

🔗 Links
https://docs.vapi.ai/multilingual
https://dashboard.vapi.ai/
https://console.deepgram.com/
https://elevenlabs.io/app/voice-lab
https://www.twilio.com/en-us/guidelines/br/voice
https://www.twilio.com/en-us/guidelines/br/regulatory
https://docs.vapi.ai/phone-calling

⤵️ Let's Connect
https://everefficient.ai
https://nerding.io
https://twitter.com/nerding_io
https://www.linkedin.com/in/jdfiscus/
https://www.linkedin.com/company/ever-efficient-ai/

## Key Highlights

### 1. Multilingual Support in VAPI

VAPI supports multiple languages for both transcription and text-to-speech. Configuration can be done in the dashboard or dynamically within the JSON object.

### 2. Transcription Providers: Deepgram & Talkscriber

Deepgram and Talkscriber are mentioned as transcription providers. Deepgram stands out because of its HIPAA compliance features.

### 3. Text-to-Speech Providers: Options Galore

Various TTS providers like 11Labs, Play.ht, and Azure are discussed, with 11Labs highlighted for voice cloning.

### 4. Dynamic Voice Selection in Code

Voice IDs can be dynamically passed in the code (e.g., in a Next.js app) to change the language and accent on the fly, allowing for personalized user experience.

### 5. Connecting VAPI to Twilio for Phone Numbers

Integrating VAPI with Twilio allows for inbound and outbound calls with localized numbers. Considerations include ISO codes and local regulations.


## Summary

Here's a summary document based on the provided video information:

**Video Summary: How to Use Multiple Languages in VAPI**

**1. Executive Summary:** This video tutorial builds upon previous content, demonstrating how to implement multilingual support in VAPI voice applications using providers like Deepgram, 11Labs, and Twilio. It covers configuring language detection and translation both in the VAPI dashboard and dynamically within the code, showcasing how to create a more global and user-friendly AI voice experience.

**2. Main Topics Covered:**

*   **Multilingual Support in VAPI:** Enabling and configuring multiple languages for transcription (speech-to-text) and text-to-speech (TTS) in VAPI applications.
*   **Transcription Provider Configuration:** Configuring Deepgram (focus on HIPAA compliance) and Talkscriber.
*   **Text-to-Speech Provider Options:** Discussing options like 11Labs (emphasizing voice cloning), Play.ht, Azure, and Deepgram.
*   **Dynamic Language Switching in Code (Next.js Example):** Implementing dynamic voice selection using voice IDs in a Next.js application for real-time language and accent changes.
*   **Integration with Twilio for Local Phone Numbers:** Connecting VAPI to Twilio to facilitate inbound and outbound calls using localized phone numbers.
*   **Navigating Regulatory guidelines:** Understanding and meeting the requirements to acquire local numbers

**3. Key Takeaways:**

*   VAPI offers flexible multilingual support configurable both through the dashboard and dynamically in code.
*   Deepgram is a strong transcription provider, especially when HIPAA compliance is needed.
*   11Labs enables voice cloning to add unique options to TTS.
*   Voice IDs can be used to dynamically select languages and accents within the code, improving personalization.
*   Integrating with Twilio allows VAPI applications to use localized phone numbers, requiring adherence to local regulations.
*   Utilize the "Ask Vapi" button for troubleshooting, which uses a custom chatbot.

**4. Notable Quotes or Examples:**

*   "The two things that it specifies are you need to have your transcribers and you need to have your voice so your text to speech."
*   "You can configure this in the dashboard which would almost mean that it's hardcoded or you can actually configure the multilingual support in uh the Json object that you're actually sending"
* Code snippets and examples are demonstrated using Next.js.
*  The Ask Vapi chatbot tool is a great resource to use when working with the API

**5. Target Audience:**

*   AI Voice Bot Developers
*   VAPI Users and Enthusiasts
*   Next.js Developers integrating with VAPI
*   Individuals interested in building multilingual AI applications
*   Developers who are familiar with using Vapi and Next.js


## Full Transcript

hey everyone welcome to nering IO I'm JD and today's video was actually inspired by one of our subscribers comments specifically around how to use other languages in vapy and then connect it to AO phone number so what we're going to look at is how you can set this up both in vappy and then with your either transcriber or Voice through with different languages with that let's jump into it all right so the first thing that we're going to do is we're actually going to go to the VY docs and we're actually going to see what it says for doing multilingual support inside VY so different languages so the two things that it specifies are you need to have your transcribers and you need to have your voice so your text to speech so when we're looking at the transcribers there's really two deep gram in talk scriber I haven't actually used talk scri scriber um I've used deep gram a lot in a hand full of different applications one of the great things about deep gram is that um it's it can be hippoc compliant so when you think about like if you're doing a customer support for a dentist office which I see a lot of uh tutorials around like technically you would want some Phi and some hippo compliance when dealing with setting up the those kinds of appointments so uh definitely keep that in mind when you're looking at these uh the others is um voice so text to speech so there's a bunch of different players and providers that you can use uh play HT a really great one use that a couple of times 11 Labs I think is probably like one of the most well-known uh this is great for grading your own voices we're actually going to look at them today as well as an example rhyme AI I haven't used uh deep gram again so you can use deep gram both for text of speech and for uh transcribers uh then open AI Azure uh and element so this one's also really cool I like using this one when you need to show almost like a range of emotion uh so that's that's really awesome there's also neats uh which I haven't used before so there's two ways that you can do this you can configure this in the dashboard which would almost mean that it's hardcoded or you can actually configure the multilingual support in uh the Json object that you're actually sending so we're going to look at both ways and you can even see here that to blow this up to uh set the minimal support you no longer need to specify the desired language when configuring the voice assistant so you don't have to do this inside the dashboard anymore and that's what this is kind of showing is that you can actually take this part of the object and put it in the assistant when you actually start your uh code or in the Json object itself in this example they're showing Azure and a voice ID and so we're going to replicate this in uh 11 Labs because we can pull the ID from there so the first thing that we need to do is we need to go over to our dashboard you want to make sure that you have two Prov your two providers set up so you have 11 labs for is what we're going to look at have deep gram make sure you grab your API Keys um one cool thing about deep gram even though I'm not sponsored by them in any way they give you a $200 credit again I just really like their service used it a handful of times uh so if you're interested in learning more about deep gram happy to do a video on them uh but what we're then going to go into is our assistant and so again there's ways to like config figure the assistant itself in the dashboard and so we have our ability to talk here I just put together a test and the first thing is is we're actually putting in the first message that we expect to see from the model in Portuguese now I don't know Port Portuguese uh but that was the um the phrase that I used from chat GPT as an opener again this is coming from one of our subscribers of how to set up a uh fappy client using Portuguese in Brazil so we're going to do our best the other thing is you can say in the prompt to speak only Port Portuguese again like the documentation is saying you don't have to do this anymore because this would mean that you're always going to speak in Portuguese and there may be uh the ability to to kind of switch you can actually talk to your agent and ask it to speak a different language and it'll start doing that the other piece is to set up your transcriber so if we come in here and look at our transcriber right here we have our provider of deep grammar transcriber or talk scriber and then we have our languages so you know it's usually set I think by default on English but we can pick any number of languages that we want our transcription to come out as so I'm pretty sure Portuguese is PT so that's what we're going to put here and then we can go into our voice and so we'll go through our first demo and what we're going to just do is use deep gram and use uh asria and we'll just do a quick test to and hopefully you can hear this see how it will come across in Portuguese icial cool so it's it is actually speak speaking Portuguese it even though we haven't specified a particular type of voice uh it's able to figure that out through the model itself right the other piece that we're going to do is now we're actually going to take this provider and we're going to pick a voice that is specific that specifically speaks uh Portuguese and has like a a Brazilian accent so if we go in here and we change it to 11 Labs there's this option to add the voice ID uh manually if we go into 11 labs and what we're going to do is first you need to go to like your voice library and you can filter by uh language here so we Portuguese selected and this is going to pick all the voices that uh we have available in Portuguese I just picked this one um what you need to do then is you uh you'll have this add to voice lab again I've already added it so we're going to go to our voice lab we can see right here this is our voice and then all we need to do is click to copy the voice ID it's going to be really important that we remember what this is or we're at least able to get it for two reasons one we're we're going to put it in the dashboard but then we're also going to put it in our code so that we can make this a uh a dynamic um voice that can be selected in our code so let's go back to the dashboard and we'll go ahead and put our voice in and you can see that it's pulling this directly from the uh The Voice from it's synced to our 11 laps so now let's go ahead and we'll run this again I'm just going to publish just for good measure and we'll go ahead and do a new testal can you speak in Spanish all right and so there you have it we have the ability to speak in Portuguese with a Brazilian accent it sounded much better than like the straightforward like deep gram uh voice because this is actually trained in this particular uh uh language and accent awesome so now what we're going to do is we're actually going to take that voice ID and we're going to look at the previous project that we had where we're actually going to implement this voice ID as basically a dynamic voice that we can leverage uh in nextjs real quick everyone if you haven't already please remember to like And subscribe it helps more than you know again this video is inspired by your comments so I'll keep doing my best to answer your questions and put together more contact for you with that let's get back to it all right so we're going to take a look at the same code base that we had uh from last week or the last video where we're basically taking their demo example for the Broadway show assistant and it's all written in next nextjs and we'll take the voice and we're going to swap it out for Portuguese and so the way that we can do this is if we look in our code there is this assistant uh create assistant dto so this is in your assistants and then assistant uh typescript file and if you look this assistant dto has like all your models the different type of voices so we have like 11 labs and we uh can take this information including like the voice ID so if you come up here to 11 Labs you'll see the uh voice ID right here so we can put in the voice ID that we have with our account it's just showing like the the provider specific ones but we're able to use the the assistant or the the voice ID that we grabbed from our dashboard and so what we're going to do is if you look at the bottom of this file you can see we have hard-coded values right we have our voice id id this one right here this Paula and this is our Portuguese ID so we're just going to do like some hardcoded example and then we're actually going to go through and show you how to make this Dynamic so if we go back to our example and we go ahead and click our uh start speaking we'll see that we're calling out we're actually getting our success calls hi I'm Paula welcome to Broadway shows how are you feeling today and we've got Paula's voice and if we save this to the voices that we just switched we should get our Portuguese voice hola B hour shows Bradway and so you can see and I know my face is kind of in the way but right here we're actually getting uh Portuguese so what's really cool about this is we can actually make this Dynamic on the Fly and so what we're going to do is we'll go ahead and and hardcode this back to English and we're going to pass these values as if we were uh maybe defining this web page by the local so just like you could start speaking in Portuguese or Spanish or German or or any language to actually initiate the the AI to understand to go to a different multilanguage or lingual support we can actually do that in code so if we come in here I've added a few things and what we're going to be doing is setting our local right now we're just going to use a select statement but we could actually do this in a dynamic fashion where we're actually looking at the uh local um maybe that's def find in the browser or maybe we're actually setting this through like a nextjs route what we can do is once we turn this on we have our Set uh our set local so we just put in our our values for English and and Portuguese we're passing the local right here in our assist button we're actually going to send that information down to our our toggle uh which is right here so we're sending the toggle call with our local we're actually going into our our hook here so our use uh V and if we have our toggle call we're actually going to do start or stop on start we're going to pass the local and then on the uh initiation of the VY start we're actually just going to say we're going to Define our message right again we could really make this Dynamic so you could address the user if they were logged in you could personalize this however you want and then we're going to pass our ID which is again this is our Portuguese voice all right so let's go back and let's do two different tests we'll start here in English to make sure that we still have Paula hi I'm Paula welcome to Broadway shows how are you feeling today cool and now we will go for Portuguese shows the Broadway EST all right and there you have it we actually have both the English and Portuguese working in the browser using the dynamic create assistant with our Jason object so the last thing we're going to look at is just really quickly uh how you can kind of find if we wanted to connect this to a phone number how would you do that and so the way I would think about doing this is if you go to twilio they have guidelines for for specific uh ISO codes right and so the local is Brazil and you have your ISO code and then you can actually look at like what do you actually need in order to be able to create a new phone number and attach it to twilio so that could be like your inbound outbound they have specific guidelines though for each uh each internationalization right so can it be INB outbound um based on where you're at the other thing I would look at is like the local uh regulatory so getting local numbers um and looking at what you actually need in order to get this set up in twilio we're not actually going to go through this because I I wouldn't be able to pass uh the identification process but there's also a great resource in vapy which really talks about like how to set up a number doing inbound and outbound calls and actually importing these directly into uh vapy the other cool thing and I know this is like right down by my face uh so hopefully you can see it we'll actually do this real make it a little little easier is uh there's this ask vappy button and this is actually pretty cool to um if you have specific question so I used it previously just to uh ask about llms you could say import uh Brazil tooo number and let's see what we get it's actually running this through uh a version of chap I'm pretty sure this is a custom GPT they if you go to their Discord they're they're actually really active they uh they someone had a a custom GPT that you could actually use I think this is following something similar so it actually read through the API interface or reference of how to do it and it's actually coming back with an example of how to do this in by curling it uh so it gives you code examples for what uh what you're expecting which I thought was really awesome so if you get stuck great little tool that they came up with um and that's how I would go about uh setting up a uh a too number all right everyone that's it for us today again thanks for the comments that inspired this video if you haven't already please remember to like And subscribe if you have any questions we'll see you in the comments happy nerding

---

*Generated for LLM consumption from nerding.io video library*