# How to Convert AI Models for Local Use in Your Browser  with Transformer.js

## Metadata

- **Published:** 1/2/2024
- **Duration:** 19 minutes
- **YouTube URL:** https://youtube.com/watch?v=CA_d08EIGCc
- **Channel:** nerding.io

## Description

In this video, I will guide you through the process of converting and running Transformer JS models locally. We will start by exploring the Transformer.js documentation and understanding how it uses the Onyx runtime to run models in the browser. Then, we will learn how to convert pre-trained models using Optimal and configure the local model settings. I will demonstrate the steps to clone the repository, create a virtual environment, install the requirements, and run the scripts to export the models. Finally, we will test the local model loading and make necessary configurations. No action is required from you, but this video will provide you with valuable insights into running AI models offline.

📰 FREE eBooks & News: https://sendfox.com/nerdingio
👉🏻  Ranked #1 Product of the Day:  https://www.producthunt.com/posts/ever-efficient-ai
📞 Book a Call: https://calendar.app.google/M1iU6X2x18metzDeA

🎥 Chapters
00:00 Intro
01:22 Custom Usage
02:50 Convert Script
04:24 Supported Models
05:25 Commands
07:11 Model ID's
07:58 Conversion
09:21 NextJS Client Example
11:17 Browser Testing
12:20 Browser Generation
13:36 Configuration
15:31 Browser Debugging
19:00 Conclusion

🔗 Links
https://huggingface.co/docs/transformers.js/index#convert-your-models-to-onnx
https://huggingface.co/docs/transformers.js/custom_usage
https://github.com/xenova/transformers.js/blob/main/scripts/convert.py


⤵️ Let's Connect
https://everefficient.ai
https://nerding.io
https://twitter.com/nerding_io
https://www.linkedin.com/in/jdfiscus/
https://www.linkedin.com/company/ever-efficient-ai/

## Key Highlights

### 1. Converting AI Models for Local Use

Transformer.js allows converting pre-trained AI models to run locally in a browser using ONNX runtime, eliminating reliance on remote API calls.

### 2. Key Configuration: Local vs. Remote Models

Setting 'allowLocalModels' to true and 'allowRemoteModels' to false in the worker.js file ensures the browser uses the locally stored model, improving speed and enabling offline use.

### 3. Finding Supported Models and Tasks

The 'supported_models.py' script provides a list of supported models and their associated tasks, crucial for configuring the conversion process and model loading.

### 4. Debugging with Browser Developer Tools

The network tab in browser dev tools helps verify that models are loaded from localhost instead of remote servers like Hugging Face. Clearing site data forces re-download for testing.

### 5. Expanding Beyond the Browser

Locally running models enables integration into Chrome extensions, React Native apps, and Electron apps, facilitating offline AI functionality and local data processing.


## Summary

## Video Summary: How to Convert AI Models for Local Use in Your Browser with Transformer.js

**1. Executive Summary:** This video provides a step-by-step guide on converting pre-trained AI models to run locally in a web browser using Transformer.js and ONNX runtime. It covers model conversion using Optimal, configuration of local model settings, and debugging techniques for ensuring models load from the local server instead of remote sources like Hugging Face.

**2. Main Topics Covered:**

*   **Introduction to Transformer.js and Local Model Usage:** Overview of using Transformer.js to run AI models directly in the browser.
*   **Model Conversion using Optimal:**  Cloning the repository, creating a virtual environment, installing requirements, and running Python scripts (`convert.py` and `supported_models.py`) to export models in ONNX format.
*   **Configuration for Local Model Loading:** Modifying the `worker.js` file in a Next.js project to prioritize local models by setting `allowLocalModels` to true and `allowRemoteModels` to false.
*   **Testing and Debugging:** Using browser developer tools (Network tab) to verify that models are loaded from localhost instead of remote servers and using Application -> Storage -> Clear Site Data for force re-download to test.
*   **Application Beyond the Browser:** Discusses the potential for integrating locally run models into Chrome extensions, React Native apps, and Electron apps for offline AI functionality.

**3. Key Takeaways:**

*   Transformer.js allows running AI models locally in the browser using ONNX runtime, eliminating the need for constant API calls to remote servers.
*   Utilizing the `convert.py` script is essential for converting pre-trained models into a browser-compatible format.
*   The `supported_models.py` script provides a crucial list of supported models and tasks, aiding in proper configuration.
*   Setting `allowLocalModels` to true and `allowRemoteModels` to false in `worker.js` is critical for forcing the browser to use the locally stored model.
*   Browser developer tools are invaluable for debugging model loading issues and ensuring models are served from localhost.
*   Local model hosting unlocks possibilities for embedding AI functionality in diverse applications beyond the browser.

**4. Notable Quotes or Examples:**

*   **(Regarding model ID)** "This is actually going out and looking at either uh the tasks that you are that you may want to use or the um the models specifically that they already have supported through hugging face because what happens is when you load this model it's actually making an API call to the through the browser to hugging face to pull this model back"
*   **(Local model usage)**: The video emphasizes that enabling local models opens the door for integrations with Chrome extensions, React Native applications, and Electron apps.
*   **(Debugging method)** : "what we can do is go to this application click on storage go ahead and clear your site and then when we go back to the network we'll actually see it load again"

**5. Target Audience:**

*   Web developers interested in implementing AI functionalities in their applications.
*   Data scientists and machine learning engineers looking to deploy models directly in the browser for offline use cases.
*   Developers working on Chrome extensions, React Native apps, or Electron apps that require local AI processing.
*   Anyone interested in exploring the capabilities of Transformer.js for client-side AI model deployment.


## Full Transcript

hey everyone welcome to nerding IO I'm JD and today we're going to be talking about converting an AI model to actually use it locally inside of your browser this actually comes from one of the YouTube comments on our previous video of actually using Transformer JS in the browser so now we're actually going to look at how can you use the models locally as well as some tips and tricks of using WM inside the browser with nextjs specifically and so with that let's go ahead and get started all right so the first thing we're going to do is we're actually going to look at the transformer. JS documentation so if you go to their essentially homepage on hugging face you can kind of see you know we went through installation and uh some of the usage last time so what we're going to today look at today is actually going through how to convert this to run locally and right here you can actually see that it talks about that it's using the Onyx run time to run these models in the browser uh and the best part is you can convert it uh to a pre convert your pre-trained models um using optimal and so one of the things I noticed is when you click this link it doesn't actually go to where uh it's just a dead link so if you actually click on custom usage you can come over here and what it does is it gives you uh all these settings that we're actually going to look at and the how to convert your model so today what we're going to do is we're actually going to set all of this up uh specifically to um convert a model you can see that you can Define your local paths here in the script this is the JavaScript that we'll be looking at the key things is you want to uh turn remote models off and then there's also a setting to allow local models we'll go through those configurations when we get to the JavaScript what we're going to focus on now though is actually converting these script so what you can do is you can actually run python uh and if you'll notice it has a model name or a path so you can actually point it to a model on your local the model name they actually have a list of supported models that you can uh actually download and then this is the configuration of of where the follow where the files will actually be saved to so you're just going to create a models folder and then basically a set of configuration plus your Onyx uh file all of this is going to get actually loaded into the browser so the first thing we want to do is we want to look at the the python script that you can actually get and so to do that you can go ahead and click this link it'll take you over to their GitHub page all right so now that we're over here in GitHub and we can actually see this script we'll notice that in if inside the transformers. JS it has a scripts folder and a convert file inside these scripts these are all useful tools that can actually give you different information and help you download the uh the AI model and then convert it specifically to Onyx so when I was looking at the convert file you know there's a lot of different things around the parameters and how this is actually building that you can dig into but they also have a lot of useful command lines interfaces so being able to understand like what are these parameters that you can actually Define is important one of the ones that is the most important is the model ID this is actually going out and looking at either uh the tasks that you are that you may want to use or the um the models specifically that they already have supported through hugging face because what happens is when you load this model it's actually making an API call to the through the browser to hugging face to pull this model back so that you can load it locally so if we want to actually take that model and put it in on our local machine or uh local or our web server we need to know what the model ID is the other tool that they they that I found incredibly useful is uh this supported models python it does doesn't actually show you how to use it but we're going to go through that today and what's interesting about this is it gives you a list of of all these different types of models but it also discusses the tasks so for instance this is a fil mask task and if we go back to our Transformers uh JS documentation and you scroll down to these tasks you can actually see what the ID is these IDs are what you're using in your script specifically so it's showing you the task that is associated with the model so right here we have the alert base V2 so now what we're going to do is we're going to actually take these scripts and try and run them locally and see what we can get out of it all right so I'm going to go ahead and pull up my terminal which I have a clean terminal here and a list of commands which I put together the first thing is just a clone through GitHub I've actually already done this so I'm in my uh folder already and I'm going to go ahead and create a uh environment so I'm going to use cond to do that the reason I do this is just to make sure that all of my pip installs for the requirements are contained in one place so we're going to go ahead and do that really quickly and if it already exists we can just say say yes going proceed anyways now we can go ahead and activate our Transformer you can see that right here it's changed between the two so before it was saying base now it's saying transform so now we've activated for me I'm going to go ahead and pip install the requirements here uh with a pip three for your system it could be pip um so the first thing I want to see is where am I I know I need to actually go into the scripts so I'm going to do CD scripts then do my pip install which didn't copy and paste and we'll go from there I'm going to pause while this actually runs through everything all right now that I have everything installed what I'm going to do is I'm actually going to look at the uh supported mod models that you can actually run so I'm going to take this one this script go back I need to CD out of scripts just so that I can actually run it so what it's saying is the module the scripts folder and then the python file which is supported uh modules what's cool about this it's now giving me all the scripts to actually run whichever supported model that I I want to use and it has a ton of them so if you know what you want to look for and you want to download a specific model you can actually look through and find these they are actually updating these I even saw that tiny La llama was one of the newer ones that just came out so what we're going to do is we're just going to take one down towards the bottom we know we're going to have a yellow's tiny and then we are going to try and run it locally again I'm going to let this run uh and go ahead and pause for a Time all right so now this is run uh we're just going to look at a couple of things so it's exporting to Onyx you had to have P torch which was installed on the requirements it's going out and validating the model and then it's exporting right here to this models folder uh and then doing all the layers so if we actually come back into Transformers JS and we have our folder open if you look you now have this models folder you can see it's grayed out because it's not checked into git and inside there we have this uh Yolo's tiny right and so this is important because what it's doing is it's installing it locally so now what we're going to do is we're actually going to take this and implement it into nextjs so I already have this file opened but in their examples they have a nextjs client and what you can do is you can go in and create uh a  all right so now what we're going to do is we're actually going to look at the example that they already have in Transformers JS and see how we can actually load this in they have one specifically for nextjs and the next client we've looked at this before but we're going to take a look uh again now and so if you go into the examples and your next client and you go ahead and go down to this workers. JS this is actually where we're going to be kind of looking at stuff so we're going to go ahead and get this uh this client up and running so if we do npm run Dev make sure you do an install first um but npm run Dev and then we're actually going to uh pull out this uh URL real quick I just wanted to give a shout out to the people that are commenting on the videos again we are paying attention trying to answer the questions as quickly as we can so if you haven't already please like And subscribe it helps more than you know if this is a topic that you're interested in please leave us some uh information in the comments we also have a course on this specifically uh which I'll leave a link in the description and with that let's get back to it all right so now that we've opened the URL that we launched we're going to go ahead and look at our Network Tab and this is important because what we want to see is the requests that are actually going out and so if we get our Network tab up and running when we do a command r or reload our screen we don't see anything getting imported and that's fine because we're looking at our xhr when we start typing in something we can see that we're getting 404 and that we're getting uh this Onyx model that it's trying to go out and grab and it's loading so what we're going to do here is we want to actually see how we can actually load this locally and get our WM which is over here in this tab up and running all right so we know we are going to be using this distributed fine tune so if we go back back to our uh list and we try and find our model we can see in our supported list that this is the actual model that our example is using so what we're going to do is we're going to copy this actually I'm going to use my copy just so I don't have to mine is using Python 3 so I'm going to copy this but you can get it directly from your support models of when you were uh looking earlier right this is the list all I did was pull uh pull up find grab that model it showed me this is the script or the command that I need to run so I'm going to go ahead and run this again I'm going to pause all right now that that one's completed we can take a look make sure that it is over here and we have it as well right so we have our Onyx file inside our mod models and we're going to now include this into our uh nextjs application all we really need to do is copy the models folder of the models that we have and copy them into the public folder of the nextjs app so we can you can either use a command line or you can just drag and drop I used the command line but if you look when you copy it right so whether you're dragging it or copying it through command line you now have this folder inside your uh nextjs public this will allow you to actually be able to load this model into the browser so next we are going to go to back to our worker JS which is right here so it's inside source and app and the worker Js this is actually how we're going to be loading this information in what this is doing is right now this is saying allow local models false which is why it was actually going out to hugging face in the first place we are going to change this to true so we do want to allow local models and then we're going to look at our examples of what other configurations are there so we can the other thing I want to do is I want to allow remote models as false this can technically be a backup uh but just for the sake of argument so that we can test we're going to make sure that we're only looking at uh local models so right now we have allow local models true and allow remote models false the other things that you can look at are where do I actually look for these models right now it's going to be defaulting to the uh the models path this is fine this works for us however if you wanted to change this you would change it with this local model path the other thing is you can actually tell it where the WM uh files are again it's fine where they are currently so we're just going to leave it as is um but you do have that uh that ability the other thing is this model name is probably different for you so you want to make sure that it is just the folder name that we downloaded so we looked this name matches this folder sometimes I think uh it has like xenova in front of it something like that so we just need to get rid of this just make sure that it's the the file path of our default which is models and our folder which is this molder this folder and our task which is the task classification all right so make sure you save that all right so we're now back in our browser we're going to go ahead and do a refresh uh and what we're going to look at is seeing if we can pull the the model itself in so we start and it automatically is pulling this in and it's pulling it in a lot faster so when you see this you can actually see that when it's going and looking for the Json model or the Json tokenizer it's actually pulling it from our Local Host you can see that right here if you can make it a little bigger whereas before this was actually going out to hugging face and uh pulling this information back so now we have this and it's running everything locally another thing that can be a little weird is when you refresh it's happening so quickly that can be a little bit of a false negative because it does need to actually load this into the the the Chrome browser it's doing that by a couple of different things so this time you saw that it's not actually fetching anything this time and the reason for that it's already cached right so if you look at wasm you can see that it's still pulling this information from uh remotely again we could actually pull this file in locally through the other parameters that we saw but if we wanted to load again what we can do is go to this application click on storage go ahead and clear your site and then when we go back to the network we'll actually see it load again so let's go and click on fetch here we'll refresh and you can see it's actually pulling everything in locally so there you have it this is how we can actually pull in the uh information locally or pull in the model locally and actually have it running just through some simple configurations this is really important for a couple different reasons like one it's cool on the web you know there is the downside of it loading in the beginning however what's really interesting about this is that uh you can load this in a chrome extension which is what our course is on or you can load it through react native for instance or you could load it through electron so as a local uh application is load like loading itself you could actually be loading that AI model as well and then using that AI model offline um I really think this is going to be extremely big and so again this is important in the sense that like you're using AI models in the local instance we gave you some tips and tricks specifically around uh how to work in the browser so so again all we had to do were some simple configurations right we had to actually load it in the public models and then the uh allow local models to True allow remote models to false we make sure that our model is following to our default path and specifying which model and which task we're actually trying to operate hey everyone I hope you en enjoyed this episode so what we learned today is specifically actually using the converting scripts that Transformer JS gives you access to some of the little tips and tricks of where to find models and things like that with the supported models that they have and then we also did some debugging and uh looking at like wasm in the browser so how to look at our networks tab clear our data and actually make sure that we're loading in those models locally so with that happy nerding see you again soon

---

*Generated for LLM consumption from nerding.io video library*