# Learn How ROSA Works in #Robotics  with #Langchain

## Metadata

- **Published:** 10/7/2024
- **Duration:** 20 minutes
- **YouTube URL:** https://youtube.com/watch?v=igQImiy9TM4
- **Channel:** nerding.io

## Description

Unlock the potential of AI-driven #robotics with ROSA, the AI assistant built for #ROS1 and #ROS2 systems! Powered by the Langchain framework, ROSA allows you to interact with #robots using natural language, making robotics development more intuitive and accessible.

Key Highlights:
- Communicate with #ROS based robots through natural language queries
- Supports both #ROS1 and #ROS2 for seamless integration across systems
- Built on Langchain, leveraging the power of #AI for efficient robotics development
- Simplifies complex robotic operations, saving time and effort for developers

Transform the way you work with robotic systems and elevate your development process with ROSA. Like, comment, and subscribe for more cutting-edge #robotics insights!

📰 News & Resources: https://sendfox.com/nerdingio
📞 Book a Call: https://calendar.app.google/M1iU6X2x18metzDeA

🎥 Chapters
00:00 Introduction
00:38 Turtlebot
01:34 Architecture
03:27 Demo
05:05 Navigation
06:04 ROS
08:50 ROSA
10:41 Demo
13:18 Code
18:14 LLM
19:53 Outro

🔗 Links
https://github.com/nasa-jpl/rosa
https://github.com/nasa-jpl/rosa/wiki/Guide:-TurtleSim-Demo
https://github.com/nasa-jpl/rosa/wiki/Developer-Documentation

⤵️ Let's Connect
https://everefficient.ai
https://nerding.io
https://twitter.com/nerding_io
https://www.linkedin.com/in/jdfiscus/
https://www.linkedin.com/company/ever-efficient-ai/

## Key Highlights

### 1. ROSA: ROS Agents with Langchain

Introduces ROSA, a package from JPL that integrates ROS (Robot Operating System) with Langchain, enabling the use of AI agents and LLMs in robotics for enhanced control and interaction.

### 2. Leveraging Langchain for Robot Control

Explains how Langchain can be used to send information to ROS's publish-subscribe model, effectively controlling a robot by defining tools as APIs to publish data to various topics.

### 3. Interactive Robot Debugging with ROSA

Highlights ROSA's ability to provide interactive debugging experiences, allowing users to query the robot's system (e.g., listing topics) and potentially build chat interfaces for robot interaction.

### 4. Dynamic Task Planning with AI Agents

Demonstrates how AI agents can dynamically plan and execute complex tasks (e.g., drawing a star) by understanding ROS topics and autonomously generating the necessary commands.

### 5. Customizable Robot Personas with LLMs

Shows how LLMs can be used to create customizable robot personas, including setting up greetings, instructions, constraints, and even incorporating humor into the robot's responses.


## Summary

## ROSA: AI-Driven Robotics with Langchain - Summary

**1. Executive Summary:**

This video introduces ROSA, a ROS (Robot Operating System) package from JPL that integrates Langchain to enable AI-powered interaction and control of robots using natural language. ROSA leverages Langchain's agent capabilities to understand user commands, plan tasks, and control robots by publishing messages to ROS topics, simplifying complex robotic operations and debugging processes.

**2. Main Topics Covered:**

*   **Introduction to ROSA:** What ROSA is, its compatibility with both ROS1 and ROS2, and its foundation on the Langchain framework.
*   **Turtlebot Example:** A real-world example of a robot (Turtlebot) and a previously built system to control it via Alexa and web interface.
*   **ROS Basics:** Explanation of the Robot Operating System (ROS), its pub/sub architecture (topics, publishers, subscribers), and its role in robotics.
*   **Langchain Integration:** How Langchain agents and Large Language Models (LLMs) can be used to control robots by defining ROS topics as tools or APIs, allowing for dynamic task planning.
*   **ROSA Architecture and Demo:** A detailed look at ROSA's architecture, including how it leverages Langchain's agent functionality to understand commands, plan tasks (e.g., drawing a star), and execute them by publishing messages to relevant ROS topics.
*   **Customizable Robot Personas:** Utilizing LLMs to create robot personas with personalized greetings, instructions, constraints, and even humor.
*   **Code Walkthrough:** Examination of the ROSA code, showcasing how Langchain agents are integrated with ROS, including defining tools for robot movement and utilizing LLMs for prompt generation.

**3. Key Takeaways:**

*   ROSA simplifies robot interaction by enabling natural language commands.
*   Langchain provides a framework for AI agents to control robots by interacting with ROS topics.
*   AI agents can dynamically plan and execute complex tasks.
*   LLMs enable the creation of customizable robot personas.
*   ROSA enhances debugging and interactive experiences with robots.
*   The use of Python in both ROS and Langchain facilitates seamless integration.
*   This tool could also potentially use other LLMs besides OpenAI, such as Claude.

**4. Notable Quotes or Examples:**

*   "We can actually invoke through our agent to show a list of topics that have Publishers but no subscribers so you can actually ask information about your robot and its system which is incredibly awesome because previously you'd have to either manually or programmatically build in very specific questions."
*   "If you think of move forward as the function of an API and your tool and it's saying based on this NLP – I know I need to move a distance of two units."
*   Example of drawing a star: The AI agent autonomously plans the steps, including calculating angles and distances, to execute the drawing task.
*   Example of using a LLM to include a funny turtle joke when it greets the user.

**5. Target Audience:**

*   Robotics developers working with ROS (ROS1 or ROS2).
*   AI/ML engineers interested in applying Langchain and LLMs to robotics.
*   Researchers exploring AI-driven robot control and human-robot interaction.
*   Anyone interested in learning how to integrate AI with ROS for more intuitive robotics development.


## Full Transcript

hey everyone welcome to nerding IO I'm JD and today what we're going to be going through is Lan chain and Robotics so what we're going to be looking at is a package called Rosa which is based on Ross Robot Operating System and how we can actually use agents or AI agents and llms inside robotics we're going to use a system like this this turtlebot and then we're actually going to go through a previous project and see how we would wire it up with Rosa today and with that let's go ahead and get started all right so first what we're going to do is we're going to talk about what is a turtlebot so a turtlebot is basically a uh it's essentially an open source robot package you can actually pay for a kit through robotis and what it has is it has a lar so basically a spinning laser on top that shoots out and then calculates distance based on information that's coming back you can add things to it so like on mine I I added uh multiple different cameras and different types of sensors uh all of these pieces you can actually 3D print it has it runs off Raspberry Pi and then these pieces are basically your uh movement so you have this uh core and basically that controls two uh Dynam dynamical wheels that allow you to turn and uh move and so you can control it like like your uh all your positioning this way so in the past put together an architecture uh where we were taking inputs from Alexa or from a web site that could actually connect to this uh system through realtime communication and that's through websockets and a protocol called mqtt and so what each one of these is doing is it's creating essentially a pub sub model or an observer pattern that is sending information to a central controller that is essentially listening to multiple different topics and passing that information uh accordingly so on the website you can use almost like a a handheld controller and then that will send different commands for our finite control and on the Alexa you could basically say start or go to this particular position and because you have already had like a room mapped for instance you could actually take that information and then tell the robot to go to a specific location once it's in this controller it'll actually send it through an mqtt Bridge which uh all of this then runs on Ubuntu and a Ross server so Ross we're going to go into a little more detail about what Ross is but it's robotic operating system you can then tell send all of your robot Telemetry so how you would turn your wheels and what you can control your speed how to navigate through something called slam and then individual commands or even understand different things like battery so for instance if you're battery was running low return to your self-charging station so this is uh something that we built a few years back but I just wanted to kind of show what is actually happening hi Welcome to our demonstration of our turtlebot autonomous Annie today we're going to show you how we control the turtlebot in two separate ways using voice control um through Alexa as well as uh a visual interface uh that we made with VJs so the first thing the voice control Alexa open turtlebot greetings from autonomous Annie you can like start autonomous Annie Annie start autonomous Annie start Annie stop econom Annie stop Alexa cancel goodbye so as you can see we can um start the the turtlebot as well as stop the turtlebot using voice control and we can also transition over to our uh UI and control using this joystick the movement of our turtlebot we can stop go backwards stop and So based on this what was actually happening is we were sending information from Alexa to a Lambda that then sends it to the iot and again goes into the nqt bridge and so what how all of this works and when we're talking about a navigation is basically you have a map which is a already kind of predetermined so the robot can know where it is essentially that's like the awareness of uh doing uh like robotic navigation because you've already mapped out your environment makes it way more complicated when you the environment is unknown and so once you have this environment mapped you can actually have different spots on the map where it's predetermined of how to go so even though you can place the robot wherever it's trying to self Orient itself uh as by all these green pixels and the liar of where could it possibly be on that map and then when you give it a command it's going to figure out how to autonomously get there based on uh where it is and what it seems in case there's any obstacles in the way so what is Ross Ross is against Robot Operating System and it's uh open source you can use Python you can use C and it's using Pub slub um subscription so you you have this idea of topics that you can publish and sub subscribe to if you've used anything like um SNS uh it's very similar concept but the important part is that we can use Python uh because as we know Lane chain is built on Python and JavaScript and so what we're going to see is we're going to start going through how can we actually leverage Lang chain to send certain type of information into our publish And subscribe in order to control a robot first we're just going to kind of look at some of the basics of what is involved in Ross So Ross executes on um different communication nodes so that you can listen to or run whatever the package is and then its node name it communicates through things uh like topics so again we're going to it's it it is not in this regard it's not a websocket right a websocket is an open connection Ross topics are fire and forget so you basically just send the message to wherever is listening and whoever is listening will take that message and execute and that's important because when we're talking about having uh a controller are knowing all of the topics that we have and then sending that information that's how we actually determine the Telemetry of how the robot operates so again it's it's all built on a pubsub or an observer pattern where you can distribute uh messages over topics so you can kind of see this is a topic and this is a topic and then this is the also a topic but it's the topic and the uh the the subscription that's getting published so you can receive multiple messages over topics and multiple subscribers can be to a single topic so you basically have a bunch of different topics and then messages are all being sent in an event driven fashion to those topics to in from those topics there are other things called services and then there's also things that are called tools that you can kind of custom and it customize and build so you can see you could build a custom calculator and that's where we're going to start talking about Rossa which is uh just which was released by JPL so through NASA and the jet propulsion labs to understand how we can actually use Lane chain inside robotics real quick everyone if you haven't already please remember to like And subscribe it helps more than you know and with that let's get back to it all right so now what we're going to do is we're actually going to look at Rossa and or Rosa and what we're what this is is this is a Ross agent and again it's from uh JPL but the really cool thing is it supports Ross one and two and it's built on Lang chain so we're going to take a look at what it can do and how we can adapt this to our turtlebot and so for my turtlebot that his name is Gus but we we are going to look through some examples so right here this this is really interesting because we can actually invoke through our agent to show a list of topics that have Publishers but no subscribers so you can actually ask information about your robot and its system which is incredibly awesome because previously you'd have to either manually or programmatically build in very specific questions in in in order to understand things like B life because that always comes off of uh your publish and subscribers you could actually use this for debugging potentially and have a much more interactive experience with your robot we could even in theory build a chat interface to uh work with this robot so what we're going to do is kind of look at the demo and then we're going to go through a couple different pieces so the first thing is that we're using Lang chain Lang chain chain core and open AI you could swap this out for uh another model potentially because again it's On LAN chain and let's take a look at at first what they built so this is the demo that they have and we'll kind of walk through what's actually happening so first we have our Master robot UI and we are telling it to draw a f draw a 5o star we're starting our agent and right here this is really important because we're invoking these functions that are actually inside of our Ross and our robot environment it's aware enough to figure out on its own based on its own system how to drive a fivepoint star so the thing it's going to figure out first is I need to move forward in a line I'm going to go to appropriate a goal to make a fivepoint star and then repeat that five times it's also actually doing the radial degrees to figure out the math of how to affect its coordinates so X Y Theta linear and angular so you can think of this as where is it when does it need to turn how fast does it need to turn each wheel in either remember there's only two wheels in order to turn uh a POS and reposition itself so this is previously we would have to send all of this or create a function that would do just this part and it's coming up with it on on the fly so now it's telling it's the plan it's going to start moving the fact that it is is coming with this up with this plan on its own and then proceeding with the commands right so it's actually sending this information you saw that Json package happening before to the topic in order to publish information again it's just rep repeating itself and it's drawing a star on its own so for me and also gives you another uh chat to work with so for me this is just so cool because it allows you to interface with your robot in a totally different way it alleviates some of the uh challenges with trying to understand more about your environment and uh you still have to figure out like slam on your own and and those those kinds of things but just the fact that it can take in and understand all the topics that you might have and what's happening with your robot through publish And subscribe it is just amazing to me so definitely going to play with this a bit more um it's not crazy difficult to set up especially if you've done some Ross before um again this is all on Ubuntu so you need to make sure you have an Ubuntu machine or um the machine that allows you to uh create a VM but they have a a great demo uh and there's even like a streaming argument so you can for latency this will definitely help with latency and they give you different uh examples so what we're going to do is we're just going to kind of look through the code and figure out what's going on with this particular agent so as you can see again so cool they are using a list of Lane chain tool functions to actually create the connections for Ross that's just so awesome and this is how you would put together a tool to move forward right very very similar to like what we were doing before but it's not as Dynamic so you're giving it the ability to understand how to move and this it looks like they're even just doing uh a template for a pro here to be able to know how to actually move forward what kind of tool it is and then just passing this invocation to understand this is really what you need based on this right so if you think of move forward as the function of an API and your tool and it's saying based on this NLP uh I know I need to move a distance of two units so let's uh dive in and take a look at the the the code itself so again so awesome that we're actually using Lang chain agents as a tool we're importing Rosa as a I really hope that's a reference to the Jetson uh but uh pulling in our tools which we're going to look at the Custom Tool and then some of our other commands right so we have our help commands we have our llm so again you don't have to use open AI like they did in this example but you could uh you could use other ones nice they have a cool Turtle uh Tool uh just letting you know that um how cool this uh Turtle tool is and so what's what's actually happening is we have the ability to uh give our or to have examples so these are just example prompts we have our commands so we can help clear get our examples and then our blastoff functions and basically instructions of what we're actually going to be doing and looks like uh this part is specific to um almost like a system prompt so again it it has some uh things here for like greting always fun to to bake stuff into your uh to your Bot and here is where we're actually going to run all of our our agent but let's actually take a look at how they're using Lane chain agents and the tool specifically so if we take a look at their uh docs this is how you can create a tool so again we saw some of their examples with the move forward this is a uh just an example of their example and we're going to look at tools and turtle and look at the the uh the agent itself so right here this is telling us what our uh command velocity publish and is our ability to remove ourselves from that uh that topic this is actually defining and adding turtle one as the uh name as well as the publisher for the velocity and putting in our twist and our Q size this is how we're actually going to stay within our bounds and this is doing the uh publishing and checking and actually doing the ma math to calculate how to accomplish that task then there are basic likely utilities so this is for like the turtle Sim so we need to spawn where the turtle is and understand where it is and then obviously kill command as well as a clear command this is also really important where are we so where are we in the position and defining our teleport so the other things that I want to go through are just kind of look at the llm and just kind of show you what they're using is uh they have two different types of credentials so you have your credential set here they're actually using using Azure and then it's literally just set up the same way that you would set up an llm in Lang chain so you could think about uh testing with this with um Claude or something similar and then the prompts right this is just a normal prompt it's giving uh a embodiment and a personal Persona you got to you know you got to name your robot make sure that it's it's doing for positive things uh love this include a funny turtle joke so definitely got some built-in Easter eggs in here in their prompt which is awesome and but there's different prompts that are doing different things the critical instructions your constraint anded guard rails so even putting in moderation and then about your abilities so in this one it's specifically talking about generate drawing shapes right like this is our hello world example it's drawing shape for the turtlebot so awesome I mean again I'm going to be playing with this more but uh super excited to see Ross and uh AI Agents come together uh it again it works very similar to how you would think of tools with an API and if you can make that distinction and then publish that information to your topics that's how you're now controlling a robot all right everyone that's that's it for us today we met Gus our turtlebot as well as how we can kind of build for robotics with L chain in the future how their subject or their subscrib in topic model works and with that happy nering

---

*Generated for LLM consumption from nerding.io video library*