# How to Build Multi-Agent AI Systems with OpenAI Swarm & Secure Them Using Portkey

## Metadata

- **Published:** 10/29/2024
- **Duration:** 23 minutes
- **YouTube URL:** https://youtube.com/watch?v=9qLkAEJol9A
- **Channel:** nerding.io

## Description

In this video, I demonstrate how to create a multi-agent AI system using OpenAI Swarm, an educational framework for managing collaborative AI agents, and secure the system using Portkey, an AI Gateway designed for enhanced observability and security.

Here's a breakdown of the setup:
- OpenAI Swarm coordinates AI agents to tackle complex tasks.
- Agents work together through defined hand offs to achieve shared goals.
- Portkey ensures secure, test oriented,  agent activity.
- Portkey provides real-time insights into AI decisions.

This approach enables building robust AI solutions that are transparent and secure, making it ideal for educational projects, research, or any scenario where multi-agent systems are needed. By combining OpenAI Swarm and Portkey, you get the best of both multi-agent management and top-tier security.

Some potential use cases include simulations, collaborative AI research, or secure AI-driven platforms. The educational framework empowers developers to experiment, while the AI Gateway keeps your operations safe.

📰 FREE Code & Resources: https://sendfox.com/nerdingio
👉🏻 Ranked #1 Product of the Day: https://www.producthunt.com/posts/ever-efficient-ai
📞 Book a Call: https://calendar.app.google/M1iU6X2x18metzDeA

🎥 Chapters
00:00 Introduction
00:40 OpenAI Swarm Overview


🔗 Links
https://github.com/openai/swarm
https://portkey.ai/
https://cookbook.openai.com/examples/orchestrating_agents

⤵️ Let's Connect
https://everefficient.ai
https://nerding.io
https://twitter.com/nerding_io
https://www.linkedin.com/in/jdfiscus/
https://www.linkedin.com/company/ever-efficient-ai/

## Key Highlights

### 1. OpenAI Swarm for Multi-Agent Systems

OpenAI Swarm provides an educational framework for building and orchestrating multi-agent AI systems with handoffs and function calls between agents.

### 2. Portkey: AI Gateway for Observability & Security

Portkey acts as an AI gateway, offering observability (logging, cost tracking), security features (guardrails, key rotation), and configurations like Canary tests for AI applications.

### 3. Custom Client Integration

OpenAI Swarm allows passing custom clients (like Portkey), enabling the use of AI gateways by modifying the base URL and headers without significant code changes.

### 4. Agent Design & Transfer Functions

The video demonstrates structuring agents with specific tools and transfer functions, enabling communication and task delegation within the multi-agent system. Agent can act solo or as part of the swarm.

### 5. Load Balancing and Canary Tests with Portkey

Portkey enables load balancing across multiple API keys and conducting canary tests by routing a small percentage of traffic to different models or configurations for evaluation.


## Summary

## Video Summary: Building & Securing Multi-Agent AI Systems with OpenAI Swarm & Portkey

**1. Executive Summary:**

This video demonstrates how to build and secure a multi-agent AI system using OpenAI Swarm for agent orchestration and Portkey as an AI Gateway for observability, security, and advanced configurations. It showcases a practical example of a trip planner application with agents communicating and delegating tasks while highlighting the benefits of using Portkey for security, logging, and A/B testing.

**2. Main Topics Covered:**

*   **OpenAI Swarm Overview:** Introduction to OpenAI Swarm as an educational and experimental framework for building multi-agent AI systems.
*   **Agent Setup and Function Calls:** Explanation of setting up agents with names, instructions (prompts), functions (tools), and transfer mechanisms for communication.
*   **Portkey Integration:** Demonstrating how to integrate Portkey as a custom client within OpenAI Swarm by modifying the base URL and headers.
*   **Portkey Features:** Exploring Portkey's capabilities, including logging, cost tracking, security guardrails, virtual keys, load balancing, and canary testing.
*   **Complex Example: Trip Planner:** Walking through a complex multi-agent system example – a trip planner application – where agents handle weather inquiries, flight booking, and hotel reservations.
*   **Observability and Security:** Showing how Portkey provides real-time insights into AI decisions, potential biases, and code generation, as well as offering security measures like key rotation.

**3. Key Takeaways:**

*   **OpenAI Swarm simplifies multi-agent system development:** Provides an easy-to-use framework for managing and orchestrating AI agents, suitable for educational purposes and research.
*   **Portkey enhances AI system security and observability:** Adds crucial features like security guardrails, real-time logging, and cost tracking to AI applications.
*   **Custom client integration allows seamless Portkey integration:** Modifying the base URL and headers in OpenAI Swarm enables easy integration with AI gateways like Portkey without significant code changes.
*   **Agent handoffs and function calls create complex workflows:** Demonstrates how to design agents with specific tools and transfer functions for effective communication and task delegation.
*   **Portkey enables A/B testing and Load Balancing:** Load Balancing across multiple keys to handle rate limiting, A/B tests or canary tests using Portkey allow for experimentation and optimization of AI models.

**4. Notable Quotes or Examples:**

*   "What this is actually doing is it's allowing you to orchestrate and do handoffs of different routines in the uh with open Ai." - Describing the core functionality of OpenAI Swarm.
*   "Port key is basically uh almost like a proxy or a middleman that allows you to have obser observability out of the box" - Explaining Portkey's role as an AI Gateway.
*   Example: The Trip Planner application, showcasing a practical multi-agent system where agents collaborate to book flights, hotels, and provide travel recommendations.
*   Describing the Guardrails functionality of Portkey, it can detect biases or whether the agent is providing code during normal Q&A interaction.

**5. Target Audience:**

*   AI/ML developers and engineers
*   Students and researchers interested in multi-agent systems
*   Individuals looking to improve the security and observability of their AI applications
*   Developers using OpenAI models and seeking better management and control over API keys and traffic.


## Full Transcript

hey everyone welcome to nerding IO I'm JD and today what we're going to go through is open AI swarm which is an educational multi-agent system and actually connecting it to something called portkey which is an AI Gateway and what that allows in the terms of observability and security as well as some cool configurations that you can do for things like Canary tests and with that let's go ahead and get started all right everyone so we're going to look at open AI swarm they released this about a week ago now maybe even longer and what it is is it's an educational and experimental framework that you can use to kind of start looking at uh multi-agents with just openingi and so there's there's some particularly cool things uh that I want to go through and then we're going to kind of look at how you can actually add observability to this and look at some security features in Port so what this is actually doing is it's allowing you to orchestrate and do handoffs of different routine routines in the uh with open Ai and so you can kind of see right in there their example right here of uh a transfer so being able to hand off from One agent to the other and what we're going to look at is we're going to dive into this example and just kind of understand uh a little bit what's going on look at some code and then uh actually go through a complex example so when we're setting up an agent in swarm basically you just have a name and this is so that you can label and kind of pass back and forth to these agents you have instructions this is essentially a prompt and then you have these functions functions are the same thing as as they would be uh before you know kind of like tools but what you're doing here is you're transferring between agents and then you have a run uh and you basically have your message history we're going to go through some other complex things that they have but basically this is uh a very simple multi-agent one thing to pay attention to though is this swarm client and so what we're going to do is we're actually going to look at the core and understand this a little bit better because it's really important of how we want to set up our observability so if we come over here and we look at the uh the core file of this swarm you can see that if you don't set anything it automatically defaults to a uh a regular open AI uh call like an SDK call so importing the the actual library but you can actually pass it your own client why is that important that's important because what this means is we can actually set up our base URL we could actually set up different um models if we wanted to as long as they're compliant with the open AI specification so what we're going to do is we're going to use that to it our advantage and we're going to actually use something called Port key and we're going to take Port key and we're going to connect it so let's go ahead and kind of look at uh how this would actually operate so if we go over to our example we're actually not really changing much right so you can see here we're actually finding our client and our open AI key is going to go into our parameter we're going to change our base URL to the port key Gateway and we're actually going to go through and passing our headers and so what this is doing is it's instead of actually going out to a model it's going to allow us to uh actually hit the AI Gateway that Port key provides so port key is basically uh almost like a proxy or a middleman that allows you to have obser observability out of the box it also has uh different security aspects you can do different configurations on all different kinds of things so you could even pass in different providers we're going to take a look at what all of this means but the most important thing is to know that you can Define what this this model or this client is and then you just go ahead and pass it in in the same as you would for your swarm you're just including client with it and this example is just going to do the exact same example of before it's just going to actually take an agent and uh do a do a handoff a simple transfer right so basically all we're saying when we say transfer to Agent B is we're actually just P passing in that agent so let's go ahead and run this so if we do python GRE all right cool so we've already got our uh our information that's coming back to us the same as it was in their example it's no different we're just passing it again through a Gateway here so let's go take a look at what that Gateway Act is actually doing real quick everyone if you haven't already please remember to like And subscribe it helps more then you know and with that let's get back to it what port key is so all right so let's take a look and see what port key is so Port key is just a uh AI Gateway but it gives you a control panel for AI apps so one of the cool things that I like about it is it's open source so you can self Host this if you want um but we're just going to go ahead and look at their their freemium model and kind of see you know the analytics of what this is and go go through a couple different features so basically now that we've actually run a uh a request through here what we're going to do is we're going to look at our logs and you can see that we've had two openai requests come through and so when we're looking at this all right so what's happening here is we're actually seeing our total cost of tokens we're actually seeing our chain or our runnable sequence if you will right very similar to like things that are happening in link Smith or link fuse we're actually able to see what's going on with our system so we'll see the the request from the user we're uh then seeing or the the system is saying this part and then we're saying our request from the user is going to transition to our H&B and then our assistant is actually doing the function call to then say go to assistant B and then I will fetch to assistant B and then can you please hold for a moment and what's happening after that is is actually going out and getting that request so now that it's actually gone through we can actually see we wanted to transfer we've transferred we now have our functions so basically between these two requests we can actually see what's going on so now what we're going to do is we're actually going to go back and we're going to see more complicated example so if we go into another file we're just going to see trip planner we're going to kind of see what's going on uh in this more complex example so again we're not changing anything other than the basic way of how we're interacting with the openai client we are then passing that client to swarm right so we have our open a I key we have our Gateway URL which is just through our base URL and then we have some default headers this is again used for Port key now we're actually going to define a couple of different things and we're going to see that we have functions for get weather we have functions for booking a flight or booking a hotel we have instructions so we want to know who this uh that this is a traveler we're actually going to introduce the idea the concept of name uh traveler being the default so basically we're going to pass in context and then we are going to look at our agents so what we did in this one is rather than actually having the functions hardcoded to these agents we've what we've done is we made sure that they all have tools right so the triage agent basically has and this is almost like our supervisor or the main agent that's going to be handling things all this agent knows is about the weather right and then our flight agent is going to be acting as our uh booking agent it only has access to this particular function and we also have our hotel agent and this one is going to be just booking our hotel and then we're going to have all these transfer functions and all this is doing is transfer transferring in between so in this example what we're going to do at the very end is we're going to take all of those agents and we're actually going to extend or append the uh the functions that are associated with this so why would we do that well let's say that these agents are actually in their own class and they want to be used in different parts of the application or by themselves so these agents can operate as individual agents on their own rather than having the transfers hardcoded to them and then we can have a different instance where we compile all these agents together and we're strictly focusing on the ability to communicate between these agents so you could think of these as these agents could be isolated in a different class and be called in different parts of your applications as a single agent and then in this particular planner or overall multi-agent we're defining our communication here this is just a helper class so all it is really doing is letting us know what's actually helping or H happening with our tools when they're being called what's actually going on behind the scenes can actually abstract that or obscure it from uh from the user this is our function where we're actually going to look at a run trip planner and so what we're going to do is we're actually going to take context and we're going to pipe in user information so you can almost think of this as like a an authentication session or some other information coming from the database or maybe even in getting this information from another chat as you're talking with the user right now we're just going to say this is just a static context we're going to pass all of that in we're stting that our history is just going to be blank we start with our triage agent and then this is actually just keeping the history alive so basically we're just going to use our command line and uh interact with this agent the first thing we're going to do is we're going to pend the user input and which is is just this and then we are going to uh start our run we will keep our uh our system alive and then as we're entering things in it'll just keep extending our messages array before it responds with our agent so let's go ahead and give this a a shot to Python and we do a trip planner oops I want to book a trip a 5day trip to Japan based on the weather so it knows who we are it knows that we're going to go to and it's looking for the current weather it gets the weather and then it is uh telling us it's a lovely time to visit so uh first maybe let's say based on the weather based on get me uh plan I just want to see if I'll actually put something together uh so it's it's saying okanawa is a good popular D destination and so let's uh go ahead and say and so now it's transferring us to the flight agent and it is uh booking our flight and it is saying our flight is booked for October 27th so this is actually two days from now uh oddly enough it's actually pretty close but we don't actually have a tool to let the system know what date it is right so this is probably thinking that it's October 23rd 2023 but if we actually wanted to what we could do is we could tell the system either based on context variables or another tool that just says uh it has access to the date and time of where the user is right so they could have their their local for instance or local great so now we have a flight book now my hotel okay so now we're transferring back to the triage agent the triage agent knows to transfer to the hotel agent the hotel agent is actually going to call the book hotel function and it's saying okay you are now booked for the October 27th to November 1st so this is really cool in the sense that it is actually taking it knows it's going to be a 5day trip because it already has that context and it knows that it is going to uh be a 5-day stay so we're going to stay in okola for uh from the uh 27th through the 1st so I think it's funny it's putting in emojis as well so we could actually take this a step further right like if we wanted to in the booking hotel or or maybe even in the triage or having a uh some other agent we could actually have a tool back up here where instead of just saying that this was booked as a hotel we could actually incorporate the stripe API we could take that information and put together a checkout link and then from the let's say the the web or even from a text message or something could actually Prov provide a link that allows the user to click and actually goes through the process of purchasing that ticket all of this is done because we have access to tools and we have the ability to kind of jump between agents or do an agent handoff based on our transfers so now what we're going to do is we're going to go back to Port key and we'll actually look at the observability of that system so now uh I just have to refresh here all right so you can see that we have a lot more requests that are coming through and every single one of these runs is it actually going through the at basically adding a message and then returning the entire history so what we're going to do is we're actually just going to look at the last one and this will give us our full context of the uh what is actually happening so so right here we can see what the the immediate part was is it's actually looking and saying your hotel specialist Health hotel booking specialist it knows I'm saying I want to go through my 5-day trip it's actually pulling out the calls that are associated with this it's getting the tool so every single request we have we now have an action here so we can see the handoffs actually H uh happening right right here we're going back to the triage agent we're going to the hotel agent we're actually seeing the information of the arguments that are being passed to book this hotel we know that we have a check-in date see right here where where we knew or I knew like it's uh 2023 that's just uh October 2023 is its knowledge based cutoff date and so it's just thinking that's uh that's where it is uh ironically you know the same day but a year year later so we know that these parameters are actually getting sent uh which is why we can actually take these things and then create a link to stripe or Expedia or some other system that could actually book this trip now when we see the tools here it's actually going through the function itself it knows what's required right it dynamically built out this function which is really cool and then actually did the transfer back to the system again it's showing us our price all the way along and then the other thing is it's doing feedback right so this is where you could come in and annotate how good was this response How well was it happened another thing we'll be going through is uh at a later date like guard rails and uh well we can just take a look at what these look like so if you come in and guard see guard rails basically you can add different guard rails to your responses so you could do things uh you know where is it like a Phi detection um you can do other things where for different biases you can check and see if there was code that was in there um like right here this is really good we've seen examples of llms that are spitting out uh code when all you're doing is asking it questions so this is a really good way of actually being able to take your systems and and put in guard rails or moderations right they have other things like virtual keys so virtual keys are great because let's say that you have an uh open AI key or I have this anthropic key as well if I don't want to expose this key to uh my system because I want to be able to rotate this key I can actually give it a give the uh the port key here I can actually change this to allow it for grabbing the virtual key as opposed to hard coding even though this is through an environment environment environment variable imagine there's a system and you have these Keys going uh in multiple different locations and you have to rotate it uh it's easier to rotate it through a control panel than it would be to rotate it in your application for instance so this allows you and your team to kind of rotate keys if you need to and you just have the uh the virtual key itself that is available the other really cool thing in uh these is the ability to load balance so if we take a look at uh we'll take a look at our AB test these both kind of work the same so right here I'm actually saying what I want to do is I want to load balance these so let's say I have the same type of I have multiple different keys so we'll just say we'll just go - one just to show an example dash2 or D3 we'd actually have three three different Keys all all we know all of them are different virtual keys but we and they're all for open AI we want to say our weight so in this instance it's saying we're going to distribute these Keys evenly so uh we'll leave this one at 35 and the rest we'll do as 30 and that means that 35% or on average 30% of the time you'll start rotating through these keys and you can think of the this is like a great way to uh deal with some caching or rate limiting issues um for instance if you you know open AI only lets you do about eight images per minute so if you rotate that through you can basically say okay well you know 30% of the the time I'm going to use a single key uh and then just rotate through these Keys the other thing that this does though is it also allows you to do Canary tests so we're just going to change this back so we can get rid of these errors at the bottom and what that means is so here I actually want to do a canary test or an AB test and I want to say that 95% of the time I want to actually use this virtual key and this prompt um and then five% of the time I want to start testing my CLA key but I also want to override some of the parameters with uh a llama model and so what's in in this instance like or the virtual key right so in this instance we're overriding but all this is doing is saying 5% of the time I want you to actually run a different test and that way I can actually observe what's happening all right that's it for us today everyone so what we went through again is kind of looking at what a multi-agent architecture looks like with open AI swarm and then actually connected it to an AI Gateway using Port key so that we could look at different security aspects observability and even doing some Canary tests with that happy nerding

---

*Generated for LLM consumption from nerding.io video library*