̽»¨ÊÓƵ

(0:01 - 0:38) Tableau, Cohere, and ̽»¨ÊÓƵ would like to welcome you to our webinar, Tableau and Cohere, Secure AI for Federal Data and Supply Chain Risk Management, SCRM. To tell you a little bit about ̽»¨ÊÓƵ, we are the trusted government IT solutions provider supporting federal, state, local government, education, healthcare, and private sector industries. We connect our customers across the U.S. and Canada with thousands of technology vendors, resellers, system integrators, and managed service providers to deliver a comprehensive range of solutions and services to meet organizations' needs. (0:39 - 1:39) ̽»¨ÊÓƵ, in conjunction with our vendor partners, is pleased to host and sponsor thousands of educational and informational events each year, ranging from webinars like today's to accredited trainings, trade shows, roundtable discussions, technology forums, users groups, and so much more. ̽»¨ÊÓƵ holds a wide variety of schedules, contracts, and purchasing agreements that streamline and simplify the IT procurement process. As the public sector distributor and master government aggregator for our vendors, reseller, and integrator partners, ̽»¨ÊÓƵ has assembled a comprehensive IT solutions portfolio that includes products and services across multiple technology verticals to support organizations with mission-focused solutions for cybersecurity, FinOps, multi-cloud, DevSecOps, artificial intelligence, human capital, legal and courtroom technology, customer experience, and engagement, and so much more. (1:40 - 2:08) To introduce Cohere a little bit more, Cohere provides secure enterprise AI for organizations that need to protect sensitive data while still moving quickly and achieving mission goals. Their large language models help users search, summarize, and generate insights from both structured and unstructured data, all using natural language. Security and governance are also built into the platform with private deployment options, enterprise-grade security, and support for zero-trust architectures. (2:09 - 2:36) Cohere helps agencies adopt AI while meeting strict compliance and data sovereignty requirements. When we talk about Tableau, we're talking about the visual analytics that helps organizations turn complex data into clear, actionable insights. Tableau connects data from multiple sources and gives users an intuitive way to explore information, identify patterns, and make faster, more informed decisions. (2:36 - 3:03) It also brings AI-powered capabilities into the analytics experience, helping users explore their data and uncover insights more efficiently. And with strong security and governance, Tableau provides a trusted foundation for analyzing sensitive enterprise data. At this time, I'm pleased to introduce our speakers for today, Josh Gilmore, Distinguished Solution Engineer for Tableau, and Mitchell Wong, Head of Solutions Architecture for Cohere. (3:04 - 3:09) The floor is yours. Excellent. Thanks, Elaine. (3:09 - 3:43) Josh, I think you were gonna pull up the slide deck. I'll give a quick intro for myself before I pass off to Josh to lead part of the presentation, but really excited to have everyone here. My name is Mitchell, so I know on the slide it said I lead our work in Canada, but I actually lead our global public sector work now, which includes defense, IC, FedSiv, SLED work, and seeing how this technology is really delivering value to our customers is really rewarding. (3:44 - 4:14) I think I have one of the most exciting jobs in the world, and I can say that pretty confidently, but I also know areas where challenges come up in implementing these solutions from a governance, security, just education perspective. So I'm really excited to talk through some of these concepts during this webinar today and show you how this is actually getting applied in supply chain logistics and risk management. So excited for this conversation. (4:15 - 4:24) Let me pass it off to my presenter, Josh, first to share a bit more on his side before I chat more about Cohere. Great. Thanks a lot, Mitchell. (4:24 - 4:35) And just to confirm, are you able to see my screen? Get a thumbs up, Mitchell? I can. Okay, perfect. Thanks so much. (4:36 - 4:52) Hi, everyone. So I'm Josh Gilmore, Distinguished Solutions Engineer over at Tableau Global Public Sector. I primarily spend a lot of my time working with defense and national securities, specifically throughout the U.S. That said, I do cover off our government accounts globally right on down to state and local. (4:53 - 5:03) I've also been spending a lot of time working with LLM integration, especially over the last three and a half, four years. A lot of high side on-premise environments as well, too. So you guys are going to get a chance to see some of that here today. (5:04 - 5:26) I've also, for probably about the same amount of time, been talking to a lot of our customers about supply chain logistics challenges that they face. So it seemed like a natural combination to bring the two together. And so to give you an overview of what it is that we're going to be talking about here today, first things first, we're going to give you an overview of some of the challenges that are being faced, also some of the solutions. (5:26 - 5:38) We're also going to give you some key takeaways as well. So this is primarily going to be slide focused at the outset. We want you guys to be able to walk away with some pieces for analysis, for using AI to be able to think about. (5:38 - 5:46) And then after that, we're going to hop into a live demo. I have about four vignettes. Think of it kind of like an a la carte menu for what it is that you'll be able to tackle. (5:47 - 6:19) And then the third part is Mitchell is going to pop back in and he is going to show you some really cool stuff with Cohere, some pieces with documents, and then also how you can automate some of your workflows directly off those docs. Then we're going to finish things off with a question and discussion towards the end. So I've been talking to a lot of our customers for quite some time, covering anything from, bus transit, fleet management, right on down to maintenance repair operations, working on some of the biggest weapons systems or vehicles globally with some of our biggest customers.

(6:20 - 6:27) And there's three key areas that tend to come up time and time again. First part is data. A lot of different data sources that are out there. (6:27 - 6:43) You may have supply chain risk management data sources from Exeger, Govini, and Teros, and then a whole series of different sources internally. These could be Excel spreadsheets, for example. That's, I'll be quite honest with you, one of the major pieces that runs the world as far as logistics is concerned. (6:44 - 7:07) Second part is how do you actually start to bring all this together under one vignette? So part of that is some of the items happen to be especially that scrim data is out on the cloud, but a lot of decision making could be taking place on prem. And how do you connect that together with these different sources that you have? And the final part is there's quite a bit of data. There's massive data deluge out there. (7:08 - 7:32) In fact, some of the data we'll be working with here today has 417,000 organizations in it, okay? And that is from a single vehicle, just to hit home on that. So the natural question, and this has been a lot more recent, is how can we take AI and begin to apply that to make sense out of it? So what I'll be walking through here today is these three pieces. I'll have an additional fourth piece. (7:32 - 7:50) If you guys saw that whole graph rag piece in today's presentation description, we'll be going over that as well. But this came up probably about three years ago. I was talking to a customer and they're like, hey, we have to get a better idea of exactly what's currently in inventory, what's about to might not be working soon enough. (7:51 - 8:04) But not only that, I have to understand what my supply chain looks like. So I was like, hey, tell you what, give me about two weeks. I'm going to go, I'm going to connect this data on in to your parts data, and we're going to help to make sense out of that. (8:04 - 8:26) And it just so happened that the decision-making resided on high-side networks, so offline networks. So how do we bring that together, make sense out of that? Then the final part was, well, now with AI, we can go and ask questions directly on top of this and be able to tap into that data. So the data that we're working with, 417,000 organizations, that stems from 82 primes. (8:27 - 8:39) So the first and initial delivery companies for those given within the supply chain. Second part to that is 15,000. So those suppliers supply those initial 82. (8:40 - 8:50) And then the three degrees of Kevin Bacon, if you think about it like that, that's 400,000. So we have 417,000 organizations that are part of this. You guys will get a chance to see that here in action. (8:52 - 9:06) As I've been going through the LLM integration piece, these are a few of the highlights that I'd like to bring on up. First part is my very first customer to get their LLMs up and running, they're up and running in jwix. That is a hard to reach offline environment. (9:07 - 9:33) It is very high security, and they're using Tableau in conjunction with their LLMs. Second part of this, program analysis, going through tens of thousands of different programs. How much money is being doled out? What sort of risks do we face? What sort of projects should we really focus on? Those are some of the questions that folks were looking to answer, in addition to combining their text with their data simultaneously to be able to answer those questions. (9:34 - 9:42) And sometimes that can be a little bit challenging for folks. But with the combination you'll see here today and using Cohere to power that, we'll see exactly that. Started the show, what we're really all here for. (9:44 - 9:54) A number of different vehicles that are out there. I've been working with some primes to get set up, provide some advice on their LLM integration side of things. And we've seen a lot of success on that front. (9:54 - 10:21) In fact, one of my customers said, okay, dashboards are great, this is thousands of times better. And this isn't a widget counter. To hit home on this, when they did their testing with 120 billion parameter model, which is very similar to the size of Cohere's smaller models, and Cohere goes even bigger as well, they ended up with 100% return in terms of their testing for higher level reasoning. (10:21 - 10:41) So what that means is, they're asking questions about, well, I have this inventory. How does that impact my manufacturing? What are some of the things that we have to look at further on down the road? So again, this is really getting at some deeper questions. In fact, what we're currently looking at right here, this is a publicly posted F35 demo from an event. (10:42 - 10:51) And this is the Tableau dashboard. So what they're using for the Alice system in the F35 is Tableau. Why? Because it works within high-side environments. (10:52 - 10:58) It works with a massive amount of data. And here you can see parts. There's mission-capable readiness of various vehicles that are in there. (10:58 - 11:10) Obviously, they're not showing real data, because I'm sure there'd be a lot of questions in mind then. But they also the ability to keep track of what's going on. So imagine if you were to combine that with AI at the end of the day. (11:10 - 11:17) And a lot of folks know Tableau for making pretty pictures and dashboards. Yeah, I'm sure we do a great job of that. But a few key elements. (11:17 - 11:29) One is, we can bring in as many data points as you need to render. Most of our competitors are unable to do that. So what that means is, when you need to map out 417,000 organizations across map, you can do that. (11:29 - 11:39) Also, no limits on the amount of data you want to bring in. In fact, on that front, on the left-hand side of things, we have the most data sources out of the box. A lot of folks will nickel and dime you for it. (11:39 - 11:50) Not us. We just want to make sure that you can bring your data together, create the data fabric that you need to, and then connect this in a cohere. That's one of the critical pieces, and we'll talk more about that. (11:50 - 12:15) Second part is that we have an ETL tool. So for those who are unfamiliar with that term, cleansing and standardization, which means you can get someone who's a logistician, someone who doesn't have a deep technical background to go in, take the data that they need to, get the provenance. Where is this data coming from? And the real cool part to this now, especially when combined with AI, you get Python, R, you want to connect the end of Palantir, Databricks as well, too, for some advanced components. (12:15 - 12:33) You can totally do that. In fact, you can have your LLM cohere build out the entire ETL framework, take care of the data science side for you using skills that we have here at Tableau for your LLM. The other part is that maybe you want to take some of these advanced pieces, weave them on into the dashboard itself. (12:34 - 12:49) The people who are doing the analysis, they don't know that the predictive modeling is necessarily running in the background. They just get the answers that they need to, maybe some what-if analysis, whatever that might look like. Then the final part is, you have the ability to deploy into your environment in a shared and governed way. (12:49 - 13:00) What does this mean? When you connect on into your LLM, that is a governed data source. Your permissions pass on over to that LLM. And the other critical piece is that we have a metadata layer. (13:00 - 13:12) Think of it like a mini Informatica or Calibra. It's what data nerds and stewards like myself really focus in on. One of the key elements there is that LLMs use those data descriptions to find relevant data. (13:13 - 13:18) I have this term here, MCP, Model Context Protocol. It doesn't mean anything to you at this point. Don't worry. (13:19 - 13:26) Give me a few slides. It will. Then the other part here is that we can connect on into your LLMs directly in a centralized and governed way. (13:26 - 13:44) So why would you end up doing this? One of the key parts is that all of the different data sources that we bring to bear, you can connect in directly to Cohere. So what that means is legacy data sources that will never see the light of day. You can bring those on in flat files like GeoJSON, for example, can then be piped directly on into Cohere. (13:44 - 13:59) Second part is, well, I don't know how to work with Python. Maybe you just need something to go through and become that data scientist on demand. That's what Cohere can do by plugging generating that automatically and not only that, leveraging the APIs as well, too. (13:59 - 14:12) So some poor soul isn't stuck having to create data descriptions for all that metadata that's going to be really useful down the road. Other part is permission side of things. You have your permissions that are being applied. (14:12 - 14:20) And then what ends up happening is a lot of different elements are passed on over to Cohere. So Cohere is only seeing what the user should see. We have real-level security as well, too. (14:21 - 14:32) And then, of course, that metadata layer, which we'll see more of here in just a bit. I'll explain what an MCP server is in just a moment. We're going to be focusing in on two areas of connectivity.

(14:32 - 14:47) These are the typical AI connectivity pieces that you'll have with on-prem. Obviously, works as well, too, within a cloud environment. We're going to focus in on the direct connection so that we can explore data, take care of the analysis dynamically on the flat using Cohere. (14:47 - 15:00) Second part of this is model context protocol. Think about like a universal adapter. When I go traveling to the UK, let's say, and I need to plug in my device, I bring a universal adapter so I can plug on into the wall. (15:01 - 15:12) That MCP server is like your universal adapter for your large language model, Cohere, in this case. On the other side is a solution. Tableau, we've created our own MCP service. (15:12 - 15:27) And what that means is that it is optimized for use with Tableau, works in your high-side environments if needed. But what it allows you to do, and you'll get a chance to see this here in a bit, is you can type in a question that you're looking for. It'll automatically find that data. (15:27 - 15:42) It'll even bring together multiple different data sources to help to answer the questions that you have. And if you get really sort of advanced with it, too, you can even have automated briefs coming in in the morning or as soon as your Tableau data is being updated. And I'll show you some of that as well. (15:43 - 15:53) Tableau agent and server, this is functionality that works directly in Tableau. It's kind of GenBI if I use that term. So you can type in what it is that you're looking for. (15:54 - 16:07) The visualization is automatically created. Forward-looking statement, what that means is it's not in the product yet, so please don't buy Tableau or Salesforce based on a forward-looking statement. Don't invest in companies based on a forward-looking statement. (16:07 - 16:18) There's no guarantee it'll be in there. Sorry, I have to say that every single time I say that because my legal overlords have to be appeased. But this fall, what we're looking at doing is releasing our LM gateway. (16:19 - 16:29) So what that means is you'll be able to plug Cohere into Tableau directly. It'll also give you some background or some summaries in terms of the visualizations or dashboards you're looking at. And finally, is lane chain. (16:30 - 16:37) This is the backbone of a lot of agentic work that is taking place. So let's say I find some data up over here. Hey, I've created this report. (16:38 - 16:53) I need to take action with that. Lane chain is something you can integrate directly into your dashboards in the background. What that means is Cohere can then take some of these findings, write this on into, I don't know, a case management framework, take additional action with the data that you have. (16:53 - 17:05) One that isn't listed here as well is that we have various skills that are available to tap on into. And so on the Cohere side, let's dive into that just for a moment. And Mitchell is going to go into this in greater depth. (17:06 - 17:15) But Cohere is a fantastic model. It can deploy on-prem. You can have sovereign AI exactly where you need to be making your decisions. (17:15 - 17:34) So if it happens to be a secure environment or otherwise, even directly through cloud via API, all the options are available. They have fantastic models. Anything from embedding models, for example, if you're working with your documents right on down to Command A+, which is a larger reasoning model that's been released, their Command A model line as well. (17:34 - 17:40) And Mitchell is going to go into that a little bit more detail. They're private, secure, just like I was saying before. So it's air-gapped. (17:40 - 17:57) You can hook up directly to this in your environment. You also need to do some fine tuning. One of my customers working with supply chain risk management data, that's exactly what they ended up doing, using it on some telemetry to be able to curate the model for the type of data that they were working with. (17:57 - 18:08) And the final part that I'm hearing most often all the time is token anxiety. I don't want to spend too much money on tokens. You don't have to worry about that here because Cohere can deploy directly into your environment. (18:09 - 18:23) And guess what? As long as you have the computational resources, you can tap into your Cohere model. And so it becomes extremely useful because what we can do is connect Tableau directly into this. We can select our data, pass that on over to Cohere. (18:23 - 18:32) Within Tableau as well, too, you can select the different models. Maybe you want command A reasoning, for example. Maybe you want a different model to be able to work with. (18:32 - 18:38) That's been fine-tuned. You can pull those on up. And finally, the fine-tuning component that I brought up beforehand. (18:39 - 19:10) And then when you bring Tableau to bear on this, this is a really critical piece that I think everyone needs to know when they're conducting analysis with an LLM. It's something that impacts every single LLM that's out there. It is just the quality of working with a lot of data and LLMs. And I don't think enough vendors really talk about it at the end of the day. One thing to keep in mind is that an LLM can get overwhelmed by too much data. It's sort of like throwing a whole bunch of books in front of someone saying, study for this exam tomorrow. (19:11 - 19:18) They're going to show up, but they simply won't know what they won't know. So certain elements will be lost. That attention, that focus will dissipate. (19:19 - 19:40) So it's really important that you right-size your data when you're conducting analysis. And what we can do in Tableau is limit that amount of data. And Tableau becomes a moving window that any user can leverage to be able to tap into the different data points that they need to and still help to optimize and get the best possible response out of that LLM. (19:40 - 19:49) Cohere, in this case, at the end of the day. Otherwise, it results in context rot, is the name for it. And so that's what I was talking about before when a model has a difficult time doing that. (19:50 - 19:55) Here's a little bit of levity to the presentation. So I was talking to one of my friends over at another big tech company. He's like, hey, check this out. (19:55 - 20:17) This probably encapsulates context rot really well. If you put in too much data, because the context window itself will allow for a lot of data, just because you can doesn't mean you should, the model will go through preference, have higher attention and focus to the front-loaded data that happens to be in there, a little bit on the back end as well. But then it loses sight over that data. (20:17 - 20:30) So a lot more granular references can, the model can have a difficult time really sort of compensating being able to work with. So that's something that we can do here in Tableau to really make sense out of it. Don't worry, final slide, I promise. (20:31 - 20:38) And what we're going to do here today is we're tapping into data with Tableau. We're then creating that dashboard. Dashboard's already created. (20:39 - 20:49) And so what we're going to do is interact with that dashboard, select the data that we want. And then what happens is you pop in a question. It can be an open-ended question. (20:49 - 21:06) You can also have a series of drop-down questions because sometimes making this leap can be a little bit more difficult for folks who are within an organization not used to conducting data analysis. That can really help to get consistent results out the other end. And then that gets passed over to Cohere. (21:06 - 21:34) So the combination of that data, the question, which are really kind of the instructions, and then Cohere goes through, provides an answer, and really helps with that efficiency of effort. So this is about reducing that cognitive load, helping to overcome maybe areas where people may not be as familiar with the data to really get out what it is that they need to, and then helping to get at that why. So what that means is they can focus their own tacit knowledge in on the places that they need to explore. (21:34 - 21:47) So it drastically reduces time on task. All right, here's the fun demo time. And one thing to point out, even though there's an F35 that shows up throughout all this, I want to emphasize this is just a widget.

(21:48 - 22:00) It could be a bus, it could be a backpack, it could be an inventory list. You'll get an idea of that here in just a moment. So I want to start off with our supply chain risk management data. (22:01 - 22:05) 417,000 organizations. I've already pre-selected some of this right now. It's some baking show magic. (22:06 - 22:11) Don't worry, I will change it here in just a moment. I have these organizations globally. I have five of my primes. (22:11 - 22:24) Right now, I have metal and mining selected. This is really about getting things like rare earth minerals, for example. But what we've done here is that we have Cohere, and I'm currently using Command A plus to get out the answers that we need to. (22:25 - 22:48) And it is going over 534 entities that we have within this area globally. So I've asked it, Dan here with my prompt, provide me with an executive brief on supply chain, highlighting potential issues, clearly explaining your rationale. And what it's doing is the critical ground combat vehicle program depends on complex network and metal and mining suppliers that provide structural alloys, armor plates. (22:48 - 23:01) So it's giving me context straight out of the gates. I can fine-tune this as I see fit. It's going through, providing a review of different scores that are being delivered, including geographic information, critical material suppliers. (23:01 - 23:08) So you guys are getting an idea. And again, this can be fine-tuned to whatever it is you want. I can ask for different elements directly within here. (23:08 - 23:23) So in fact, why don't we do this? Let's switch this on up. And here, you can see, click the Analyze in GNI button to load data for large language models. One of the things to keep in mind is that you want to force a model to use that data because sometimes models can create hallucinations. (23:24 - 23:38) And this lowers the propensity for hallucinations by forcing the model to work with the data that we have. The other part is, let's focus in on China. There are compliance issues, especially when it comes down to working with any sort of defense materials. (23:38 - 23:49) And here, you can see I have 749 organizations in my second tier. Why don't we go on in? Let's go take a look at electronics and equipment right on over here. And I'm going to say Analyze with AI. (23:49 - 24:10) And so what happens is the data, the instructions over here, I have additional instructions in the background I haven't brought up. I'm just kind of keeping things nice and simple here, too, to then provide us with the analysis. So I want to hit home that if someone's going in and asking questions about specific industries that are being impacted, they want to bring certain elements to the forefront, that would take a long time normally. (24:11 - 24:26) Maybe assigned to a subordinate, they'd have to go and be like, oh, hey, I'll be here about a day or two later, provide you with a report. What it's doing is it's going through, it's focusing on the areas that we need it to for the audience that we want. I'll unpack that more here in just a bit. (24:27 - 24:50) So we're taking this supply chain data that can be very difficult to work with, especially at the scales that we're talking about, and delving on into what it is that we need to. Not only that, I have a self-reflection component built in here. I think it's always important to say like, hey, is this covering off all the answers that we need to? Yeah, it's grounding data, it covers supply identity, location, industry, cyber scores, but lacks some supply chain context component. (24:50 - 25:00) So it's calling out like, you know what, I could actually use this additional data. In fact, within Tableau, we're able to point at that data. And this is the tip of the iceberg because a more complex dashboard. (25:01 - 25:22) I've let the user select what data they want to dynamically ground with on the fly. So that's the first portion. Second part is, well, what happens when we start to connect this together with the 417,000 organizations with all of our component data? So this is helping to get at the so what. (25:22 - 25:47) So when someone says, well, you know what, I actually have a backlog over here. Well, what does my supply chain look like behind the scenes? So catastrophic risk score, that's discrete in space and time, could be a hurricane, for example, those kind of pieces. And typically, what would happen in this demo before AI is like, hey, I found this over here, I can go check and put in a purchase order, go and explore more behind the lift van doors, or seeing some of the challenges with different vendors who are out there. (25:47 - 26:09) But the key part is, is that now I can go in and ask AI cohere to go through and assess this for me. Again, going back to, well, what does this mean? What does it mean in the context of the different elements that I have here? In fact, we have a maintenance repair operations piece for this. So let's say you had a whole series of tickets that you needed to be able to work through. (26:09 - 26:27) What do you prioritize? What's on the back burner? What's actually being impacted as well, too, by your supply chain itself? So here it is, Bluff, the F35 supply base exhibits bimodal risk profile. Most suppliers demonstrate strong cyber posture. And by the way, this is notional data as well, too. (26:27 - 26:40) So where you do see some of the actual names popping up, not that. I've asked it to give me direct IDs so I can reconcile that if I need to with my data sources. But again, this is giving you an idea of what the art of the possible might look like. (26:40 - 26:51) This could be anything, could be a bus. In fact, because they didn't give us actual F35 parts data, that's what I'm using on the back end here. I'm using actual bus data to be able to link on in. (26:51 - 27:10) I know it takes away from some of the magic of the F35, but I'd probably have the authorities knocking at my door if I had that data to begin with. Next part, what if we had a network of different suppliers who are out there? So think about that. I could have an entire network. (27:10 - 27:31) Or what if I had a whole series of product orders and I use Cohere to extract my entities, which I've done, and then I create something called a knowledge graph. So when you guys may have read about Graph RAG, for example, it's something that's becoming a little bit bigger in the industry. Maybe you guys have heard of companies like Pinecone moving towards Graph RAG. (27:32 - 28:00) There are other different organizations who are currently leveraging networks. Here's the key takeaway with this, is that the relationship between my entities actually gets passed on over to Cohere. And what Cohere is able to do is infer based on those findings, not just the documents themselves, but it takes into consideration those relationships in between those and starts to give me a better idea of what that means in context. (28:00 - 28:10) So it can be far more robust in terms of what it's able to deliver on this front. I've used it in different contexts, countering influence operations, areas like that. Extremely useful. (28:11 - 28:36) And the other portion is that if I was to connect this on up, it's not currently connected up with my entire document corpus at this point in time, I could connect this in with Cohere embedding models. So it could go in and find relevant pieces of data, bring that back, and help to create an extremely robust answer across millions of documents, for example. So it's really powerful what Cohere can do, and we'll get into that here in just a moment. (28:37 - 28:48) But again, you can see the dynamic analysis unfolding right before us. All right. So last piece, and this is where the MCP portion comes into play. (28:49 - 29:07) So at Salesforce, you guys probably may or may not know, Slack is part of the family, and here we are in Slack. But also think about this potentially being something like Teams, tons of interoperability. What we're currently looking at right now is a Cohere MCP agent. (29:08 - 29:28) I've created this agent. It's something that I use for like aisle five environments, areas like that, that sometimes some of the cloud-based AI capabilities can't reach. And not only is it an agent that's simple to deploy, it's all Cohere powered, but what it's able to do is first, I can have interactions directly with it. (29:28 - 29:39) I put together some baking show magic just so you guys wouldn't have to wait or see things get typed on out, not the fastest type for some days. And so I'm like, hey, it's like, hey, I'm here to help. Here's what I can do. (29:40 - 29:45) Automatically give me an overview. Great. I'm able to pull my supply chain risk management data. (29:45 - 29:55) So the 417,000 organizations, that data set, in addition to about 250 others are on my Tableau server. So it's like, yeah, hey, I found it. Here it is. (29:55 - 30:01) It's got cage codes. It's got everything associated with it. Can you go through and give me a DQA on this front? Sure can. (30:02 - 30:08) Here's the summary. This is what we're looking at. Some of the stuff is kind of truncated to make it work a little bit better so we can get into it. (30:09 - 30:24) But the real meat of this, and this is cool because it's also telling me like, hey, here's some of the scoring elements that are built in, some of the considerations and the conclusion. Well, I'm an executive working on supply chain vulnerabilities for my defense business. Can you provide me with an overview? Sure enough, here it is. (30:25 - 30:30) Yeah. North American providers going on through 417,000 organizations. I wasn't lying. (30:31 - 30:38) It goes on and it starts to break out the report for me. This is extremely useful. In fact, this is the first time I've had a model do this. (30:38 - 30:50) It actually traversed on through the rest of my Tableau dashboards. It's like, hey, we know you're working in this. I went and I found a visualization I think is going to be complementary to the findings in the report that I've created. (30:51 - 30:56) So that's pretty cool. Goes in, you can grab your visualizations as well. It's checking out that metadata at the end of the day. (30:57 - 31:23) And so what I ended up doing too was like, hey, let's just go create a report for this. So this is me actively going and asking questions against it and getting the returns right here. The other side to all this is that if I back on up for a moment, what I was able to do is create an alert that is based on my position and my location as it's entered into Slack. (31:23 - 31:44) So in this case, I happen to be a director who's working in Phoenix. And I was like, hey, can we create an automated alert? So as soon as the data gets updated, but I only want the metrics and specifically the metrics filtered to my context. So what it's doing is it's automatically turning these around. (31:44 - 31:59) Again, all coming from code here being delivered right across from my Tableau data. Here it is, different component delivery, 30 triggering the alert, supplier's findings, and it's helping me to get out what is that I need to. I can fine tune this, get it at the so what, have visualizations appear. (32:00 - 32:31) I have different versions of this, but to give you guys an idea, this can be very powerful and help to automate the analysis in the background. There's another piece, which I just don't have time to show today, which actually has triage where CodeGear will have multiple agents. So as new items are coming in, inventory, maybe supplier information's updating, it can go through and triage some of the supplier risk, some of your, let's say, deliveries that are required, highlight those, but also do really complex pieces in terms of having a scoring framework. (32:31 - 32:38) So very powerful stuff at the end of the day. Anyways, I'll call it at that. I'm going to pass it over to Mitchell, and Mitchell's going to walk you through what Cohere can do. (32:39 - 32:42) Thanks. Over to you, Mitchell. Thanks, Josh.

(32:43 - 32:55) Let me bring up my screen share. Really appreciate the depth and the amount of examples that you walk through. I think it's really illustrative of some of the power of the technology and the flexibility. (32:56 - 33:08) So let me bring up my screen here. I'm going to go full screen. And let me actually say two or three words on some Cohere background before I dive into this slide. (33:08 - 33:38) I know there's a lot here. But as was already shared before, the main thing to emphasize is Cohere started as a LLM lab, one of the only labs in the world. And one of the main benefits is everything we do, we package up and ship to our customers to let you host on-prem privately, so you can fully control it and use it in the way that is most appropriate for your work and your mission. (33:40 - 34:04) And if we see where that came from, ultimately it is because one of our co -founders, Aiden, worked at Google Brain and was actually on the seminal Transformer paper, which is attention is all you need. So he actually authored the technology that is powering a lot of the capabilities we're seeing today. So that's the T in chat, GPT, for example, the generative pre-trained Transformer. (34:04 - 34:35) They had trained that model, saw the benefits of it, but Google was keeping it within their premises, using it for Google search, but not sharing the technology more broadly. So Aiden and two other co-founders started Cohere to really democratize the technology and provide more access to people so we can experience the benefit. And everything we've been doing since then has been to democratize and provide more control and benefit of the technology. (34:36 - 34:50) So we'll start here at the ground floor. We continue to be an LLM lab and train many high quality models. So we have our generative model here, which is Command A plus, as Josh was talking through as part of his demo. (34:50 - 35:02) It was integrated in a few pieces of the workflow there. But we also have really strong retrieval models. So this is how you make sure you don't get that context wrong. (35:02 - 35:15) You make sure you get the highest quality data to the LLM. So you answer the question accurately, consistently, and with low latency. So if you have a 100 page book, you don't send the whole book to the model to answer the question. (35:16 - 35:38) You send the five lines that are most critical and have the actual information for answering your key question. And this works also for not just text, but also images and multilingual, multimodality information that's in PDFs, documents, Excels, whatnot. So that's our retrieval stack. (35:38 - 35:52) And then we also have a transcription model and a mini coding model here. So that rounds up the LLM family. But really, we've been around for just over six years now. (35:52 - 36:18) And when working with customers, we found a lot of places where they would try to build solutions with this really powerful technology, but not get good results because of the way they basically took their text, extracted the text, or took their documents, extracted the text, chunked them. If you don't chunk it in the right way, you might actually not even bring in the right information to answer your questions against. You actually might miss it or lose it. (36:19 - 36:41) So all of these design considerations can actually lead to you getting a good result or not. And that's why not only do we build models, but we also have applications, solutions around that, which help manage the end-to-end orchestration of these models, also known as harnesses for these models. So you can think about your business problem. (36:42 - 37:04) You can think about your mission and get value instead of how you wire these together. So the two levels of abstraction, the main two applications are Compass, which is our end-to-end search stack, which incorporates a lot of our retrieval knowledge and talent. We have the best in breed retrieval models and the kind of some of the forefathers in that space. (37:04 - 37:31) And then we also have North, which takes the end-to-end search stack, adds on that generative layer and allows you to do basically document review, document Q&A. So if you're doing manufacturing and you need to figure out, I'm running into this problem, what are the steps I need to take to resolve this? It can pull up the right diagrams, images, and synthesize that information for you. So these are the applications that we build on top of the models. (37:31 - 38:43) Again, all privately deployable, all with the goal to serve the mission and to democratize and bring the power to your fingertips. The last thing I'll touch on, and this is kind of a concept that's coming up more and more, is the idea of governance around your whole AI stack. So depending on where you are in terms of your AI maturity, you may be, and we have this whole spectrum when we think internally and talk to customers, but you may be early in the journey, just testing out work with a few of these models, maybe using mostly consumer applications where it's permissible, but we know that there are other, maybe more advanced users as well, where you have some sort of tool already that has access to AI and you're using it in your day -to-day, right? But regardless of where you are on your journey, there is this idea of your users, and in this case it can be employees, developers, or even agents making calls to different models, different harnesses, which we call assistants or coding agents, and then also AI that exists. (38:44 - 39:18) So including Salesforce, Tableau, all of those services, right? So many of your folks, employees, analysts are making maybe direct calls to all of these services, but none of that gets governed. So one thing that we're building at Cohere is also that governance layer where these direct calls, which are not access controlled, are not governed by any policy, are not following any proper kind of procedures that you've implemented. That goes away, and we help resolve that. (39:18 - 40:03) There's a few benefits, obviously, that we can call out. I'll just touch on one of them and let you kind of read through the rest, but basically 13% of enterprises rate their agent governance as adequate, which means that the other 87%, if I've done the math right, don't actually know all of what's running, what's calling what, where the data is going, and that's a risk and a concern, right? You have to be able to track that. So at Cohere, I know I've mentioned the models and the applications that sit on top of those models, but I want to introduce you to this idea of this broader stack that's available, and first it's built on the infrastructure. (40:04 - 40:30) So you have maybe cloud infrastructure, or you have in some cases your own infrastructure on-prem, or even air-gapped, where you want to serve your workloads. You then have, and Cohere essentially can work on any of those due to how we deploy our models in containers. You also then have the intelligence, right? So those could be Cohere models, those could be open weight models, those could be other models you want to have access to. (40:31 - 40:48) So those are the models I already talked about, but then there's the data and context, right? Now this could be connecting to other systems you already have. A big piece of this could be Tableau, like data that's already stored in Tableau. This can also be your documents that you're bringing to bear. (40:48 - 40:59) So parsing those documents, having them live in that end-to-end compass search system. So data and context is really critical. The next layer up is the control plane. (41:00 - 41:16) So this is the area that many people are now getting more familiar with, or is becoming top of mind in terms of the conversation. But you need a way to be able to govern everything that you're doing, and that control plane is critical. So Cohere also provides that. (41:17 - 41:29) And then finally, the application. So that's North that sits on top of all of this. And essentially what we're providing is that vertical stack for customers, fully privately deployable, sitting within your data boundary. (41:30 - 41:55) And this will become more and more important as AI becomes more integrated into our daily workflows, where we trust it to do more of the work that is critical to the decisions we make today. And as we become more and more, hopefully, AI native organizations. So hopefully that is helpful as a visual of thinking about where the space is going. (41:56 - 42:15) What I want to do is talk through an example. So I'll show a small sliver of how this comes to life on North, which is that application, Cohere's platform, and how people are finding value using the technology. So here's some background for this situation. (42:15 - 43:09) But essentially, let's imagine you have a quote from a parts supplier, and you have a sub that you're trying to outfit. So we have this supplier called Poseidon Optics, they provide a quote that is a certain number. But the key thing to highlight here is that, essentially, the money is not basically in the overall quote, we're in the price table, it's actually in the terms, this is where a lot of the money moves, you have to have an analyst, or a legal person, understand the technical details, the legal aspects, to be able to read through the fine prints, and get to the understanding of actually, is what's on the price table the final number, or is there more that we're going to have to pay if we agree to this quote, right? There's often hidden details and information that is captured in this multi -page document that is sometimes easy to miss. (43:09 - 43:45) So you can imagine missing one key line in that document can cost millions of dollars, for example. So we'll show that example, and walk through, basically, this idea of like a senior analyst who normally would take three hours to review such a document, could miss information, could not know how to action certain risks that they see, and essentially, maybe they don't know what recommendation to offer up. But we'll show a workflow, end-to-end workflow, which takes that procurement document, and then makes that recommendation, does the follow-up, and automates the work. (43:46 - 43:58) So essentially, let me jump into that demo here. Since I've motivated already, I'll just walk through what that looks like at a very high level. So this is North, our application. (43:59 - 44:23) I can kind of go through a ton of background, and really just show you kind of the front end, for example. You have this chat experience here that most people are familiar with, but that's not where I want to spend a ton of time. I'm not going to cover the items that should be considered table stakes in terms of how you build an agent, selecting tools to connect to that agent. (44:24 - 44:43) I really want to dive into this example that's a lot more tailored and specific. So in this case, you can imagine this as a multi-agent workflow, and you can build this very simply by just creating new nodes within this no-code orchestration layer. So here, we have a pretty simple one. (44:44 - 44:56) It seems like it has a few steps, but it's actually pretty simple. The first agent, and I'll just open this up, has a task that it needs to perform. So it's an expert procurement document analyst. (44:56 - 45:08) It has a set of tools that it's connected to. So if I zoom in a here, you'll see that it's connected to my file. So I can upload procurement documents for it to analyze. (45:09 - 45:19) It also has access to a data interpreter. Maybe it has to do some calculations to figure out what are the risks based off certain terms that are present in the document. But it has access to tools. (45:20 - 45:44) And then finally, a model that's powering it. So in this case, we're using a model called North Large, but as I showed on the other slide, you can connect multiple models and bring the model that you think is best for the use case, because North is a flexible, interoperable harness. So again, each of these nodes are agents that have their own task, tools, and model or brain that's powering that process. (45:45 - 46:04) So at a high level, we analyze the document, pull out the key information, and then this agent will make a determination on whether or not this document requires human review. So this is essentially just a conditional. So if we require human review, then we'll send it to this node, which is a human review node. (46:04 - 46:32) It allows a person to basically look at the terms, the information extracted, and determine what decisions should be made. And if not, then essentially, it will go for confirmation or go for acceptance, it gets converted to this different format and uploaded into a different system. The more interesting path is maybe if something requires human review, the person will decide whether or not it is approved or not. (46:33 - 46:51) So if it is not approved, then basically, we have to send an email summary of why it's not approved, summarizing the conditions and the risks that are highlighted. If it is approved, then we go through a similar flow as before and export it in an appropriate format. So that's the high level there. (46:51 - 47:04) Let me walk through an actual example here. So this is a run I did just before this webinar. But essentially, this is the procurement, synthetic procurement doc that we have for this scenario. (47:05 - 47:23) So Poseidon Optics, this is the quote of the itemized materials that we're procuring. And you can see here, there is that price table, right? There's a few terms and conditions captured here. We could read through them, but I want to more so walk through the agent output. (47:24 - 47:44) So what happens here is we can see those steps translated into the task that we performed in this workflow. We can see, first, we have that document analyzer, which essentially, one thing that's really great is we can see all the steps and all the little subtasks that were performed. So all the calls along the way. (47:44 - 47:57) So it's extracting all of these pieces of information. So it sees the net payment terms, it's collecting that information, and then basically summarizing it in its output for us. So this is the document analyzer. (47:58 - 48:15) Eventually, it makes a determination. And essentially, we can go to that human review step, and we see that determination is that it is high risk. So it scores at 7 out of 10 in terms of risk, summarizes the information here, and has a table with the main risks that are highlighted. (48:15 - 48:35) So one, the top one here is basically additional financial surcharges. So the summary here is that there's a quoted price, but then there's also an additional 5% handling fee. And for this particular quote, it accounts for or equates to about $600,000, which is huge. (48:35 - 48:55) If you miss this line, that's a $600,000 mistake, you're making a $600,000 decision based off that one piece of information. The other things that are highlighted is essentially some price volatility. There's information in the document about basically a 14 day validity period. (48:55 - 49:22) And if we go beyond that period, the price will increase automatically by 8%, which is almost a million dollars here. So huge risks that exist. If I take us back to the document really quickly, to kind of round us off on this demonstration, we can see these terms are captured here, this kind of logistics handling fee of 5%, and then the validity period over here. (49:23 - 49:39) So quotation valid for 14 days, and that 8%. So all embedded within the terms and conditions of the document. Ultimately, after that kind of review, it makes a bit of a recommendation on how we should go forward in terms of negotiation points. (49:40 - 49:48) And what I decided was to not approve this. Actually, if I go back here, I didn't approve it. And I said the price volatility is too high and the terms are not favorable. (49:49 - 50:11) So ultimately, that then triggers the email workflow. And I'll just take us directly to my drafts here. And you can see, essentially, I'm emailing our procurement department about this specific quote, and it summarizes all of these details for me, the key concerns, and essentially our recommendations on how to address this. (50:11 - 50:42) So basically, the fact that we've created this multi-agent system, connected it to the necessary tools it needs to do its work, whether that's files I can upload, whether it's a data interpreter, or at the very end here, an email tool, it's able to make decisions, reason over it. We can check those reasonings, and then it can actually, all the way to draft emails on our behalf. So taking a multi-hour process and making it minutes. (50:43 - 51:06) So it's quicker time to value, higher accuracy, and huge cost savings overall. So that is the document procurement workflow that I wanted to talk through. The last thing I'll touch on before I open it up to questions for five minutes here is, when I talk about governance, I'll give a sneak peek of some of what that looks like. (51:06 - 51:35) So one thing that's top of mind for a lot of folks is, okay, which agents are being used? What does the usage look like? So I'm opening the back end here a bit and going to the analytics panel. So this is a demo environment that I'm showing you, but essentially you can see what the number of active users are over time, for example, for this demo environment. And unsurprisingly, if you can look at all these dips here, you can probably take a guess at what these dips are. (51:35 - 51:53) You'll find that they correlate with weekends, which there's less people logging in on weekends, which is understandable and good for recharging. You can also see the message count over time. So you'll see that even during some weekends, there are some long running automations that are happening in the background. (51:53 - 52:14) So actually there could be large numbers of messages being passed, even though people aren't signing in, just because a lot of the work is happening in the background. You can see the tokens that are being passed through. So when Josh talked about people talking about token cost, token usage, and token explosion being top of mind for a lot of people, yes, that is definitely the case. (52:14 - 52:29) You can monitor the tokens that are being passed through here. You're not being charged on tokens, but it's still good to understand the load on the system, right? How much people are sending through the system. And then automations here. (52:29 - 53:30) So this is important to highlight because essentially I'm running an automation and I showed you one that worked out, but there will be times where based off the input, based off how someone has set up an automation, it might not work. And you want to be aware of where breakage exists, where people are not having success, where there's frustration in the system, right? So to be able to come into the backend and basically see successful runs, unsuccessful runs, is really key insight to having a stronger system, more value for users within the organization. So I don't want to go into the kind of the full depth of all the functionality you can see and all the governance, but I think that basically motivates what are some of the questions you can answer when you have stronger governance over your AI state and how is that helpful for you? So I will end the screen share here and maybe I'll turn it over to some questions that might come from the audience. (53:31 - 53:42) Yeah, absolutely. Let's see here. And really quickly, as those questions are getting pulled up for folks online too, you may have seen the MCP portion on North. (53:42 - 54:01) That's actually one of the areas we just didn't have time to cover it, where you can hook directly on into your Tableau MCP server, be able to get your answers directly within North. So in addition to working within the context of those documents, be able to bring everything together under a unified roof. Yeah, thank you both so much. (54:01 - 54:23) Those are excellent insights and examples that you shared. As we move into the Q&A portion, please feel free to submit any questions you have using the Q&A button located at the bottom of your screen. If you have a question, now is the best time to ask as Josh and Mitchell are the experts and we'll do our best to answer as many questions as we can as time allows. (54:32 - 54:46) Let's see. It looks like we have a question. How do you stop the model from seeing data a user is not authorized to see? Yeah, we can take the first crack at that. (54:47 - 55:09) And together, both combination of Cohere and Tableau can ensure that your data is as safe as possible. One of the primary ways on Tableau, first we have a robust series of permissions that apply to various groups, users, every single object, whether that's a dashboard, an ETL flow and data source, all directly within Tableau. We also have things like real-level security. (55:09 - 55:20) So maybe certain users are only supposed to see certain data. And what happens is that data then only gets passed on over to that user. So that's one of the ways to control at the ground level. (55:21 - 55:33) We also have virtual connections as well, where you can put in together various policies. And on the policy side, get a chance to see what those look like. Very robust, ton of different rules to be able to apply to those data sources. (55:33 - 56:17) Key thing about this is that as all those are being woven on in and a user is only supposed to see or get access to the data that they're supposed to be connecting on into, only that data makes its way on over to the LLM. On the MCP side of things, if you have a JSON web token, for example, or probably more likely OAuth within your MCP service, that will take on user permissions, privileges, and real-level security as well. So as you ask questions, just like I was you before in Slack, or even directly through Cohere North, what will happen is, again, only that data is viewable by the LLM and allowed to be analyzed, used, or generate the products based on that data. (56:17 - 56:49) I'll pass it over to Mitchell. Yeah, I think you provide a fantastic answer, so not much more to add. Essentially, we have this idea of delegated access where essentially we want the agent to operate on the user's behalf so it would understand the identity of the person that's tasking it, essentially, and then that agent only has access to the classification and scope of information that the users themselves would have access to. (56:52 - 57:10) Awesome, thank you both. Kind of the same question, when Cohere gives an answer, how does an analyst verify it? It's a great question. So one of the things that we trained out of the box is essentially for our models to be able to produce citations. (57:11 - 57:36) So in all of its responses, it doesn't just provide that text answer, it provides a text answer with inline citations. So you know that this sentence came from this part of the document that it's grounding on, right? So we provide the citations as one of ways for people to have trust and transparency in the system. Another one of the key ways is something I showed a bit in the demo, but essentially looking at the reasoning traces as well. (57:37 - 58:06) You can look directly at, based off the document it's ingested, which document is it accessing, right? Making sure it's accessing the right document. You have the full traceability and audit trail for how it got to its final response. You can look at the exact fields of the information that maybe OCR or visual reasoning model extracted from that PDF or that Word document, and then understand how that propagated downstream to the final response. (58:06 - 58:29) So really citations, the reasoning trace are two of the key ways that customers really understand and gain trust in the output of the LLM for verification. Awesome, thank you so much. Thank you all for joining us today, and a special thank you to Josh and Mitchell for sharing their insights and expertise. (58:30 - 58:44) We hope this webinar has been informative and valuable for you and your organization. If you have any additional questions or would like more information, please feel free to reach out to our Cohere team at ̽»¨ÊÓƵ. Their contact information is currently displayed on your screen. (58:45 - 58:47) Thank you again and enjoy the rest of your day.