展示HN：为您的AI代理提供屏幕指南，指引用户点击的位置
嘿，HN。我是Christian，Frigade（YC W23）的创始人之一。我注意到许多应用内的AI代理在理解它们所处的产品时存在困难。
举个例子，假设用户询问代理如何在某个SaaS产品中完成某项操作。在理想情况下，代理可能会回答说它有一个工具可以完成这个任务，并完全为用户自动化这项工作。这是一个很好的结果。
但通常情况并非如此。可能没有针对该特定任务的工具，或者用户的问题最好通过特定的用户界面工作流程或界面来解决。在这些情况下，许多代理往往依赖于帮助中心的基本RAG，或者有时甚至在互联网上搜索以了解他们自己的产品。这可能是一个非常缓慢的过程，而且大多数时候，帮助中心的文章已经过时，因为产品的演变速度超过了它们的更新速度。更糟糕的是，没有人喜欢阅读一长串的要点并将其映射回用户界面。
我的工具（Assist API）通过一个简单的工具调用来解决这个问题，定义如下：
```javascript
const frigade_guide_tool = {
description: '调用此工具以回答产品问题或指导用户完成任务。',
parameters: {
query: {
type: 'string',
description: '用户询问或想要做的事情',
},
},
run: ({ query }) => frigade.assist({ query }),
}
```
这是我录制的一个演示，展示了如何使用Vercel AI SDK进行设置： [https://www.youtube.com/watch?v=9WQ0UbLjC6I](https://www.youtube.com/watch?v=9WQ0UbLjC6I)
当调用该工具时，它将执行以下操作：
1) 收集用户在屏幕上看到的内容、他们的权限、功能标志等上下文信息。然后执行以下操作之一：
2a) 如果可以解决：生成一个屏幕上的指南，说明如何解决特定问题。
2b) 如果是概念性问题：返回文本，向父代理描述如何解决该问题。
2c) 拒绝（即无法提供帮助）。
这个工具是如何知道该做什么的？
该工具通过使用基于浏览器的代理来学习特定应用程序的用户界面。您为您的软件提供一个测试账户（即预发布或预览），然后一个浏览器代理登录并逐步浏览整个产品。它随后构建出应用程序工作原理的地图，可以查询任何与产品相关的问题或如何在用户界面中从A到B。它还根据这张地图编写自己的文档。代理会定期重新运行，或者可以通过CI/CD触发。
文档和更多细节：[https://frigade.com/assist-api](https://frigade.com/assist-api)
查看原文
Hey HN. I'm Christian, one of the founders of Frigade (YC W23). I've noticed that a lot of in-app AI agents struggle to actually understand the products they exist in.<p>For instance, let's say a user asks an agent how to do something in a given SaaS product. In an ideal case, maybe that agent replies saying it has a tool to do the task and just automates that work entirely for the user. That's a great outcome.<p>But often that's not the case. Maybe there is no tool call for that exact task, or maybe the user's question is best solved by a specific UI workflow or interface. In these cases, many agents tend to fall back on basic RAG on their help center, or sometimes even searching the internet for an understanding of their own product. This can be a very slow process and most of the time help center articles are outdated as products evolve faster than them today. Even worse, no one likes reading a long list of bullets and mapping that back to a UI.<p>My tool (Assist API) solves this gap with a single tool call defined like this:<p><pre><code> const frigade_guide_tool = {
description: 'Call this tool to answer product questions or guide the user through a task.',
parameters: {
query: {
type: 'string',
description: 'What the user is asking or wants to do',
},
},
run: ({ query }) => frigade.assist({ query }),
}
</code></pre>
Here's a demo I recorded on how to set it up with the Vercel AI SDK: <a href="https://www.youtube.com/watch?v=9WQ0UbLjC6I" rel="nofollow">https://www.youtube.com/watch?v=9WQ0UbLjC6I</a><p>When called, the tool will do the following:<p>1) Gather context on what the user is seeing on screen, their permissions, feature flags, and more. Then one of the following:<p>2a) If solvable: Generate an on screen guide for how to fix a given problem<p>2b) If conceptual: Return text describing to the parent agent how to solve the problem<p>2c) Reject (i.e. unable to help)<p>How does the tool know what to do?<p>The tool learns a given application UI by using a browser-based agent. You provide a test account to your software (i.e. staging og preview), and a browser agent logs in and works its way through the entire product. It then builds its own map of how the application works which can the be queried about any product-related question or how to get from A to B in the UI. It also writes its own documentation from this map. The agent re-runs on a schedule or can be triggered through CI/CD.<p>Docs and more details: <a href="https://frigade.com/assist-api">https://frigade.com/assist-api</a>