-
Notifications
You must be signed in to change notification settings - Fork 57
Add GibsonAI toolkit #493
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Add GibsonAI toolkit #493
Conversation
@torresmateo Now I fixed them accordingly. Let me know what you think. |
raise ValueError(f"Validation error: {error_msg}") from e | ||
|
||
|
||
@tool(requires_secrets=["GIBSONAI_API_KEY"]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does GibsonAI gives full read and write access with this API key?
I would love to see this scoped to a scope like sql:delete
. Otherwise, any user with access to this tool may delete the database entirely, and the secret is shared for the entire project.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@torresmateo I can understand your concerns. Thanks for sharing them!
If you are thinking it is too risky to allow users to make edit, create or delete operations. Lets keep only Query tool and we are okay to showcase this capability. We can wait until you support remote MCP server connections.
We just onboarded our docs and there might be missing docs. But Nathan can reply soon for this question.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not immediately against tools that insert or delete data. I worry about tools that do so very openly and without scopes. This could be ok if the database has proper RBAC setup, but I can't see how that might work with the current setup, as the user executing the query will be identified solely by GIBSONAI_API_KEY
, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Keys are scoped with distinct permissions (e.g. read/write/delete)
These API keys are managed/named on the GibsonAI side (either via the CLI or web app)
We identify the project, specific database, and permissions all from the API key
The /query
endpoint that this tool uses parses the query to determine what types of statements are allowed:
- Schema statements, e.g.
CREATE
/ALTER
, are always forbidden SELECT
statements requireread
on the API keyUPDATE
/INSERT
statements requirewrite
on the API keyDELETE
statements requiredelete
on the API key
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @nathanchapman, this is indeed a better pattern than what I originally thought.
I have a couple of concerns about DML verbs, however, a couple of questions here:
- Arcade is a platform that is intended to offer tools in multi-tenant apps, and the API key will be set at the project level. As such, it seems that the only option for agent builders that need to store anything is to use a single overly-permissive API key. I'd love to hear your ideas on how an agent builder may implement a pattern where one user authenticates to GibsonAI and allows write operations, while allowing a different user to connect to the same database with read only access.
- @Boburmirzo, I think the validation attempt here is a good starting point, but I would prefer a specific allow-list type of validation. If you want to make this very versatile, perhaps this may come from the configured schema on the GibsonAI platform (e.g., internally check against the schema endpoint to see if what comes in the table name matches allowed tables, etc). For example, while I agree that escaping those strings in your
DeleteRequest
is useful, but it's one of the strongly discouraged defense options by OWASP, in favor of prepared-statements, stored procedures, and allow-list input validation. Of the top of my head, your approach is open to having a number of sub-queries pass the table name validation, for example.
|
||
|
||
@tool(requires_secrets=["GIBSONAI_API_KEY"]) | ||
async def insert_records( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here again I worry about allowing inserts too broadly. A lot of the AI-based SQL attacks are injecting instructions into tables through endpoints that allow write operations openly. @evantahler I would love your input on this.
I think:
- We should have purpose-built insert tools that validate each argument using prepared queries under the hood
- With the right permissions, we could ask the LLM to prepare a query and then someone with the right access may decide if it's worth running it or not.
@Boburmirzo could you point me to the GibsonAI permissions docs? I can't find it easily on the docs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here's generally how we are thinking about SQL tools - https://blog.arcade.dev/sql-tools-ai-agents-security. If tools are writing, it's best to scope down the permissions and then make very specific tools that only modify known params, rather than crafting the whole SQL. You could have a few secrets - GIBSONAI_API_KEY_WRITE
and GIBSONAI_API_KEY_READ
for different tools.
Yes, we care a lot about roles and access :D
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@torresmateo @evantahler I definitely understand where y'all are coming from.
on Arcade, is it possible for this particular tool to be a sub-call/agent-delegation from a user-implemented tool? e.g. the only path to gibson (as the storage layer) would be via an explicitly supported tool in a wrapping agent?
e.g. combination of this pattern: https://ai.pydantic.dev/multi-agent-applications/#agent-delegation-and-dependencies and what y'all mention in the Examples of Operational Tools
section of the post you linked above.
This would be nice in that it allows the LLM to still craft the exact query, but can scope it's capabilities to supported tool definitions
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Totally agree with what Nathan is highlighting here — I support the direction he's pointing to like proposing agent-side control to protect downstream tools.
From an Arcade product positioning perspective, one of the key opportunities I see is simplifying authentication and authorization between agents and providers like MCP. Right now, it feels like all the responsibility for securing these interactions is being pushed down to the provider side. But I believe this is actually a great moment for Arcade to shine — by taking on more of the security model (like how API Management platforms protect APIs), Arcade could become the trusted middle layer that standardizes and enforces access control between agents and tools.
This could also open doors for stronger partnerships and broader adoption, especially in enterprise settings where fine-grained control and auditability are non-negotiable.
Would love to hear your thoughts on that. @torresmateo @evantahler
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nathanchapman Arcade can't control how agent builders will connect the tools to their agent. So while it is possible to implement this pattern, it's not a guarantee that it will be a sub-agent. If I understand your suggestion correctly, this will put the enforcement of defenses such as prepared statements on the agent developer, rather than closer to the data.
I'm not against having a tool that stores memories, records, etc into a database, as long as they're tightly scoped and the roles for write operations are clearly delineated and only users with such access can do those operations.
@Boburmirzo While I can see your point, I do not agree that it's a good idea to move the protection boundary away from the data layer. For traditional APIs this may be more sensible, as we have to assume the developer is a trusted actor that will not intentionally allow SQL injections to happen. LLMs are not trusted actors in the system however, and should be outside of the trust boundary for the data layer.
|
||
|
||
@tool(requires_secrets=["GIBSONAI_API_KEY"]) | ||
async def update_records( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is another problematic tool for me with respect to data security. Without properly scoped permissions it could be too easy for somebody to replace all the data with something else
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is an artifact of @Boburmirzo trying to address your first comment on the PR.
I'd vote for a single tool (e.g. execute_query
) to keep the implementation as simple as possible and because we handle all permissions checks on our end on the API endpoint.
If we're able to protect the singular execute_query
behind explicit tool definitions in other agents, that'd make the most sense to me
Summary
Add support for GibsonAI database integration to the Arcade toolkit, enabling agents to evolve database schemas and run SQL queries against multiple relational databases.
What is GibsonAI?
GibsonAI is your AI-powered “database engineer” that lets you design, build, deploy, and scale production-ready serverless databases using natural-language prompts. You can create a database schema for your new project idea using the GibsonAI App by chatting with AI. Under the hood, GibsonAI spun up a new real MySQL/Postgres database. You can create multiple database environments and update the existing schema in AI chat.
For AI Agents, GibsonAI can be used as Test labs for AI Agent where you can run SQL queries in a sandbox environment and Durable Memory Layer – “Short, mid & long term memory for Agents”.
Description
Adding a new tool package
arcade_gibsonai
that provides database interaction capabilities with GibsonAI's Data API. The integration follows the similar patterns as existing database tools (likearcade_postgres
) but communicates via HTTP API instead of direct database connections.Motivation
Demo
I recorded the demo of using the new tool: https://drive.google.com/file/d/1tZYx7NnnBLu5z0ku1o7Kr7UYAq4Eo1M0/view?usp=sharing