Restack Gives Product Teams the Reins To Own AI Agent Behavior
Andrés Tapia, co-founder and CEO of Restack, says companies are going about AI all wrong.
Tapia and his fellow co-founder and CTO, Philippe Brulé, first worked together 10+ years ago at Mister Spex, an omnichannel optician. After going their separate ways, the pair teamed up again to help companies wrangle their data: how to move it; how to edit it.
Thus was the first iteration of Restack: a cloud service for data engineers to deploy applications with one click. For about two years, that was the mission — until ChatGPT entered the scene in 2022 and completely upended technology, work, and the way we think about everything.
Suddenly, Restack’s customers began approaching Tapia with new requests: “‘I’m using Restack for moving data from here to here,’ Tapia paraphrases. ‘We are now starting to build some AI products. Can we use your platform as well for these AI products?’”
Technically, he says not much needed to change to suit the new demands — but it proved a major turning point. Soon, almost 90% of their users were seeking help with AI: “So we said, ‘We’re going to stop what we[‘re] doing, and we’re going to focus 100% on the AI use case,” Tapia recalls.
But they weren’t the only ones moving that way.
So Many New AI Tools — So Many New Mistakes
Just as quickly as ChatGPT and near-ubiquitous AI agents seemingly cropped up overnight, so did the tools to help build AI products: Langchain, Bercel, etc. And despite its relative newness, the process of building an agent has quickly become relatively standard: Define what your agent is supposed to do (for many, that’s customer support); connect the right tools and data sources (like FAQs); go live.
“But the problem is, it doesn’t end there,” says Tapia. Once you go live…, your agent is answering real people — and [it] start[s] to hallucinate.”
One healthcare provider learned this the hard way before coming to Restack.
They had created a customer support agent to field questions like, “How much is X going to cost?” and “Does my insurance cover this dentist or not?” Handy in theory — but not when most of the answers to those questions are wrong.
“They ha[d] a lot of cases where… the customer complained, ‘I went to a dentist, and actually, this dentist is not covered with my policy,’” Tapia explains.
Unfortunately, anyone who’s chatted with a bot knows these hiccups aren’t uncommon.
Tapia thinks he’s pinpointed the root of the failure. From his perspective, most companies overlook the second — and very crucial — part of creating AI agents: testing behavior.
The Part Everyone Misses: Testing Agent Behavior (Not Just Outputs)
“First, you build the technical part. But then…you need to ensure the agent works properly — that’s [where] we didn’t see that there was anything in the market,” Tapia says — and where he says Restack offers something different.
Companies start by building an agent. Then, they can define what Tapia calls “behavioral metrics,” qualifiers that go beyond the typical functional or performance metrics most tools rely on.
For example, for the aforementioned healthcare provider, behavioral metrics include questions like, “Is my agent mentioning my competitor?” “Is my agent checking the policy before it recommends a dentist?”
Unlike objective signals (like agent response time or resolution rate), the answers to these behavioral questions are subjective and thus not easily checked with a simple function or rule-based test — but they do more to stop chatbot answers from going sideways.
There’s another layer of evaluation, too. “We [have] this LLM as a judge,” explains Tapia. “Basically, another LLM is going to check if this LLM is actually doing the right thing or not.”
And if the agent doesn’t meet the LLM judge’s expectations? Then it’s time to iterate, i.e., “change the prompts; change the tools you provide; change the context,” he says.
When AI Goes Live, the Mistakes Show Themselves
It’s the behavioral metrics and the test-and-iterate process that Tapia says most companies are missing when they begin AI projects.
“Every company wants to implement AI,” he says. “They know in the back of their head that—somehow—they could save money and be more efficient, [but they do] not exactly [know] the use case.”
Understandably, a lot of companies are jumping into AI eyes-closed, headfirst because they fear the oft-touted maxim: “Join in, or get left behind.” And product teams are often the ones under pressure to figure it out and deliver — fast.
In a search for viable use cases, customer support usually seems like a natural fit. The next step is assessing available frameworks and then kicking it to engineering teams to tinker for a few months before going live.
That’s when Tapia says the mistakes reveal themselves.
During the first live test (say, sending 10% of customer support conversations to the agent), “People [i.e., customers] start to complain,” says Tapia. “And then management complains… Then the product team sa[ys], ‘This is the product I ship; I’m responsible for what it’s doing.’”
But when product teams circle back to engineers to ask what went wrong and how to fix it, answers aren’t cut and dry.
There’s No Quick Fix
From the engineering side, everything was right — right tools, right integrations. But the non-deterministic nature of LLMs means outputs are still going to vary.
It’s not encouraging. Perhaps that’s why Gartner predicts 40% of agentic AI projects will be cancelled by 2027. Internally, Tapia says discussions often echo the same concerns:
“[For companies that…build the[ir] first agent…, there [is] a big chunk of people saying, “AI—bye. It doesn’t work. Let’s stop it.’”
It’s only with these failures, he claims, that teams realize they can’t evaluate behavior with a classic Q&A. Namely, Tapia says, they realize “‘We need to test in a different way. We need to call the agent ourselves, try to simulate cases, and see… the performance of the agent and the behavior for those questions.'”
But that testing is easier said than done.
“When the product team wants to test that, they find out they can’t — because everything was built from a point of view of software development,” explains Tapia. “The tool is built there. If they want to change anything, they need to go to the software team and tell them, ‘Change the prompt. Change this tool. Remove this context from the context store.’”
Obviously, a multistep iteration process adds bulk and time. But building AI agents is new terrain, so it makes sense that companies revert to known processes for software development.
There’s the catch, Tapia points out. He argues that building AI agents isn’t a software development challenge; it’s a product challenge. And until companies change their approach, he predicts continued failure ahead:
“More companies [are going to] jump into AI and be unhappy with the results…because, basically, they didn’t understand that AI is not the same as building a software solution.”
Stop Treating AI Like Software; Treat It Like an Employee Instead
So, how should companies think about AI implementation?
“You have to think about AI [the same] as you think about new employees in the company,” Tapia advises.
That means onboarding, training, and regular development so that, over time, both parties can better understand what needs to be done. This behavioral refinement, he claims, is the key to success — and what other tools overlook.
Done the Restack way, product teams are empowered to own AI agent behavior and test and ship it like normal product features.
“You build your agent the way you build it, but the product team is going to have a layer of features…so they can—by themselves in our user interface—iterate and change those prompts, tools, everything—without needing to go to the tech team.”
It’s a shift from the default AI deployment processes — one that gives product teams more control over agent behavior and perhaps finally brings it in line with user expectations.
