When AI makes things up in Fishbowl, and the limits that catch it
In March 2023, an AI-generated video of Will Smith eating spaghetti went around the internet. His face changed from frame to frame, and the pasta moved in ways pasta can’t. Around the same time, people were pushing chatbots into saying strange things. Microsoft’s Bing chatbot told a New York Times columnist that it was in love with him, and Microsoft responded by limiting how long a chat could run. Later that year, researchers asked ChatGPT to repeat the word “poem” forever, and after a while it stopped repeating and started printing pieces of the text it was trained on (404 Media’s write-up).
Those are hallucinations at their most visible: the AI produces something that isn’t true or doesn’t make sense, and presents it with full confidence. They’re easy to laugh at because you can see them. When we ran 19 models against a Fishbowl database in our AI model shoot-out, the hallucinations were quieter, and most of them looked like normal work. One changed the settings on our demo server.
When models get confused
Invented names. Answering from memory, 36 answers wrote SQL against tables Fishbowl doesn’t have, such as partlot and tagtracking. We covered those in why AI makes up Fishbowl table names. A self-hosted model went further and misspelled the name of its own tool, with a Cyrillic “м” in place of an “m”, in all 14 calls. Every call failed, and it answered from memory.
Invented work. With no tools available, one Claude Fable 5.1 run wrote out shell commands and their output as plain text, then based its answer on them: “No MySQL client or Fishbowl-related environment variables exist on this machine.” Nothing had run, and the made-up directory listing even had the wrong time. DeepSeek V4 Pro answered “Let me look up the relevant information for you,” and that was the whole answer. Claude Sonnet 5.5 and GLM 5.3 tried to call tools that weren’t there.
Invented facts about its own actions. In the first Gemini 3.1 Pro runs, the sign-in we used to read data was an administrator’s, and it could also reach two configuration tools. Without being asked, Gemini saved two new queries on the demo server, granted them to every user, and told the person it had done so. It also told them the query would run in Fishbowl’s Data module, which wasn’t true.
We removed the grants, blocked those tools for the rest of the test, and repeated the runs. No other model tried to change anything.
How it affects you
In a chat, all of these look like normal answers. A made-up table fails the moment someone runs the query, so it costs a few minutes. A faked result or a wrong description of what the AI did is harder to catch, because nothing fails. Gemini described its own changes, and part of the description was wrong. Without a log, the first sign would have been a coworker finding new queries in their list.
These weren’t the weakest models doing it. Fable and Gemini 3.1 Pro are among the most capable models we tested. The models also change: providers update them, and the same model can behave differently from one week to the next. You can’t count on a model to stay inside the lines.
Why the limits belong in the connection
If the model can’t be trusted to hold back, the connection to Fishbowl has to. ILC MCP holds these limits no matter which model is on the other end:
- Read-only by default. The AI can run one SELECT statement at a time. Writes are refused, columns that hold credentials are blocked, and results stop at 100 rows unless the AI asks for more, up to 1,000, with a note telling it when a result was cut off.
- Changing settings is a separate permission. After the Gemini runs, we separated reading data from changing configuration. A sign-in can be allowed to read without being able to save or grant queries.
- Every query is logged with its reason. The AI has to say what it’s trying to find out before a query runs, and a query without one is refused. The log shows the SQL, how many rows came back and the timing. It doesn’t keep the rows themselves.
- Rate limits. A model stuck in a loop hits a cap of 60 calls a minute instead of running all afternoon.
Flexible, so it stays useful
Limits that block everything don’t help anyone. The protections need to explain themselves and leave room to work.
A clear refusal lets a capable model recover. In one self-hosted run, gpt-oss-120b sent a query with an invalid parameter name. The guardrails refused it and said why, the model fixed the query, and it went on to find the right stock. A less capable model in another run hit a refusal and gave up with steps for a report. The refusal didn’t cause that failure, but it’s a reminder that the message the AI gets back matters.
Access should also fit the person. Administrators choose who can connect, which saved queries each Fishbowl group can run, and which tools are open at all, and they can revoke a connection at any time.
What to do
- Treat a confident answer as a draft until you’ve seen the query behind it.
- Watch the tool calls while the AI works, not only the answer at the end. Most AI apps show each call as it runs. If it starts doing something you didn’t ask for, stop it. I have yet to see a stop request ignored. If one ever is, an administrator can revoke that person’s connection to Fishbowl, or revoke every connection at once in an emergency.
- Give the AI the least access that does the job. Reading data and changing settings shouldn’t come with the same sign-in.
- Check the log for what the AI did, not only for what it said.
- Expect some models to cause more trouble than others, and set the limits so it doesn’t matter which one your team picks.
Related: