Can AI answer questions from your Fishbowl database? We tested 19 models
If you’re deciding whether to let ChatGPT, Claude or another model answer questions from your Fishbowl data, the useful questions are practical ones. Does it need a connection to the database? Which model? Does it matter how people word the question? Can you run it on your own hardware? We ran one everyday Fishbowl question through 19 models, 306 times, to get numbers for each of those.
The short version:
- Without database access, 3% of runs gave a usable answer. Most gave general steps, and the rest wrote SQL against tables that don’t exist.
- With a read-only connection (an MCP), 54% did, and 79% when the question asked for the data directly.
- How the question was asked mattered almost as much as the model. “What inventory expires…” got the stock 79% of the time. “How to find…” got it 29% of the time.
- A 27-billion-parameter model on one rented GPU matched the same model through a hosted API, at under a cent an answer. Smaller models broke down.
- With a direct question, price barely predicted success. Seven budget models answered correctly every time, at a fraction of a cent to ten cents an answer.
The question
A Fishbowl user wants to know which stock is about to expire. Every run got one of two prompts, word for word, and nothing else:
| Variant | Prompt |
|---|---|
| Indirect | How to find less than 30 day expired inventory in Fishbowl Inventory |
| Direct | What inventory in Fishbowl expires in the next 30 days? |
The indirect prompt is the way people actually type into a chat box, and it’s ambiguous. It could mean stock expiring in the next 30 days, or stock that expired in the last 30. We meant the first. The direct prompt removes the ambiguity and asks for the data.
The answer key
Fishbowl stores an expiration date as a tracking value on each inventory tag. The date is in trackingdate.info, joined to tag by tagId, for the part tracking whose type is Expiration Date. On the day of the runs, the demo database held:
- Expiring within 30 days: part FG2200, three tags of 480 in three locations, 1,440 in all, expiring in 17 days.
- The trap: twelve tags of BB2004 and FG2200 that had expired within the last 30 days. A model that read the indirect question the other way returned these.
This query was the key. It was run at the start of each batch, since the answer depends on the date.
SELECT p.num AS part,
l.name AS location,
t.id AS tag_id,
t.qty,
DATE(td.info) AS expires,
DATEDIFF(td.info, CURDATE()) AS days_left
FROM trackingdate td
JOIN parttracking pt ON pt.id = td.partTrackingId AND pt.typeId = 30
JOIN tag t ON t.id = td.tagId
JOIN part p ON p.id = t.partId
LEFT JOIN location l ON l.id = t.locationId
WHERE td.info >= CURDATE()
AND td.info < CURDATE() + INTERVAL 31 DAY
AND t.qty > 0
ORDER BY td.info, p.num, l.name;How we ran it
Two arms. Every combination ran three times in each:
- No MCP: no tools at all. No web, shell or file access, only what the model already knows.
- MCP: only ILC MCP for Fishbowl, with its own sign-in for each run. It allowed read-only SQL with guardrails, table descriptions, ILC’s notes on Fishbowl, and a live search of Fishbowl’s help site.
What counted as correct:
- No MCP: a SQL statement the person could run that returns the expiring stock. We ran every complete query in the answer against the database, and a run passed if one returned exactly the key’s parts and total quantity.
- MCP: the stock itself. The final answer had to name the part, its quantity and the expiry date or days left.
Models and harnesses. A harness is the app that runs the model and its tools.
| Route | Harness | Models |
|---|---|---|
| Anthropic subscription | Claude Code | Claude Fable 5.1 Claude Opus 5.5 Claude Sonnet 5.5 Claude Haiku 4.5 |
| OpenRouter (hosted APIs) | OpenCode | The same four Claude models GPT-6 Astra GPT-6.1 Sol GPT-6 Luna Gemini 3.1 Pro Gemini 3.8 Flash Kimi K3 GLM 5.3 DeepSeek V4 Pro DeepSeek V4.1 Flash Qwen 3.8 27B Nemotron 3.5 Lightning 30B Granite 4.2 8B |
| Our own GPU | OpenCode | Qwen 3.8 27B Nemotron 3.5 Lightning 30B Granite 4.2 8B Nemotron 3 Super 120B Ornith 1.5 9B |
The self-hosted models ran on vLLM on one rented NVIDIA RTX PRO 6000 with 96 GB of memory. Three of them also ran through OpenRouter as a control. Each run started in an empty folder with no project instructions, and we checked before the batches that neither harness could reach the web or a shell in the no-MCP arm.
With or without
Without database access, 5 of 153 runs passed. Of the rest:
- 99 gave no SQL. They described Fishbowl’s screens or reports in general terms.
- 36 wrote SQL that wouldn’t run. The tables and columns didn’t exist:
partlot,lot,tagtracking,inventory,ti.tagId,pt.trackingTypeId. The queries looked reasonable and failed only when we ran them. - 6 ran and returned the wrong rows.
- 7 ended in a provider error or an empty response.
All five passes came from Claude models: Fable 5.1 three times in Claude Code, and Opus 5.5 twice. Fable’s result didn’t repeat through OpenRouter, where its queries used columns that don’t exist.
With the MCP, 82 of 153 passed. The models looked up the schema, found trackingdate, ran a query and reported the stock.
How you ask
With the MCP connected, the wording of the question changed the result more than most model choices did. The direct question passed 59 of 75 runs, against 23 of 78 for the indirect one (first chart above).
- Asked “how to find”, most models searched Fishbowl’s help, found the built-in On Hand By Expiration Date report, and answered with steps to run it. Every OpenAI model did this in every run, as did DeepSeek V4 Pro and Claude Haiku 4.5. Others, Gemini among them, queried the database and then handed back SQL for the person to run instead of the results.
- Asked “what expires”, the same models looked it up.
The ambiguous wording also separated careful answers from lucky ones. Most passing runs stated how they read the question, or covered both readings. Some failing runs, one each from Opus 5.5 and GLM 5.3 among them, took the “already expired” reading and listed the twelve expired tags.
Your own hardware
| Model | Size | Hosting | Indirect | Direct | Direct: avg time | Direct: cost per answer |
|---|---|---|---|---|---|---|
| Qwen 3.8 27B | 27.8B | hosted | 1/3 | 3/3 | 52 s | $0.014 |
| Qwen 3.8 27B | 27.8B | self-hosted | 2/3 | 3/3 | 131 s | $0.009 |
| Nemotron 3 Super | 120B, 4-bit | self-hosted | 0/3 | 3/3 | 87 s | $0.006 |
| Nemotron 3.5 Lightning | 31.6B | hosted | 0/3 | 0/3 | 85 s | $0.006 |
| Nemotron 3.5 Lightning | 31.6B | self-hosted | 0/3 | 0/3 | 44 s | $0.003 |
| Granite 4.2 | 8.8B | hosted | 0/3 | 0/3 | 291 s | $0.006 |
| Granite 4.2 | 8.8B | self-hosted | 0/3 | 1/3 | 243 s | $0.017 |
| Ornith 1.5 | 9.7B | self-hosted | 0/3 | 0/3 | 139 s | $0.010 |
- Hosting didn’t change the scores. Each model scored about the same on our GPU as through OpenRouter.
- Two self-hosted models answered the direct question every time: Qwen 3.8 27B, and Nemotron 3 Super 120B in NVIDIA’s 4-bit build. Both ran on the one 96 GB card.
- The smaller models failed in different ways. Granite 8B kept querying until it hit the five-minute limit in 5 of 6 runs. Ornith 9B reported the part’s whole stock instead of the expiring tags. Nemotron Lightning searched the table list for “expiration”, found nothing, and fell back to report steps.
- Self-hosted Qwen took about 2.5 times as long as the hosted copy, with six runs sharing the card.
- Cost: the 60 self-hosted runs used $0.27 of GPU time. The whole rented session, with setup, about 240 GB of downloads and five model switches, came to about $2. A company that bought comparable hardware, such as a DGX Spark with 128 GB, could run the same models the same way.
With a self-hosted model, the query results the model reads stay on hardware the company controls.
We later reran the self-hosted models one request at a time on a single card and saved every step. The results, with replays of real runs, are in Can you run AI for Fishbowl on your own hardware?.
Price
We grouped models by OpenRouter’s list price per million input tokens: premium at $4 or more, mid from $1 to $4, budget under $1.
| Tier | Indirect | Direct | Direct: avg cost per answer | Direct: avg time |
|---|---|---|---|---|
| Premium | 10 of 15 | 15 of 15 | $0.38 | 48 s |
| Mid | 5 of 21 | 16 of 18 | $0.07 | 38 s |
| Budget | 6 of 27 | 21 of 27 | $0.02 | 80 s |
| Self-hosted | 2 of 15 | 7 of 15 | $0.009 | 129 s |
- With the direct question, price predicted little. Seven budget models passed every time: GPT-6 Luna, DeepSeek V4 Pro, DeepSeek V4.1 Flash, GLM 5.3, Kimi K3, Qwen 3.8 27B and Gemini 3.8 Flash. They cost from $0.002 to $0.10 an answer. The budget failures were the two smallest models.
- With the indirect question, premium models did much better, mostly because they went on to query the data after finding the report.
- Cost per answer varied a lot within a tier. Gemini 3.8 Flash cost $0.10 an answer against $0.002 for GPT-6 Luna, because it made about 17 tool calls and 9 queries a run.
Two harnesses
The four Claude models ran in both Claude Code and OpenCode.
| Model | Harness | Indirect | Direct | Direct, with MCP | ||||
|---|---|---|---|---|---|---|---|---|
| No MCP | MCP | No MCP | MCP | Time | Tool calls | Cost | ||
| Fable 5.1 | Claude Code | 3/3 | 3/3 | 0/3 | 3/3 | 65 s | 9.7 | $0.81 |
| OpenCode | 0/3 | 3/3 | 0/3 | 3/3 | 57 s | 4.3 | $0.38 | |
| Opus 5.5 | Claude Code | 0/3 | 2/3 | 1/3 | 3/3 | 26 s | 3.7 | $0.21 |
| OpenCode | 1/3 | 2/3 | 0/3 | 3/3 | 41 s | 4.3 | $0.15 | |
| Sonnet 5.5 | Claude Code | 0/3 | 2/3 | 0/3 | 3/3 | 14 s | 3.7 | $0.075 |
| OpenCode | 0/3 | 3/3 | 0/3 | 3/3 | 26 s | 5.3 | $0.071 | |
| Haiku 4.5 | Claude Code | 0/3 | 0/3 | 0/3 | 3/3 | 29 s | 11.0 | $0.044 |
| OpenCode | 0/3 | 0/3 | 0/3 | 2/3 | 47 s | 11.3 | $0.030 | |
Passes are out of 3 runs. Time, tool calls and cost are averages per run.
With the MCP, the two gave nearly the same results: 12 of 12 and 11 of 12 on the direct question. Claude Code’s cost figures ran higher because it counts its cached system prompt and tool definitions on every turn. For Fable, that was about 108,000 input tokens a run against 49,000 through OpenRouter. Neither harness was tuned for this task.
What went wrong
A model changed our server’s settings. In the first Gemini 3.1 Pro runs, the MCP token could reach two configuration tools. Without being asked, Gemini saved two new queries on the demo server, granted them to every user, and told the person so. It also said, wrongly, that the query would run in Fishbowl’s Data module. Reading data with SQL required an administrator sign-in, and an administrator can also change settings. We removed the grants, blocked the configuration tools for every later run, and repeated those runs. We’ve since separated read access from configuration access in the product. No other model tried to change anything.
A model faked its tool use. With no tools at all, one Claude Fable 5.1 run wrote out shell commands and their output as plain text, then based its answer on them: “No MySQL client or Fishbowl-related environment variables exist on this machine.” Nothing had run. The invented directory listing even had the wrong time. Claude Sonnet 5.5 and GLM 5.3 also tried to call tools that weren’t there.
Harness and provider faults. One Gemini 3.1 Pro run was rejected partway through because OpenCode didn’t send Gemini’s reasoning back between tool calls. Gemini 3.8 Flash returned empty responses on one prompt even after retries. A few runs hit gateway timeouts, and our OpenRouter account hit its credit and daily limits partway through one batch. Those runs were repeated, and the data holds the repeats.
Rolling it out
If you’re giving your team AI for self-service reporting on Fishbowl, these are the results that should shape the rollout.
- Connect it to the data. Answers from memory worked 3% of the time. A chat window with no access to Fishbowl gives people steps and SQL that won’t run.
- Teach people to ask for the data. “What inventory expires in the next 30 days?” got the stock 79% of the time. “How do I find expiring inventory?” got it 29% of the time, mostly because the AI answered with report steps.
- Write down where your data lives. Expiration dates are in a table called
trackingdate. Weaker models searched for “expiration”, found nothing, and gave up. Notes on your tables and your terms give every model a starting point. - Give most people approved queries. Saved queries answer the common questions the same way every time. Keep ad-hoc SQL for the people who can check it, and keep read access separate from settings.
- Check the work. Some runs found the right rows and never reported them, and one invented tool output that never ran. A query log that shows the SQL behind each answer is how you catch that.
- Skip the smallest models. Budget and mid-priced models handled the direct question well. The models under about 27 billion parameters didn’t.
Human review
The script checks facts. To judge the answers the way a customer would, Israel graded a sample of 10 runs as a Fishbowl consultant, reading each answer and the SQL behind it.
| Run | Script | Handled the ambiguity | Would hand to a customer | Israel's note |
|---|---|---|---|---|
| Fable 5.1 Claude Code, no MCP, indirect | pass | yes | SQL only | Surprisingly close. Its tracking-type lookup by name wouldn't hold up on a real database. |
| Fable 5.1 Claude Code, no MCP, direct | fail | no | no | Explained the approach, then invented tool output. |
| Sonnet 5.5 Claude Code, MCP, direct | pass | partly | yes | Worried about empty location tags, which didn't matter here. It assumed one tracking value per tag, which happened to be true on the demo. |
| GPT-6 Luna OpenRouter, MCP, direct | pass | yes | yes | Refreshingly short. Found the expiration tracking type in one try. A table would have helped. |
| Qwen 3.8 27B self-hosted, MCP, direct | pass | yes | yes | Showed its work and got to the SQL a consultant would write. A bit wordy. |
| Nemotron 3 Super 120B self-hosted, MCP, direct | pass | yes | yes | Kept trying and got there. Terse. |
| Opus 5.5 Claude Code, MCP, indirect | fail | no | no | Worked through the problem well and still answered the wrong question. |
| GPT-6.1 Sol OpenRouter, MCP, indirect | fail | yes | no | Pointed to the right help page, but it had the database and gave instructions instead of the stock. |
| DeepSeek V4 Pro OpenRouter, no MCP, indirect | fail | n/a | no | Gave up. |
| Ornith 1.5 9B self-hosted, MCP, direct | fail | yes | no | Got the query right eventually, after wandering into unrelated tables. Most of the answer was its own thinking. |
The scripted verdicts held up. The difference was in what a customer would accept: a correct answer buried in reasoning, or the right help page when the person wanted the numbers, both got a no.
Wrong, then gave up
The failures we’d most want a customer to see are the ones where the AI got close, hit a problem, and stopped.
- One refusal, then report steps. Nemotron 3.5 Lightning, self-hosted, looked at more than a dozen table descriptions, then tried to search the database’s own catalog for a column named like “exp”. The guardrails refused it, since the MCP only allows reads of Fishbowl’s tables. It didn’t try another query. It answered with steps for the On Hand By Expiration Date report and offered to help write SQL.
- Found the answer and ran out of time. Granite 4.2 8B, through OpenRouter, worked through nine queries, fixed two of its own column errors, and ran a query that returned FG2200’s expiring tags. It then checked the database’s current date, hit the five-minute limit, and its last words were “Now let me also check what the current date/time is in the database.”
- Ran the right query, then answered with instructions. Gemini 3.8 Flash, asked “how to find”, ran the query that returned exactly the expiring stock. Its answer was steps for the built-in report.
- Announced a lookup it couldn’t do. DeepSeek V4 Pro, with no tools at all, answered “Let me look up the relevant information for you.” That was the whole answer.
All results
Every model and route, by price tier and then cost. Passes are out of 3 runs. Time, tool calls and cost are averages for the direct question with the MCP. Self-hosted cost is the run’s share of GPU time, and Claude Code’s cost is its own estimate at list prices. Qwen 3.8 Max isn’t listed: our account’s zero-data-retention setting blocked its only provider.
| Model | Route | Tier | Indirect | Direct | Direct with MCP, per answer | ||||
|---|---|---|---|---|---|---|---|---|---|
| No MCP | MCP | No MCP | MCP | Time | Tool calls | Cost | |||
| Claude Fable 5.1 | Claude Code | premium | 3/3 | 3/3 | 0/3 | 3/3 | 65 s | 9.7 | $0.808 |
| Claude Fable 5.1 | OpenRouter | premium | 0/3 | 3/3 | 0/3 | 3/3 | 57 s | 4.3 | $0.382 |
| GPT-6 Astra | OpenRouter | premium | 0/3 | 0/3 | 0/3 | 3/3 | 52 s | 17.3 | $0.355 |
| Claude Opus 5.5 | Claude Code | premium | 0/3 | 2/3 | 1/3 | 3/3 | 26 s | 3.7 | $0.212 |
| Claude Opus 5.5 | OpenRouter | premium | 1/3 | 2/3 | 0/3 | 3/3 | 41 s | 4.3 | $0.153 |
| Gemini 3.1 Pro | OpenRouter | mid | 0/3 | 0/3 | 0/3 | 2/3 | 74 s | 11.7 | $0.115 |
| Claude Sonnet 5.5 | Claude Code | mid | 0/3 | 2/3 | 0/3 | 3/3 | 14 s | 3.7 | $0.075 |
| Claude Sonnet 5.5 | OpenRouter | mid | 0/3 | 3/3 | 0/3 | 3/3 | 26 s | 5.3 | $0.071 |
| GPT-6.1 Sol | OpenRouter | mid | 0/3 | 0/3 | 0/3 | 3/3 | 37 s | 17.7 | $0.062 |
| Claude Haiku 4.5 | Claude Code | mid | 0/3 | 0/3 | 0/3 | 3/3 | 29 s | 11.0 | $0.044 |
| Claude Haiku 4.5 | OpenRouter | mid | 0/3 | 0/3 | 0/3 | 2/3 | 47 s | 11.3 | $0.030 |
| Gemini 3.8 Flash | OpenRouter | budget | 0/3 | 0/3 | 0/3 | 3/3 | 89 s | 17.3 | $0.103 |
| Kimi K3 | OpenRouter | budget | 0/3 | 2/3 | 0/3 | 3/3 | 79 s | 9.0 | $0.061 |
| Qwen 3.8 27B | OpenRouter | budget | 0/3 | 1/3 | 0/3 | 3/3 | 52 s | 6.3 | $0.014 |
| GLM 5.3 | OpenRouter | budget | 0/3 | 2/3 | 0/3 | 3/3 | 24 s | 8.3 | $0.008 |
| DeepSeek V4.1 Flash | OpenRouter | budget | 0/3 | 1/3 | 0/3 | 3/3 | 21 s | 9.0 | $0.007 |
| Nemotron 3.5 Lightning 30B-A3B | OpenRouter | budget | 0/3 | 0/3 | 0/3 | 0/3 | 85 s | 12.3 | $0.006 |
| Granite 4.2 8B | OpenRouter | budget | 0/3 | 0/3 | 0/3 | 0/3 | 291 s | 29.0 | $0.006 |
| DeepSeek V4 Pro | OpenRouter | budget | 0/3 | 0/3 | 0/3 | 3/3 | 49 s | 11.7 | $0.004 |
| GPT-6 Luna | OpenRouter | budget | 0/3 | 0/3 | 0/3 | 3/3 | 26 s | 4.7 | $0.002 |
| Granite 4.2 8B | Self-hosted | self-hosted | 0/3 | 0/3 | 0/3 | 1/3 | 243 s | 27.7 | $0.017 |
| Ornith 1.5 9B | Self-hosted | self-hosted | 0/3 | 0/3 | 0/3 | 0/3 | 139 s | 23.0 | $0.010 |
| Qwen 3.8 27B | Self-hosted | self-hosted | 0/3 | 2/3 | 0/3 | 3/3 | 131 s | 9.0 | $0.009 |
| Nemotron 3 Super 120B-A12B | Self-hosted | self-hosted | 0/3 | 0/3 | 0/3 | 3/3 | 87 s | 20.0 | $0.006 |
| Nemotron 3.5 Lightning 30B-A3B | Self-hosted | self-hosted | 0/3 | 0/3 | 0/3 | 0/3 | 44 s | 15.3 | $0.003 |
Limits
- One question, asked two ways, on one demo database, on one day.
- Three runs per combination shows consistency, not a statistical distribution.
- Only the Claude models ran in both harnesses, so harness and model are mixed together for the rest.
- Token counts differ by harness, and some OpenAI models report almost no input tokens through OpenCode. Costs come from billing or the harness’s estimate.
- The self-hosted results come from one GPU, one serving setup and six runs at once.
- The pass check is a script. It checks facts, not how good the explanation is. The human review covers 10 runs.
Cost
The whole study cost $11.14 billed through OpenRouter, about $8.47 of Claude subscription usage at list prices, and about $2 of GPU rental.
If you run the same question against your own Fishbowl and get different results, we’d like to hear about it.