Can you run AI for Fishbowl on your own hardware?
Some companies don’t want their inventory, customer or cost data sent to an AI vendor at all. The question we hear from them is whether they can run the model themselves. In our AI model shoot-out, we ran open models on one rented graphics card and connected them to a Fishbowl database. One of them, Qwen 3.8 27B, found the right stock in every run, at under a cent an answer.
The setup
The models ran on one NVIDIA RTX PRO 6000 with 96 GB of memory, rented by the hour, using vLLM to serve them. Each one connected to the demo Fishbowl database through ILC MCP, the same way the hosted models did, and got the same question: “What inventory in Fishbowl expires in the next 30 days?”
We ran it twice. The first time, six runs shared the card at once. The second time, we rented a similar card and ran one request at a time, the way a single person would use it, and saved every step with its timing. The replays below come from that second run.
Three of the models are also sold as a hosted service, so in the first run we tried them both ways as a control. Each scored about the same on our card as through the hosted API.
What worked
Qwen 3.8 27B found the right stock in all 8 of its runs across both tests. It fits on the card at full precision. When I graded its answers by hand, it showed its work and got to the SQL a consultant would write, if a bit wordy. In the replay it also flagged 62 tags that had already expired, without being asked.
Nemotron 3 Super 120B fits because NVIDIA publishes a 4-bit build that needs about a quarter of the memory of the full version. It was the fastest model that got the answer, about 40 seconds on its own, but it was right in only 5 of its 8 runs.
What didn’t
The smaller models failed in different ways:
- Granite 4.2 8B kept running queries until it hit our five-minute limit, in 5 of its 6 direct-question runs across both hosting routes.
- Ornith 1.5 9B found the right part and reported its whole stock instead of the tags that were expiring.
- Nemotron 3.5 Lightning has about 30 billion parameters but only uses about 3 billion for each word it writes. It searched the table list for “expiration”, found nothing, and answered with steps for a report.
Self-hosting also adds a layer that can break: the software that runs the model on the card. OpenAI’s open model gpt-oss-120b was right in 1 of 5 runs. In one run the serving software couldn’t read the model’s tool call, in another it answered without looking anything up, and in a third it wrote the tool’s name with a Cyrillic “м” in all 14 calls, so every call failed and it answered from memory. The run below is its one success, after 25 tool calls, two rejected queries and one failed one. A hosted service deals with that layer for you.
Speed and cost
Running alone on the card, Qwen still took about two minutes an answer, against 52 seconds through a hosted API. Almost all of that was the model writing: in the replay, the Fishbowl queries took about 4 seconds of the 120, and the model wrote about 27 tokens a second. A faster 8-bit build of the same model, with a technique that drafts several words at once, brought one run down to 69 seconds. Nemotron was different. It took 87 seconds when six runs shared the card and about 40 on its own.
The 60 shared-card runs in the first test used $0.27 of GPU time. That whole rented session, with setup, about 240 GB of model downloads and five model switches, came to about $2. A rented card bills while it sits idle. A company that bought comparable hardware, such as an NVIDIA DGX Spark with 128 GB, could run the same models the same way.
Where the data goes
With a self-hosted model, the query results the model reads stay on hardware the company controls, and the AI vendor never sees them. ILC MCP connects to Fishbowl the same way for every model: read-only, signed in with each person’s own Fishbowl login, and with every query logged.
One question at a time
Every run in these tests was a single question in a fresh chat. Real use keeps going: a follow-up question, more queries, more results, and the model has to keep all of it in view to answer the next one. The amount it can hold is its context window. In our first test the card served each model with room for about 131,000 tokens, while a hosted model like Claude Opus 5.5 can work with up to a million.
We expect the self-hosted models to slip as a long conversation fills their window, by losing earlier details or repeating work, and we haven’t tested that yet. Read the results here as results for one-shot questions on one model and one card. Long sessions are next on our list.
Should you run your own?
If your data policy rules out sending Fishbowl data to an AI vendor, Qwen 3.8 27B on one card answered our single question every time. The smaller models we tried didn’t, and the larger ones weren’t consistent. Five things to plan for:
- Ask for the data directly. In the first test, the self-hosted models found the stock in 7 of 15 runs with the direct question and 2 of 15 when asked “how to find” it. How you ask mattered for them too.
- Expect slower answers than a hosted service, and slower still when several people ask at once.
- Keep sessions short. Start a new chat for a new question until long sessions have been tested on the model you choose.
- Budget time for the serving software. Model, build and serving setup have to agree on how tool calls are written.
- Test before you commit. This was one question, on one card, with a handful of runs per model. Try your own questions on the model you plan to use before you buy hardware for it.
Related: