Previously, I described why I built a RAG system for answering questions about the original series of Star Trek based on the content of episode transcripts. This post is more about how I did this. In addition to going over the implementation details of the system, it’s also a story of how I used LLMs to assist me actually writing software in different ways.

This project originally started in a scratchpad that I have of assorted scripts that call Ollama to play with various ideas. Many of these were “vibe-coded” one-off Python scripts from ChatGPT 5, which all called Ollama for local inference. I was already using Ollama because work wouldn’t give me an API key to use GPTs programmatically, so it became the primary inference provider for this system.I initially used Llama 3.1 8b for inference, as it was the model I liked the best at the time, that I could also run 100% on GPU with enough context to do the RAG lookup. Gippity recommended several vector databases to use for the RAG portion, and I initially selected Chroma. Given that I was using Chroma, all-MiniLM-L6-v2 was the embedding model I originally used, as that is the default in Chroma. I also added FlashRank to the system to do re-ranking.

After testing this locally in my terminal with running Python scripts directly, I started building out a UI for it here in Starbase Zebra. That was really an excuse to start playing with HTMX, which I had wanted to do for a while. After I had the basic endpoints running and had stubbed out the UI, I ended up sitting on the project for a bit because I didn’t want to pay for inference, and I didn’t want users of this thing to call my home PC. So, I put this thing on the shelf for a bit, and some time went by.

Meanwhile, at work, we finally got a usage-based Enterprise subscription for Claude Code, and I began using it for work. That was pretty inspirational for me, because I got a much more practical sense of what actual usage-based costs for inference were, and I realized that I could probably pay for an inference API to run Ship’s Computer and still keep the costs really low. So I started looking at API providers. After a lazy search, I found that DigitalOcean provides Serverless Inference at adequately competitive prices. The benefit of that is that I didn’t have to sign up for it; I already host Starbase Zebra on DigitalOcean. But there were so many models to choose from… how would I know which one to pick?

Enter Claude Code, and Trek Trivia from Apogee Software. Trek Trivia was originally released in 1988, in 10 volumes with 100 trivia questions about the original series of Star Trek. I happened to have copies of these games readily available, so I had 1000 multiple-choice trivia questions I could ask AIs to answer in a scripted manner, if I could find some way to extract them. I asked Opus 4.8 to try this, and it came up with an elaborate scheme involving runtime debugging, inspection of string tables, and screen scraping to try to see if it could extract all of the questions and the answer key. And it… promptly failed to succeed at that task, usually getting stuck trying to do the debugging to find the answer keys and input. Then Fable 5 became available, and I asked it to try this task; building off of what Opus had tried, it one-shotted the exercise.

With 1000 trivia questions and correct answers extracted, I now had a test set for figuring out which models to use. I had Claude also review the RAG code, and it recommended changing the embedding model away from all-MiniLM-L6-v2, because it truncates embeddings at 256 word-pieces, which loses a lot of fragment content. After trying 3 other embedding models, I replaced it with Qwen 3 Embedding 0.6b. Then I ran the entire test set of 1000 questions against a set of models locally. Once I had a few test runs done, I used them to figure out about how many tokens the entire thing would take, so I could then test a select number of models hosted in DigitalOcean to try with similar methods. I also used Claude Code in non-interactive mode with my Pro subscription to see how well some Claude models would do, for a comparison. I’ll post the whole table in another blog post, but here are the highlights:

• The average accuracy of all models, whether hosted locally or not, was 64.37%
• Generally, the larger a model it was, the better it did.
• My RAG system improved the accuracy of a model between 18 and 33 percentage points.
• The best local model I tried on my PC was Laguna XS.2, which had 70.8% accuracy
• Local models that did lots of thinking would fail to produce answers, because they’d blow through the context window I had set and then not produce the single letter answer.
• The best model I tried was Claude Sonnet 4.6 - which answered 85.3% of the 1000 questions correctly… and would have cost $71.92 to run the test if I paid at API rates!
• Haiku 4.5 was really good at 76.5% accuracy, and would have only cost $32.45 at API rates.
• DeepSeek V4 Flash had 74.2% accuracy, and cost only $0.50 to do the entire test set.
• Gemma 4 31B had 72.1% accuracy, cost $0.87 to do the test set, but improved the most between its non-RAG responses and its RAG responses (+29.4 percentage points) while still answering all of the questions.

A note about accuracy: these 1000 Star Trek trivia questions include questions which are not answerable from the episode transcripts. So, 100% accuracy is not actually possible for any model against this test set. Which is fine.

So… based on that, I initially selected DeepSeek V4 Flash as the model, and I rolled that out to production on Starbase Zebra as another Django app. I used Claude Code give the whole thing a facelift, and give it the design based on Spock’s science station that it has now. Also when shifting it to Starbase Zebra, I changed the vector database from Chroma to sqlite-vec, as I use SQLite as the database on Starbase Zebra.

However, after using it a bit, and also finishing the non-RAG evaluation runs (they take a long time to do), I realized that Gemma 4 31B might actually give better answers, and actually be better at retrieval even though its accuracy was lower. To test that, I had Claude create a quick script to let me blind A/B test DeepSeek and Gemma against each other. I found that in cases where I preferred one response over the other, I always preferred Gemma’s response, and that Gemma often retrieved details that DeepSeek didn’t. So I just recently switched it to use Gemma 4 31B for inference instead.

So yeah, that’s the whole thing. I’ll post the table here in a second.