Why I Built PocketGround

PocketGround tests whether edge models, including quantised versions, work well in RAG systems.

I wanted a straightforward way to compare the small models I could run locally. I was choosing one for document Q&A in a Meta Quest app, and being able to load a model was only the first step. I still needed to know how well it would use the documents I gave it. That question became PocketGround.

The pipeline is simple. The app retrieves five passages from a document and hands them to the model along with the user's question. That's retrieval-augmented generation, or RAG: find useful text, then ask a model to answer from it. There's no tool calling or agent loop in this setup.

The job sounded narrow, but it had more parts than I expected. The model had to answer from the supplied passages instead of improvising. It had to turn down questions that had nothing to do with the document. And it had to behave sensibly when the evidence was missing, when two passages disagreed, or when someone had slipped an instruction into the text.

People have already studied small models in RAG. The SLM RAG Arena lets you compare small-model answers side by side, and MiniRAG looks at on-device retrieval. Their datasets, prompts and grading rules weren't mine, and I needed to know how particular quantised files would behave with my kind of prompt, my five passages and my awkward questions. So I started writing my own test, and it grew into PocketGround.

Quantisation matters here because it compresses the model weights, making the file smaller while potentially changing answer quality. Those compressed files are the versions I would actually deploy, so I wanted them in the comparison too. In the results, B means billions of parameters. Q4_K_M and Q8_0 name the quantisation formats used for the weights; the Q4 version uses roughly 4-bit weights and produces a smaller file than the Q8 version.

Keeping the retrieval fixed

The most important decision was to freeze retrieval. Every model sees exactly the same question and exactly the same five passages, so a better result can't come from a luckier search.

That isolates the generation stage, but it doesn't isolate the weights. What I'm really comparing is a configuration: the model file, its native chat template, the runtime build and the settings I ran it with. Change the template or the runtime and you may get a different answer from the same weights.

It also means PocketGround doesn't tell you whether your retriever is any good or how a full end-to-end pipeline will perform.

The first dataset is built from the U.S. Fire Administration's Fire Safety for Older Adults, a public, practical 24-page booklet. Fire safety is one case study. The benchmark is about small models in RAG, and it needs more domains before it can say anything broader.

What the questions look like

There are 40 questions. The 32 core cases fall into three groups:

• 20 document questions, where the answer is in the booklet. How often should smoke alarms be tested? How far should a candle be kept from anything that can burn?
• 8 off-topic requests the model should refuse. Some are obvious, like a cake recipe or Python code. Others use the word "fire" to mislead, like how to fire an employee or how to disable Windows Firewall.
• 4 handling cases: a price the booklet doesn't give, a question missing the details needed to answer it, two passages that conflict, and an instruction injected into the context.

Eight more questions are about fire safety but aren't covered by the booklet, like what PASS stands for on an extinguisher. The prompt allows reliable general fire knowledge here, and I track separately whether the answer says it came from outside the passages. These sit in a public supplement outside the core score. I made that 32/8 split after collecting the results. The prompt, the answers and the judgements didn't change, and every raw output is still there.

What the first release covers

The first release includes seven models across three families:

• Qwen: Qwen3.5-0.8B and Qwen3.5-2B.
• LFM: LFM2-1.2B-RAG, LFM2.5-1.2B-Instruct, LFM3-1.7B and LFM3-Omni-2B.
• MiniCPM: MiniCPM5-1B.

All seven were tested in Q4_K_M format. Qwen3.5-0.8B was also tested in Q8_0, giving eight weight files. Its Q4 file was tested again on the newer runtime used for LFM3, bringing the total to nine configurations. Omni was tested with text input and output only.

Each configuration answered the same 40 questions twice, for 720 recorded responses. That's still 40 distinct questions. The tests ran on 27 and 28 October 2026, on an Apple M3 Pro laptop with 18 GB of memory. Headset speed, memory use and battery consumption haven't been measured.

All recorded configurations, Run 1. Run 2 produced the same core scores. The Qwen3.5-0.8B Q4 file appears in both test groups because it was run with two different runtime builds.

What you can learn from the results

The core totals range from 10 to 29 out of 32. No tested configuration passed every case. But if you're choosing a model for an app, you'll want to look past that total and ask which requirements it missed.

That's why PocketGround keeps document answers, off-topic refusals and evidence handling separate. You can compare each model on the behaviour your application needs, then open an answer and see exactly why it passed or failed. Parameter count, quantisation, file size and answer time are available alongside those results. Timing comparisons stay within each test group because the runtime builds differ.

Quantised versions get the same treatment. You can compare the tested Q4 and Q8 files against the same questions, without assuming in advance how much the extra precision will help. These results apply to those files and settings; they don't establish a rule for every quantised model.

The failures are part of the report too. Some answers reached the output limit, some missed the requested facts, and some failed to refuse an unrelated question. Thinking was switched off throughout, but explicit thinking markers still appeared in some outputs. The exact responses stay visible, including unfinished answers. I want you to be able to inspect the behaviour behind a score, not just accept my judgement of it.

What it doesn't prove yet

I reviewed every distinct answer by hand. When a repeat run produced exactly the same text, I verified the match and reused the judgement instead of reviewing it again. There's no independent expert review or agreement check yet.

The 40 questions were built around one source document, chosen on purpose and are related to each other. So I wouldn't use these counts to claim one model is statistically better than another, or that any of them is reliable in general. PocketGround is an exploratory benchmark, and I'd rather it stay that way in how it's described.

Try it, and add to it

The part I find most useful is reading the answers. The interactive report shows each model's actual reply next to the passages it was given, and that tells you far more about how a model fails than a table does.

If you're choosing a small model for your own RAG setup, run the questions against it. The repository needs only plain Python and works with a compatible local chat-completions server. Nothing is graded automatically. You review the answers yourself, and the methodology explains how I judged mine.

If you have a document from another domain, a model I haven't tried, or a judgement of mine you disagree with, open an issue or send it over.