As I’m building out STEAMCOACH, I am figuring out which models I should use to run my question-generation system—specifically, for F=ma practice problems.
To start, I decided to use open-weight models running on my local PC to save costs. I tried Qwen 3 14B, Phi-4-reasoning-plus, and GPT-OSS. It was a struggle to find models that were good enough for the task while also fitting within my PC’s VRAM and memory. I was doubtful of how these smaller models would compare with frontier lab models, but based on my past experiences with research in automatic question generation and scientific agents, Qwen and Phi perform just fine—close to the level of GPT-5.
I started running the system with Qwen, but found that the quality was not great and generation took forever. I figured the 14B model might be struggling to operate within my PC’s VRAM, even with FreeToken. I prepared Phi-4 in Ollama and, out of curiosity, asked it to generate a complex F=ma question while I closely monitored its reasoning traces.
I saw that it was hitting upon some interesting question ideas, but it always second-guessed itself. It never exited. With every new idea, it would deliberate and might even continue fleshing out the problem, but sooner or later it would always decide that the idea would not work.
That was why the open-weight models were not functioning satisfactorily. Even when instructed not to second-guess themselves and to output the first idea they had, they still second-guessed themselves every time.
I reluctantly made the switch to GPT-family models. Here, there was a plethora of options. I compared GPT-4, o4-mini, GPT-5-mini, and the fully fledged GPT-5. I found a large difference between the first two and the second two. Interestingly, though, for this use case there was not much difference at all between GPT-5 and its smaller version in the quality of the questions they produced.
I am still trying to figure out why this is, but for now, I’ll take the similar performance at a lower cost.