AQG using retrieval augmented generation is a newly established but already extensive field, with many proposed improvements to LLM question-answer creation abilities. However, one area where LLMs are still lacking is in the realistic creation of complex olympiad problems, even with methods such as RAG. I propose that this is because LLMs merely contextualize and reason about surface-level question text when trying to generate problems. This might be fine for shallow problems, such as fact-based questions, but it is severely inadequate for more complicated methods.
I propose that looking at solutions, storing them as skeletons with pre-set operators, and embedding those solution skeletons would create much higher quality questions. I am also investigating whether this method on lower-quality open-source and open-weights models can compare with the base ability of proprietary high-quality models, which would improve equality. For now, I am considering AI-as-a-judge techniques for evaluation.