I wrote about the nature of the bug here: [Android] Flaming Durtles - Android app with offline support - #797 by ejplugge
How I fixed it is with the debug log. Flaming Durtles continuously writes a debug log of what it’s doing, and via an option in the settings menu, a user can upload their debug log to me.
When this problem was first reported, I was very confused and annoyed by it. So I added some more information to the debug log related to the choosing of questions and the judging of answers. I pushed a new update to the Play Store with that logging and waited for someone to report the problem was still happening, so I could then ask them to upload their debug log to me.
This happened yesterday morning, when a user with the problem contacted me by email. She uploaded her log, and it showed exactly the sequence of events leading to the bug. Comparing the logging to the code, I managed to reconstruct the sequence of events and I figured out the cause of the problem.
I still couldn’t reproduce it myself, but since I knew what was causing it I could just remove the cause, and that’s what I did.