We have reverted an update we made a couple hours ago.
Some bad news.
A subset of reviews done between 12:10PM PDT to 01:20PM PDT did not get registered. Unfortunately, there is no way for us to recover these reviews. This is a something we are really sorry for and it was definitely something we didn’t anticipate. What this means is there are some reviews in the mentioned time period which will appear in your review queue again at their previous SRS state. Again, this applies to a subset of reviews done in the aforementioned time period.
At the technical level this is what happened.
Before the change, we used Memcached as a cache store. We also use Redis, mainly for background-job queueing. Today, we transitioned to the same Redis instance for cache storage. When the Redis instance exceeded its memory allotment, it started using the LRU algorithm to manage memory. That algorithm started dropping some queued background jobs, many of which were for post-review processing.
So the question is: what are we going to do to make sure this doesn’t happen again.
- We have reverted back to our previous caching set-up.
- We had specced out changes to make activity tracking more robust a while back and planned on tackling it later in the quarter. This goal has now been reprioritized and is now on the top of our list. With what we specced out, recovering reviews would have been trivial.
Again we are really sorry this has happened. If anyone has any further questions or concerns feel free to leave them in this thread or contact us at hello@wanikani.com.