Yeah this is the part I agree was handled really shoddily, they shouldn’t have let people keep using the website, especially cause that obviously exacerbates any existing performance issues.
Well I typed that before I saw his reply, regardless, I agree with Chelsea that they should have handled it better.
Out of curiosity, why are the reviews processed asynchronously at all? It’s surely not all that resource intensive of a process to do DB updates, and there’s no calculation necessary beyond authentication and sanity checks, right? The client decides if the answer is correct and all the server has to do is store the updated review data. Doesn’t that also contribute to the problem of the review page lacking the last item done?
Are you guys using some kind of an event-driven model, where some enrichment happens as part of a pipeline, or is this simple CRUD? Is there a database that stores all of the review events, or simply an entry in a table per user, per item? Super curious about the stack.
1 Like
Sorry about review and lesson queue issues this morning.
I am not going to make any excuses. It is -8 GMT here and we just literally woke up to this problem; myself the last half hour. When I saw it we immediately went to work. And I was able to identify the issues within minutes and put in the additional resources to address it. We unfortunately don’t have a 24 hour staff to monitor the site, but we do get to work to fix major issues immediately when notified. This review/lesson queue issue was a rare one-off. Lessons are still learned from this and we’ll look into autoscaling solutions.
14 Likes
No worries! Thanks for being so communicative. These things happen. <3
2 Likes
Thanks for getting it handled so quickly. We understand these things happen.
3 Likes
Thank you, Viet, for your hard work!
It’s easy to just complain and badmouth the site without actually understanding what’s going on. Truth is, WK is generally very stable, in the 6 months I’ve been using the site this is the first “major” incident, and was promptly solved. These things happen.
Thanks for the update, thought you’d initially been looking into it yesterday as there were some discussions about this earlier but it seems that got lost in between talk about the self-resurrecting-burns issue. Hope it’s all stable now and you guys can go back to business as usual.
It was a solution to an old problem which currently is not a problem anymore.
We have been working towards revamping the system since the beginning the year to better accomodate the operations and problems WK faces today. Yesterday’s update was a major one since we switched reads over to a new model (Improved response times, especially for API, some bugs appeared, but were addressed shortly after). Eventually we will get rid of the async. But one step at a time 
4 Likes
Thanks for fixing it swiftly! 
I forgot about the time difference, although it’s in the afternoon here when it happened to me, it was still morning there where you are haha.
お疲れ様です。
I am unfortunately still seeing things come back up that I’ve just reviewed. Not sure what could be causing the issue on my end if anything not sure if @rfindley time line script is acting up on my end.
Are they actually showing up during your review session or is the review number on the indicator greater than zero?
I just checked your account and I am seeing an empty review queue. Is this the case now?
Sorry @viet I got busy and didn’t come back. At this moment, I’ve got 9, which i just did (its 1513) the review number on the dash shows zero but, at the same time its saying available now.
If you refresh the page does the status reflect correctly?
No it does not, i’ve even logged in and out, no luck
And as of now, i have reviews to do, so i’m not sure which is what. (the review counter is showing 5 )
ok just did that 5, it was ones I’ve done, it then immediately loaded up 2 more then 1 and yet another single. all 9 of which I’ve done in the last… hour ish time frame. now that those have cleared out , i’m showing 25 minutes to my next review, 10 lessons and no pending review.
Ahh okay. If it happens once more, please let us know. We were zoning in with some settings for the new autoscaler we have implemented just within the last hour and the background worker queue was backed up a little. I think thats what was causing the issue. It shouldn’t happen anymore.
and unfortunately on my next review session it has done it again, 6 items, I litterally just did. the review counter is zero but its showing available now, and the timeline script is showing them, as the exact ones i just did. i restarted the browser to double check. Not sure what else to do @viet
Just to make sure I understand whats going on…
- On the navigation bar, it is showing 0 for reviews.
- On the dashboard, it shows Available Now (not on the time line script, but the status which we render to you).
- You click on the review link, start the session, and the six items you just reviewed are showing up in the quizzing session.
- Those six items are still in the queue. You have not answered them?