We’re changing what’s stored in jStorage for reviews. This work is still in progress but wanted to give time to process and try out the change. The details of the implementation are still being worked out. I’ll add more to this thread later, along with a link to preview.
jStorage can’t handle all the review items for some users with a lot of reviews. For these users, reviews don’t even load. As we put more data in a single item (ex. adding audio and pronunciations, types), items get bigger, and the number of items that jStorage can handle decreases.
In addition to fixing bugs, making this change should make reviews load faster for everyone. (The bigger your review queue the more noticeable the improvement will be).
Here’s how the new approach works:
- Change the reviewQueue to only store IDs instead of the entire review item
- Make an extra request to get item data when we need it. This happens when populating the activeQueue.
Scripts shouldn’t rely on reviewQueue having all the items for review. They should request item data instead. The item data could come from the WaniKani API which is versioned and stable, or our internal API.
You can now try this change out on https://preview.wanikani.com
Some more details on how it works:
On load
- Fetch from /review/queue?minimal=true this returns an array of subject IDs, store these in reviewQueue. Note that script compatibility mode will continue to use /review/queue without the minimal parameter.
- Pop 10 IDs from the review queue.
- Fetch the subject data for those 10 IDs (/review/items?ids=1,2,3,4,5,6,7,8,9,10)
- Add the subjects to the activeQueue
When a review is completed
- Pop an ID from the end of the reviewQueue array
- Fetch the subject data for that ID
- Add the subject data to the reviewQueue
11 Likes
Just asking because of [UserScript] Integrated Custom SRS, will the change be for /review/queue or for the localstorage review queue alone? Also, will the lesson queue get the same treatment? That can be long as well if someone skips vocab lessons. And finally, will the review data be altered?
1 Like
So no more craby-sama on the rotating background? n-n
1 Like
@Gorbit99 /review/queue will continue to return the same payload, /review/queue?minimal=true will return IDs only. In both cases, the results will be stored in jStorage under reviewQueue.
We won’t be touching the lesson queue in this round of updates, but the plan is to consolidate and reuse code when we can, so it will likely happen in the future.
Review data won’t be altered in this update, if you’re referring to the structure and contents of an individual item like you see in currentItem.
@daikirai not quite yet but maybe we’ll get to that point…
2 Likes
Is there a variable we can use to determine if Script Compatibility Mode is enabled?
3 Likes
I wanted to answer with WaniKani.wanikani_compatibility_mode, but then I noticed that this is only defined on the lesson page.
1 Like
@Sinyaven @rfindley both lessons and reviews now have WaniKani.wanikani_compatibility_mode
3 Likes
Plan to push this change out to production tomorrow.
2 Likes
I’ve put a [notice] in the Reorder Ultimate 2 forum thread, advising people to switch to Compatibility Mode if they want to continue using the script.
Javi, I’m really appreciative of the advance notice you and the team have been giving us scripters. I wasn’t able to take advantage of the heads-up this time due to wrapping up a project, but my hope is that, while I have some spare time during the holidays and transition between projects, I can put together a proper forward-looking replacement for the Reorder script, so users who rely heavily on the current script won’t be stuck in Compatibility Mode for the long haul. I definitely want to help keep the forward momentum on Wanikani going with as few bumps in the users’ road as possible 
15 Likes
Edit: You can ignore this post, question was answered by another forum member.
If I complete a review, does this mean that continuing on to the next item blocks until the activeQueue is updated. I.E., going on to a new review after completing each review now always incurs the latency of one round trip network request? (Previously, there was no such latency because the only network request after completing a review was submitting the review, which did not block continuing on to the next item.)
A couple of different possible solutions:
- Preload the next 1-2 items from the reviewQueue before they enter the activeQueue. Store the preloaded data privately to avoid any further changes to the jStorage api.
- Or, only block on the network request when the item that hasn’t loaded yet becomes the currentItem and make a code change so that any new item entering the activeQueue must wait at least 1-2 reviews before becoming the currentItem.
Are you seeing this in practice or just speculating? There are always 10 items in the active queue (unless you have fewer than 10 items left to review), but if one of those 10 items is completed after reviewing correctly, a new 10th item would have to be added to the active queue. I can imagine the situation you just described happening if they didn’t change anything to account for the fact that the new 10th item doesn’t already have it’s data loaded.
In addition to your proposals, another option would be to select the next review only from the remaining 9 items in the active queue while the data for the new 10th item is being retrieved from the server.
1 Like
Speculating, based on what I imagine the most straightforward way of implementing it might be, since WK has historically avoided adding complexity to the codebase to reduce latency.
I think that might be what proposal #2 was saying? (Note that there probably still should be a fallback to blocking if the user somehow completes reviews fast enough to empty the activeQueue before the new item finishes loading.)
1 Like
Ah, I see. Yeah, they are pretty similar. They are slightly different implementations to achieve the same outcome.
1 Like
I thought it was clear from Javi’s description, minus one omitted detail, which I guess is what you are asking about:
- activeQueue is initially populated with the first 10 items from the reviewQueue.
- Upon completing an item (both reading and meaning):
a. Pop an item ID from reviewQueue
b. Initiate an asynchronous fetch for that item’s details.
c. Render the next item in activeQueue (not the item from the pending fetch)
d. When the fetch completes, append the item details to the activeQueue.
(b) and (c) above would be interchangeable.
1 Like
What you described is certainly how it should work to avoid the issue @est_fills_cando speculated about. But the OP just says
Make an extra request to get item data when we need it. This happens when populating the activeQueue.
And
When a review is completed
- Pop an ID from the end of the reviewQueue array
- Fetch the subject data for that ID
- Add the subject data to the reviewQueue
Which I still find ambiguous. (Also should that last one say to add the data to the activeQueue?) It doesn’t really make clear if rendering the next review item to the user happens before or after the activeQueue gets the new item appended. That’s an important detail to know if it’s asynchronous or synchronous from the user’s perspective.
1 Like
What happens if the activeQueue is empty at the time it wants to render but there is a request pending? Then it would need to block on the network request. So assuming they do block in that case, my thinking is they might block in every case for simplicity. Of course, I would be very happy for my speculation to be wrong.
Normally, I would just look at the code myself to check, but I don’t have an appropriate development environment to do that in right now.
I think that means your review session is done. I assume that for each item you complete, it will pull in the next available item from reviewQueue to activeQueue.
The scenario I am thinking of is:
- There are 11 items left to review.
- User completes an item.
- Due to a combination of unusual server lag and the user being quick, they complete 9 additional items before the server request for the new activeQueue item completes.
In this case, the activeQueue will be empty, the review session will not be over, and it needs to block the rendering until the server request happens. But if they decided not to worry about this case, then I suspect they would have implemented it as you indicated.
@est_fills_cando @seanblue,
Here’s what it does:
- It fetches the details for 100 items – the last 100 items in reviewQueue – and stores it in
memo[].
- As each item is completed, that item’s data is removed from
memo[].
- After every completed item, it checks how many empty slots there are in activeQueue. It attempts to fill these empty slots from
memo[] based on the last N itemIDs in reviewQueue.
- When
memo.length drops below 20% capacity, it restocks the full 100 asynchronously.
So, the only wait should be upon initial load while it loads the 100 items’ details. All other item details are fetched asynchronously while you are working your way through those 100 items. activeQueue only holds the last 10 items from reviewQueue, though.
4 Likes