Funny, I forgot this is how Heatmap is by default, I’ve modified the local install to do exactly this already haha. Of note, in addition to the variables I suggest also having fallbacks (the default wk colors’ hex values for example).
2 Likes
I might switch over eventually (there were no variables when I wrote the script), but for now there is already a theme chooser in the settings
2 Likes
I was using the heatmap a lot this year but when I logged onto Wanikani today, it won’t show any of my previous progress (for reviews at least). It reset everything. Surprisingly my lessons heatmap never had this problem as I can see my progress all the way back to 2018 which is when I started.
Is there any way I can get my previous progress back?
1 Like
Since WK disabled the API the Heatmap is only able to gather review data on a per browser basis. If you switch device or browser, or clear your browser cache, the data is gone forever (well, until WK reinstates the reviews API). Sadly there is no way to get the data back if you don’t still have it on a different device or browser
2 Likes
Here’s a bit of a PSA to any using this.
Please regularly back up your review cache using the steps given by Kumirei in this post (click on it to see it formatted):
Specifically this method, because it has finally happened to me where my cache is gone and I thought recovering it with my own method of saving the indexeddb files and just replacing them in my browser profile location would make it so the browser rebuilds that db. It does not. In fact with firefox it kind of freaks out and says there isn’t a database at all until you restore what you were trying to replace (and even then it seems like you need to restart the browser to maybe see it restored in the console).
As for why my cache is gone? I think Open Framework does cleanup of old files because it’s not only my review cache gone, there are some other entries that were used by other scripts that were deleted. I know Open Framework is doing it because when I attempted to restore the old idb files, Open Framework immediately logged messages to the console saying it was deleting about 10 entries because they’re old. Now, I don’t know why some entries seem to be exempt from this (that aren’t settings, all settings appear to be exempt), but that’s to investigate for another time.
I’m not at a complete loss though. The file that the idb was using is still there (in my backup, not the live one), and with a hex editor I can clearly tell which one was storing the review cache. The trouble is, the files are first compressed using snappy (and unfortunately firefox forgets to perform the bitwise complement on the computed crc32c before handing it to the masking function, so any tools out there for decompressing framed snappy files will treat the firefox ones as corrupt due to checksum mismatch; I needed to figure out how to undo the masking mistake and compute the correct checksum value, grab the source code of snzip and learn how to both follow and write C to create a fictitious “mozilla-format” for it then compile and build that into my own .exe. After that I could decompress the file. And sadly that brought me to the next roadblock: the uncompressed file is still encoded, and I can’t tell how. Viewing it in the hex editor gives a couple minor insights, like that it’s probably utf-16 instead of 8, but other than that my only option at this point is inspecting firefox’s source code to actually look at what they’re doing to encode it. And all of this is only possible because of open source; if you’re coming from a closed source browser and run into issues like this I genuinely don’t know if you can recover anything at all.
So yeah. While the API is down, back it up. Frequently. Possibly routinely when you finish your last reviews for the day. I’m probably gonna write a small script that will simplify the export (and maybe also do import). Shouldn’t be too hard, there’s already Settings Exporter/Importer and I made my own script that exports transitivity pairs up to my current level. Will share here if I get around to that.
4 Likes
SUCCESS!! I finally found someone out there who made something for working with firefox’s sqlite databases and the files used by them for idb where they successfully ported the decoding from the source code in C++. Turns out that encoding is completely unique to firefox and isn’t documented anywhere outside of that single file in the source code (and documented is a strong word here). I was going crazy looking at the uncompressed data in a hex editor trying to reverse engineer what was going on (I had only just figured out how the length of the names of keys was being encoded but that was it). The python module on gitlab is here.
I grabbed a zip of that source and (after opening it up and fixing any complaints my editor had, including a typo in a file where value was used instead of val) then figured out how to run it in the interpreter rather than installing it with pipx so I could decode just the single file I have and it worked! I have my entire review cache in plain text.
What’s left now is for me to write that exporter/importer, because my cache array is over 36000 entries long and I do not think firefox would particularly like it if I tried to paste that into the console. Question is, how do I go about writing it? I’d like your input here @Kumirei if that’s alright.
Right now I’m thinking there are 3 options. (1) Write my own separate script for the import/export functionality that is pretty much just a modified copy of Settings Im-/Exporter; (2) Modify Settings Im-/Exporter to also be able to do the review cache; (3) modify and submit a PR for the Review Cache script to have its own interface for im-/export of files.
Which approach is best here?
I, Lupo, am a huge dunderhead.
I just discovered that on my HDD I use for things I don’t frequently access, in my Manual Backups folder, on February 19th this year, I exported and saved my review cache array in a text file. The me of 4 months ago anticipated this whole situation, and prepared for it. But they did not prepare for me to completely forget about it and not look for it. Valuable things have been learned in this journey.
Oh also, I apparently exported it in a way that it was the compressed data, because I didn’t know there was any compression going on. So I just spent an hour trying to fix the import lol. But here it is! My wonderfully ADHD wanikani journey:
2 Likes
I haven’t had any issues there, actually.
I think any of these options are fine, but #2 and #3 would be nice since anyone could use it. I would lean towards #3 since it would be a bit out of the scope for the Settings importer/exporter. Although I would accept a PR for either.
2 Likes
I need a better explanation of how to do the backup/import method described. As someone that has never used the browser console before (and isn’t a programmer), it feels like a looooot of steps are not mentioned.
How do I reach the review cache in the console?
Where am I supposed to copy/paste the line to make it into a string?
In the new device, how do I insert that line into the review cache?
Is there an easier way to do this? 
Edit: I did figure it out after some googling, and now I won’t lose data while I’m on a trip and only have my laptop for a while. Thank you!
1 Like
Heatmap disappeared for somebody else but me? 
I can see it in Tapermonkey, but not in WK’s UI.
Did you recently disable and then re-enable it? Or did you recently clear your browser cache?
As I understand, in either of these cases the data which is used by the script will be deleted and if there is no data to display the heatmap will not appear in the UI. Once you do some reviews there will be data to display. However, there will be no historical data, just data starting from today.
Nope, I know that we lost API to get all reviews history and now it’s based purely on local data. So my data is still should be here, but just not displayed. I even can access it’s settings menu from WK interface, so just curious if there any base-scripts updates or new dependencies/conflicts/etc.
I saw that they changed the SRS counts stuff, so you might need to move it to a new position until I can get it fixed? You should still be able to access the settings from the regular menu
1 Like
Yup! I use the Cockpit dashboard script. With the Cockpit enabled, the Heatmap disappears. Disable the Cockpit and the Heatmap comes back. Do you use the Cockpit too?
2 Likes
Heatmap still works for me with the SRS Progress section changes. All position options work just fine, so there shouldn’t be anything for you to fix in that regard.
2 Likes
Ah, changing position worked fine for me, thank you!
Correct, I’m using it with Cockpit. But in my case I just moved it to another position as below and now can see it again.
1 Like
Thanks! I’ll try that too.
1 Like
@Kumirei As a dirty hack, how much can you stash in the ‘meaning_note’, ‘reading_note’, and ‘meaning_synonyms’ of an arbitrary item? Like stick the date someone last did reviews and lessons in meaning note and reading note (respectively) of say, 復習, then look it back up to increment the study streak if it was yesterday. Or make all your data synonyms of 情報.
Might not be enough to save all the heatmap data, bit if you’re just going for streak information that multiple browsers can access… ? Seems like meaning_synonyms is an array of arbitrary text the server will save for you. Save data in some text encoding you can parse back out…?
Hm turns out there’s a 64 character max length on a synonym. I wonder if there’s a max number of synonyms per item? Might be able to add a synonym for every key:value pair.
2 Likes
I guess that still wouldn’t detect if someone did reviews on mobile or a browser without the script.
3 Likes
Honestly not an impossible idea - 1000 characters per item (meaning/reading_note) is more than enough to record previous reviews of each, and the Wanikani API allows fetching all user notes or even just user notes updated after a certain date-time, which would provide a simple way to sync from other devices.
As you say it won’t work with mobile / devices without the script but it could still be better than having no sync at all?
It would however likely be quite a lot of work to get set up and working!
2 Likes