Wanikani Open Framework [developer thread]

WaniKani API And Third-Party Apps
1

Wanikani Open Framework

Wanikani Open Framework (“wkof”) is a user-created framework for rapidly developing web browser userscripts for Wanikani.

Features

  • Simplifies interfacing to the [Wanikani API], allowing rapid development of data-rich addons for Wanikani.
  • Caches Wanikani API data to improve responsiveness.
  • Integrates into the Wanikani menu, allowing userscripts to easily add links for Settings or opening a script interface.
  • Provides a Settings Dialog API for creating stylish dialogs for editing your script’s settings, and functions to load and save those settings in browser storage. (See [Self-Study Quiz] for examples).
  • Allows rapid data queries from the Javascript console via the global wkof object.

Repository

[Github]

[Typescript types] by Kumirei

Published scripts/sites currently using the framework

Additional Filters (seanblue)
Advanced Context Sentence (abdullahalt)
Bulk Add Kanji User Synonyms (normful)
Burn Manager (rfindley)
ConfusionGuesser (Sinyaven)
Dashboard Leech Tables (Dani2)
Dashboard Progress Plus (rfindley)
Dashboard SRS and Leech Breakdown (seanblue)
Double-Check (rfindley)
Exact Review Time (Chrysus)
Expected Daily Reviews (Kumirei)
Fast Abridged Wrong/Multiple Answer (DaisukeJigen)
Heatmap (Kumirei)
JLPT, Joyo, Frequency filters (Kumirei)
Leaderboard (Dani2)
Leech List (ukebox)
Lesson Cap (valeth)
Lesson Examples Audio (seanblue)
Lesson Hover Details (seanblue)
Levels By SRS (Kumirei)
Level Duration (Kumirei)
More Reviews Text Replacer (RysingDragon)
Notify (DaisukeJigen)
Progress Percentages (Kumirei)
Radical & Kanji Mnemonic Tooltip (irrelephant)
Reorder Ultimate 2 (xMunch, updated by rfindley)
Self-Study Hide Info (rfindley)
Self-Study Quiz (rfindley)
SRS Distribution Charts (Kumirei)
SRS Grid Details (DaisukeJigen)
Ultimate Timeline (rfindley)
Upcoming Lessons (Chrysus)
Visually Similar Kanji Filter (Kumirei)
Vocab Beyond (normful)
WKStats Projections Page (UInt2048)

…and more!

55 Likes
2

A sample client script might look something like this:

// ==UserScript==
// @name        Wanikani Open Framework Sample Client
// @namespace   rfindley
// @description wkof_sampleclient
// @version     1.00
// @include     https://www.wanikani.com/*
// @require     https://www.greasyfork.org/<URL of released framework>.js
// @copyright   2017+, Robin Findley
// @license     MIT; http://opensource.org/licenses/MIT
// @run-at      document-end
// @grant       none
// ==/UserScript==

(function(global) {
    'use strict';

    // Modules that our script uses.
    var modules = 'apikey, item_data, user_data, settings';

    // Load the modules.
    wkof.include(modules);

    // When the settings module is ready, install our settings.
    wkof.ready('settings').then(install_settings);

    // When all modules are ready, run startup.
    wkof.ready(modules).then(startup);

    function install_settings() {
        wkof.Settings.install('My Sample Script', '/plugins/settings/my_sample_script', [
            {name:'time_fmt', label:'Time format', type:'dropdown', values:['24-hour','12-hour']},
            {name:'bkcolor', label:'Background color', type:'color_picker'},
            {name:'lessons_per_day', label:'Lessons per day', type:'integer', checker:lesson_range_checker}
        ]);
    }

    function startup() {
        // TODO: Your main script logic goes here
	}

})(window);
6 Likes
3

I like this idea a lot. The API you’ve sketched out here looks solid too. This plan generates a number of ideas for me:

1

TamperMonkey caches the @require’d script aggressively. Which is great because there’s very little delay to the script, but bad if that script functioning means it has to fetch more script code (to work around the cache-invalidation of the script or to dynamically choose which script code to fetch). I wanted two things which I couldn’t get out of an @require’d script: keep my scripts as fast as possible AND update the @require’d script as soon as it changes.

2

Assuming either one of those two constraints above don’t apply (or that you’ve another approach that meets them), then a common pattern I have in my script is:

  1. As soon as possible update the page based on the last-known state of the world.
  2. Fetch data to update the state of the world.
  3. Re-update the page based on the now-up-to-date state of the world.

3

I’ve implemented a smart cache of the WK API data. It’s faster to make a request to my site than to make even 3 parallel last-modified-at requests to WK API. I could quite easily put together an endpoints that served up a unified payload for subjects + assignments + review data + user + etc. Either complete data sets or deltas. Either data for all types, or for selected ones.

4

I’ve been thinking that centralizing state across browsers is pretty handy. E.g. for the app store I’m leaning towards reminding the user of scripts they have installed in other browsers that they don’t have installed in the one they are currently using. E.g. for leech training, I want to centralize keeping track of the state of training for each leech. I’m not sure if I’m proposing to offer some generalized store, I’m just saying that this is a need for the fancier end of the script spectrum.

5

Scripts authors miss a feedback mechanism. It would be great to know stats about how many installed (and more importantly uninstalls) a script has had. It would also be great to know what kinds of exceptions your script is throwing and in which browsers (e.g. I was missing a polyfill for older IE versions).

6

I can imagine it would be simple for this framework to make building a new page on top of a 404 page quite easy. There are a number of pieces to get right which I had to figure out for my app store script which are pretty much going to be the same for any similar effort.

7

The framework could present an API on top of the WK data. E.g. if I have an assignment in variable a the framework could make it so that a.subject or a.subject() returns the associated subject, etc.

I’m sure there’s more things that are interesting to consider. #1 & #2 are the most important to me as a script author. The other aspects are nice to have.

2 Likes
4

@hitechbunny,
Thanks for the thoughtful response! Some responses of my own:

One option is for the bootstrap to store scripts in indexeddb or localStorage. If they are present and up to date, load them from there, otherwise let the browser decide whether to use page cache or not.

Do you mean rollout of script changes?

One of my personal requirements, for security purposes, is to not load code from a place where it can be changed without me knowing. With a script manager, it prompts you to update. And dynamically loaded code in the script should point to a versioned URL that can’t be updated. If the script needs to change one of its dynamically-loaded components, that should be reflected in the top-level script by changing the versioned URL of the dynamic code.

I’m going to have state variables and events to let you know the status of the cache, so that should serve your purpose well.

Do you have a sense of whether it will it still be faster when you have thousands of users?
As an aside, the framework could accept 3rd-party modules, too, so individual script writers can choose their data source by including the corresponding module.

Yes, absolutely. A ‘sync’ module is something I’ve thought about. You could make it as simple as “everything under wkof.sync_data will be sync’d”. And if individual scripts want to sync separately, that’s something they can implement.

I agree. As long as any sort of data collection is opt-in and clearly spelled out.

I have a prototype that creates a ready-to-use page (on top of a 404 url), including breadcrumbs and a link to return you to your previous location. This works on the forums and the main WK site.
image

I figure the data API will be the most-discussed module. I know people are interested in adding supplemental data to WK items, and add their own external item lists (e.g. kana-only vocab, or joyo kanji not covered by WK, etc). I’m open to any structures that retain that flexibility. :slight_smile:

One of my biggest criteria is the ability to search for items by nearly any field, such as “kanji with same reading”, or “kanji with same radicals”, or “Level 1-10 items at Enlightened level that aren’t up for review for at least two weeks”.

2 Likes
5

I would love to join after my internship has ended, though it may be done by then.

6

When does your internship end?

7

Would it make sense to (also) allow the search criteria to be a function for more complex searches? Then you could search for all items in a list or all items where some calculated value from fields meets some criteria.

8

In June 2018, so it’ll be quite some time.

9

Indexeddb only lets you query by one field, but you’ll be able to use a cursor (database index pointer) which allows you to pull out records by additional fields. So yeah… I think that’s going to be possible. For simplicity, though, and since the record set is still actually pretty small as such things go, I may leave it to where you just put out the array by one criteria, then filter the records yourself afterward using the array.filter() function.

10

I guess it depends on how complicated you want the implementation to be, but I think there is a moderate benefit of taking more than one criteria.

The most flexible version I see is where the single parameter in the search method is an array, representing the list of queries to union together. Each entry in the array is an object where the key value pairs are the individual queries to intersect together. And one of the key options could be custom which takes a function (assuming that can work; otherwise having the caller do one last filter themselves may be reasonable).

So for example (pseudo code):

var searchCriteria = [
  {
    srs_level: 5,
    time_until_review_at_least: 5 days
  },
  {
    srs_level: 6,
    time_until_review_at_least: 10 days
  },
  {
    unlocked_less_than: 2 months ago
  }
]

var data = wkof.search(searchCriteria);

This search would get you the distinct items that are either (srs 5 and 5+ days until review) or (srs 6 and 10+ days until review) or (unlocked less than two months ago).

If you take the list in the framework’s method, you can have one place that does the union (and therefore removes duplicate entries) as well as the intersection.

For what it’s worth, my motivation is being able to find all items from a leech list such that each item is a certain time until next review, but where that time is different depending on the SRS level (hence the first two conditions in my example).

11

No worries, you’ll be able to do as fancy of criteria as you want. The question is whether to make the framework decide the best way to filter the data, or leave it to the script writer.

In your example, I think I would search by srs_level between 5 and 6 (inclusive). Then, on the returned data, you’d simply run a filter:

var data = wkof.search({
  keyname: 'srs_level',
  ge: 5, // Greater than or equal to
  le: 6 // Less than or equal to
});
// Apply the remaining criteria
data = data.filter(function(item){
  if ((item.unlocked_date > ago(2 months)) ||
      (item.srs_level===5 && item.available_date < ago(5 days)) ||
      (item.srs_level===6 && item.available_date < ago(10 days)))
    return true;
  else
    return false;
});
12

Not sure how you’ve set up your idb for use with IndexedDB, but I’ve had good experiences using localForage before.

13

It’s in times like these that I wish I had bothered with JS :sweat:.

1 Like
14

I looked at localForage and considered using it, but ultimately decided to make use of the indexing features of indexeddb. I’ve simplified its access significantly with a pretty small codebase. Using it looks something like this:

var db = new Idb('wkdata',[
  radicals: 'slug,character,level',
  kanji: 'slug,reading,meaning,level'
  vocabulary: 'slug,reading,meaning,level'
]);
db.put('radicals',[
  {slug: 'stick', character: .....},
  {slug: 'leaf', character: .....}
]);
db.get('radicals', {keyname: 'level', eq: 5})
.then(function(data){
  // Do something with the data
});
15

It’s not quite the same since you can’t do an OR with different criteria in the initial search. So your version only gives unlocked_date > ago(2 months) && srs_level >= 5 && srs_level <= 6, whereas mine would do unlocked_date > ago(2 months) without those other restrictions.

Of course, my original example can be achieved with your sample code by doing multiple requests to wkof.search for the OR’ed sub-queries and doing a union.

Either way, I can understand why one of your goals would be to keep the framework nice and light. :slight_smile:

16

For the settings I’m not sure what you saw, currently I’m adding buttons into the section headings, and store the settings locally. I inject the bootstrap.js button and modal styles so that it looks like on the WK content pages, however. For example @DaisukeJigen Fast Abridged script adds a settings menu to the WK menu itself.

Still, I was working on a few things that might be interesting for other scripts.

  • Something called WKInteraction, it watches the page and generate events when a review was answered, new question loaded, reviews finished (leaving the page), the “more details page” in lessons and reviews loaded, etc. It can be done nicer by looking at WK’s javascript code itself, but it does the job.
  • I added parts of bootstrap.js and the wanikani styles like character-grid to lessons and review pages

I’m currently working on a review timing script now and reuse some resources, I separated the styles and events by adding the userscript namespace to the identifiers, but they are loaded several times anyways. I didn’t want to bother with finding out how to make scripts dependent on each other in Tampermonkey.

The API stuff from @jeshuamorrissey also looked quite nifty, but I’m not using the API at the moment, I added my own static JSON DBs.

17

Ahh, yeah, I was thinking of @DaisukeJigen’s scripts. I was looking at your Semantic/Phonetic code at the same time, so I guess I mixed them up.

The WKInteraction thing sounds interesting and useful. There are several scripts that interact on that page. Maybe having a common module could prevent some of the interference between scripts.

18

I did make a ‘settings’ js file, utilizing JQuery UI, that puts settings under the WK menu. It’s an include script on greasemonkey. Does the job, but could definitely use some work. Little tidying up, add some more functionality, etc.
But of course, the source is free for the taking.

19

Yes, there are several scripts messing around with MutationObservers and listening to jStorage, and it’s hard to find out which elements to watch, and sometimes you need want a combined event … It’s better to have high level events and just worry about handling them.

20

Ahh, yeah, I overlooked the fact that you weren’t limiting SRS level on the unlocked_date.

So yes, it would require either multiple searches, or just return all records and filter. That’s one of the general limitations of the browser’s indexeddb: it accepts only one search criterion, with either a single value or an upper and/or lower bound.