Wanikani Open Framework [developer thread]

[seanblue]

But, of course, all of the code is working together for a single purpose.

I guess that’s the key question here: Are all filters working for a single purpose, or are all filters working for a separate purpose?

Here are some examples:

• Recent Lessons: My recent lessons filter uses Date.now() to check if the lesson was taken in the last X hours. Apparently this one I didn’t pre-calculate once because I wrote this filter before the prepare method was added.
• Time Until Review: I’m precalculating Date.now() once for this one. I’m using it to determine the hours until the next review.
• Failed Last Review: I will probably use Date.now() similarly to the “Time Until Review” filter, though I haven’t written it yet so I can’t be 100% sure.

I doubt there will be a significant impact if I use a different Date.now() for each of the three filters. I just wanted to raise this case to you so you could consider it and decide whether or now it makes sense to precalculate it once.

[rfindley]

@seanblue,
Per the [Api V2] thread, I’ve updated the Open Framework to support /srs_stages, plus opened up default support for any future endpoints so I won’t have to explicitly add them (unless they don’t conform to the current format for some reason).

The recommended way to fetch /srs_stages is via:

wkof.Apiv2.get_endpoint('srs_stages')
.then(do_something);

Note that, even though it caches the endpoint, it will still ask WK if the endpoint has been updated. So, if you want to limit how often it checks for an update, you can look at the date it was added to cache:

if (wkof.file_cache.dir['Apiv2.srs_stages']) {
    var fetch_date = new Date(wkof.file_cache.dir['Apiv2.srs_stages'].added);
    var days_since_fetch = Date.now() - fetch_date;
}

[seanblue]

I’m not familiar with caching requests in general, let alone how WaniKani’s API or your framework does it. Do you recommend bothering with that extra check on the local cache, or is it okay for the request to check with WaniKani each time? I honestly don’t know how much work would happen on WaniKani’s side if I make the request (with cache=true) each time.

---

EDIT: Also, is get_endpoint('srs_stages', true) the right way to get the cached results? Looking at the source code, I don’t see how passing in just true will work. Please let me know if I’m missing something.

[rfindley]

Looking at the source code, I don’t see how passing in just true will work. Please let me know if I’m missing something.

Lol… no, you’re right, and I’ve edited the code above. I was thinking of the wkof.load_file() function, which accepts a parameter for caching. The get_endpoint() function always caches.

Anyway, asking WK for updates on the /srs_stages endpoint should theoretically be a light load. It just seems wasteful to check at every page-load when changes to the SRS intervals are so rare, and I was also thinking of oldbonsai’s comment, “I’d cache that data real good. We don’t see changing the timing on those any time soon.”

I was thinking that limiting it to maybe once a day would be sufficient, but that may just be premature optimization.

[rfindley]

I’ve added [documentation] for the Settings module to github.

[Kumirei]

A new script to add to the list

[Userscript] Level Duration 2.0

I have found that the original Level Duration script is not entirely accurate, and is rather bulky. Normally I would just go into the script itself and modify it to my wishes, but it’s a bit of a mess; so I decided to make a new one. This script utilises WKOF to fetch the current level’s items and find the earliest unlocked item instead of just using the first time the script saw you at the level. Thanks to this the script is now more accurate and much smaller. I didn’t add any display options…

[rfindley]

Kanji progression

Also since we don’t have PMs, TamperMonkey is telling me line 79 and 390 in the Open Framework are missing semicolons.

Thanks, got it. I’ll put the changes into github, but I’ll wait for the next release to push to greasyfork.

[BobaGakusei]

It might already be answered, but are there any risks associated with installing the Open Framework?

I’ve concluded that I don’t know enough about the Open Framework’s implementation. This has caused me (perhaps unreasonably) to avoid installing it or any of the associated scripts.

Is all of the information stored in the browser (e.g., in indexed db), or is it sent off to a separate server?

What are the chances that @rfindley goes rogue? Or more realistically, would all these scripts be completely dependent on @rfindley to keep the Open Framework up-to-date, and what happens if @rfindley is no longer available to support it?

[rfindley]

@BobaGakusei,
The only ‘go rogue’ risk from the Framework itself is if you leave your TamperMonkey (or whatever) set to update automatically.

I intentionally set up the Framework to be traceable, and unchangeable by anyone (even me) without the user consenting to a version update (i.e the update setting in TamperMonkey/etc).

All of the external URLs inside the framework are contained in the core JS file and the CSS file:

* Core: 'https://greasyfork.org/en/scripts/38582-wanikani-open-framework/<specific_version>'
* CSS: 'https://raw.githubusercontent.com/rfindley/wanikani-open-framework/0c414bb4bc8ecaee35d4ee7463eadc5816d69504/jqui-wkmain.css'

Neither of those files can be modified after they’re published, so if you’ve pre-screened them, you’re safe until the next time you update. (As safe as trusting TamperMonkey/etc can be, since you’re trusting the folks that wrote that).

The content that those links point to are unchangeable because:

• The GreasyFork.org URLs point to specific versions, and they don’t allow modifying files after they’re published (you have to publish a new version)
• The github links are for specific commits, so again they can’t be changed once they’re committed.
• The googleapis.com links are solely controlled by Google.

The full list of external links is:

(In the core file:)
	var supported_modules = {
		Apiv2:    { url: 'https://greasyfork.org/scripts/38581-wanikani-open-framework-apiv2-module/code/Wanikani%20Open%20Framework%20-%20Apiv2%20module.js?version=266930'},
		ItemData: { url: 'https://greasyfork.org/scripts/38580-wanikani-open-framework-itemdata-module/code/Wanikani%20Open%20Framework%20-%20ItemData%20module.js?version=266242'},
		Menu:     { url: 'https://greasyfork.org/scripts/38578-wanikani-open-framework-menu-module/code/Wanikani%20Open%20Framework%20-%20Menu%20module.js?version=260444'},
		Progress: { url: 'https://greasyfork.org/scripts/38577-wanikani-open-framework-progress-module/code/Wanikani%20Open%20Framework%20-%20Progress%20module.js?version=262841'},
		Settings: { url: 'https://greasyfork.org/scripts/38576-wanikani-open-framework-settings-module/code/Wanikani%20Open%20Framework%20-%20Settings%20module.js?version=267483'},
	};

	published_interface.support_files = {
		'jquery_ui.js': 'https://ajax.googleapis.com/ajax/libs/jqueryui/1.12.1/jquery-ui.min.js',
		'jqui_wkmain.css': 'https://raw.githubusercontent.com/rfindley/wanikani-open-framework/0c414bb4bc8ecaee35d4ee7463eadc5816d69504/jqui-wkmain.css',
	};

(In the CSS theme file:)
	background-image: url("https://ajax.googleapis.com/ajax/libs/jqueryui/1.12.1/themes/smoothness/images/ui-icons_222222_256x240.png");
	background-image: url("https://ajax.googleapis.com/ajax/libs/jqueryui/1.12.1/themes/smoothness/images/ui-icons_454545_256x240.png");
	background-image: url("https://ajax.googleapis.com/ajax/libs/jqueryui/1.12.1/themes/smoothness/images/ui-icons_2e83ff_256x240.png");
	background-image: url("https://ajax.googleapis.com/ajax/libs/jqueryui/1.12.1/themes/smoothness/images/ui-icons_cd0a0a_256x240.png");
	background-image: url("https://ajax.googleapis.com/ajax/libs/jqueryui/1.12.1/themes/smoothness/images/ui-icons_888888_256x240.png");

If I’m no longer able to maintain the framework, the full and up-to-date source is available on both GreasyFork and Github. Anyone can fork it and create their own version.

[Ethan]

Okay I’m going to sink my teeth into this now, looks exciting!

[rfindley]

Great! Check out the documentation at https://github.com/rfindley/wanikani-open-framework
Lots of working examples, though not quite as much ‘getting started’ as I’d like yet.

[Ethan]

How did you generate the md file? It’s beautiful

[rfindley]

I’m glad you like it!
I wrote the content in a text editor in raw markdown.

To simulate github’s .md rendering while under development, I set up an nginx web server to redirect .md URLs to an html with ?file=xxxxx.md, and the html loads commonmark.js to convert the markdown to html, and prism.js for code highlighting with a github CSS color theme.

[Ethan]

I’m tinkering again…
What is your contribution policy on this project?

[rfindley]

Nice! What do you have in mind?

I haven’t really put together a contribution policy yet, but…

Essentially, I’ll be happy to accept contributions that stick to the main goals of the project, plus a few design guidelines, and a few caveats.

Main goals:

• Things added to the framework should be of general interest so that it doesn’t grow into a large collection of one-off stuff that may be difficult to maintain when Wanikani changes. If something is useful, but isn’t both ‘generic’ and ‘general interest’, consider whether it may be worth building a separate library that can be installed alongside the framework (such as seanblue’s Additional Filters script, which is ‘general interest’, but has some specificity that makes it not quite ‘generic’, such as its particular definition of ‘leeches’).

• Stick to 80/20 design: The API should aim to be simple for 80% of use-cases, but should leave room for the other 20% to customize. In other words, using the framework should be relatively painless for newbie scripters, but we still want it to be flexible enough that experts won’t feel locked into a one-size-fits-all API.

Design guidelines:

• Always think about how your code might interact with multiple client scripts. This includes race conditions, script load order, manipulation of the DOM, use of variables in global space, etc.

• Visual elements should be as independent of WK as practical. For example, the Settings and Progress dialogs are placed in a top-level DOM container created by the framework in order to avoid dependencies on the Wanikani layout and CSS. If Wanikani did a total site makeover, I think those modules would still function the same. There are some exceptions, of course. The Menu module depends on Wanikani’s interface. But it’s also light-weight, so it will be easy to maintain.

• If creating a new module, be sure to follow the general design of the existing modules:

  • Wrap everything in an anonymous function to prevent accidental overlap in the global space.
  • Make a ‘Published Interface’ section at the top of the code that gives a good idea of what the module does.
  • Use heirarchy to keep things organized, especially in the wkof object.
  • Modules must set a state variable when they are ready to be used. See the notify_ready() function in Apiv2.js for example.

Caveats:

• I can’t (and don’t want to) own the framework forever, but I do want to curate it somewhat for a while so it stays clean and true to its purpose (e.g. “Too many cooks spoil the broth”). My hope is that, if people want to continue adding to the framework, someone will show enough interest to take over curation. Because, as things are right now, if the framework breaks, WK is going to suddenly be a very different place for a lot of users!
• I don’t have a lot of time to spend on integrating new stuff. So, since it’s a lot faster to review code than to write it, I would encourage keeping in frequent contact so I can help guide the direction of anything intended for integration. I’m more than happy to answer questions and give suggestions, so don’t hesitate to ask .

I can give out my email address if anyone wants to discuss anything offline. But if the discussion is useful to the developer community, just post in this thread.

[rfindley]

Speaking of modifications to the Open Framework… With only about 30min of work, I managed to get the framework working in a staging area for the new wkstats.com site. That’s really exciting because it means it’s going to be easy for other developers already familiar with the framework to be able to contribute to the stats site. My hope is that scripters can develop their ideas in userscript form on top of the stats site, and then those ideas can be quickly integrated into the site.

I’ll post more detail as I make progress.

[Ethan]

I am in awe of your code and trying not to feel stupid with how narrow my focus has become from my job. It’s hard for me to look at something like this and not think that the author just sat down and coded the whole thing over a weekend or afternoon.

I’ve made a couple pull requests but sometimes think the problems I solve will either cause more problems, or the solutions were already considered and discarded for some reason.

[rfindley]

Heh, nothing to worry about. Just put some ideas out there and we’ll talk through how they might be implemented. You learn (or relearn) by doing.

[rfindley]

@seanblue @normful @Kumirei @DaisukeJigen @valeth @Ethan

I’ve updated the recommended code for checking if the Open Framework is installed. You may want to consider updating your scripts accordingly.

The intent is to give the user a choice about whether to redirect to the Install page if the Open Framework is missing or outdated.

The new recommendation is [here].

Thanks to @seanblue for recommending the use of confirm() instead of alert().

[normful]

Thanks for the notification (and all your work on this framework!). Updated!