Wanikani Open Framework [developer thread]

[rfindley]

By the way, you can skip the save() at this point in the code:

    settings_dialog.load().then(function(){
        wkof.settings.fawa = $.extend(defaults,wkof.settings.fawa);
        settings_dialog.save();  // <-- can skip this
    });

Since you’re always going to merge defaults anyway, you can just wait until the user changes something before saving. (Not that it really matters, though. It’s just my OCD kicking in )

[rfindley]

@DaisukeJigen,
I’ve added an optional ‘defaults’ argument to wkof.Settings.load().
Now you can:

var defaults = {...};
wkof.Settings.load('my_script', defaults).then(do_something);

It just uses $.extend() like we talked about above.

---

For anyone that hasn’t been following the discussion:

If you upgrade your script, and it gains a new setting in wkof.settings.your_script_name, the “defaults” argument of wkof.Settings.load('your_script_name', defaults) will now populate any missing settings without overwriting any existing saved settings.

[seanblue]

@rfindley I’m quoting from my question and oldbonsai’s answer in the APIv2 thread. I don’t see if there is a way to get this data out of the open framework. I want to add a filter with this criteria.

@seanblue

I’m trying to get all of the failed reviews in the last X hours. Is that possible with the data currently given by APIv2?

From an item’s review_statistics, I’m thinking that I should check data_updated_at to do the X hours part of the filter. I don’t see a way to get all wrong reviews in that period, but I think I can find if the most recent review for an item was wrong. I think I could do that by checking meaning_incorrect >= 1 && meaning_current_streak == 1 (and same for reading). I think the first part is necessary to avoid including when the first review was correct, since the initial lesson doesn’t add to the streak.

Does it sound like that would work? Can anyone think of a way to make that more concise?

@oldbonsai

You can also use reviews, grab everything with the updated_after filter, then look for ones with incorrect counts that’re greater than 0. Then you don’t have to worry about streaks and all that from the statistics, plus you’re getting all the reviews for a subject that happened during the time, even if there’s more than one.

It looks like you don’t have the data_updated_at data point for review_statistics, and I don’t see detailed review information anywhere. Am I missing something or is that data not available through the framework right now. At the very least, could you add the data_updated_at data point for the review_statistics endpoint?

[rfindley]

@seanblue

It looks like you don’t have the data_updated_at data point for review_statistics, and I don’t see detailed review information anywhere.

You’ll need to use the Apiv2 module for the /reviews endpoint. I don’t cross-link the /reviews against the /subjects in the ItemData module because there could be a half million datapoints in /reviews. I figured, in most cases, a scripter will want to fetch a subset of /reviews data, so I won’t cache all of it.

Here’s the function you’ll want to use:

GitHub - rfindley/wanikani-open-framework: Open framework for writing...

[seanblue]

That makes sense, thanks for the information. Given the way this works though, it seems difficult to have a filter where the hours data point can be changed for reviews. I’m sure I could get this to work, even with the pagination, but it might be more hassle than it’s worth. And if people have more than one filter with different last_update values for the same /reviews endpoint, it could overload the API more than I’d like.

For this reason, I’m wondering if it might be better if I had the hours data point as an overall setting for my Filter Pack script. Then I could at least load all the data once on page load.

But I’m still worried about timing issues with this approach. If someone opens the Self Study script (in my use case) before all pages from the /reviews endpoint have returned, there could be some issues using a filter that relies on the review data being there. Do you have any suggestions to avoid this problem?

[rfindley]

I don’t know enough about what you’re doing to understand half of what you said.

Regarding pagination, fetch_endpoint() handles that for you. It returns one big array of everything you requested.

If you have more than one filter relying on /reviews, and those filters are aware of each other, I’d recommend just selecting the oldest date for last_update, and share the relevant subset of the results with each filter. I couldn’t think of a good generic way of handling /reviews inside the framework without possibly storing way more data than intended.

The timing is certainly a concern, and one I had considered several times. My conclusion was that, when someone asked me how to deal with filters that are asynchronous, I would add support for a promise in the filters to wait until it’s ready. I’m happy to do that.

[seanblue]

@rfindley

Regarding pagination, fetch_endpoint() handles that for you. It returns one big array of everything you requested.

That’s good, thanks for that.

@rfindley

I don’t know enough about what you’re doing to understand half of what you said.

In short: Self Study for recent reviews, where “recent” is configurable.

@rfindley

The timing is certainly a concern, and one I had considered several times. My conclusion was that, when someone asked me how to deal with filters that are asynchronous, I would add support for a promise in the filters to wait until it’s ready. I’m happy to do that.

It seems likely that people will want to use the Self Study script for reviewing recently failed items, or other criteria related to reviews. Are you thinking something along the lines of passing a promise into a filter’s options, and then using it in the Open Framework to only apply the filter when that promise is resolved? Something along those lines should work very well for my use case.

Sample Code

var reviewsPromise = new Promise();
var reviews;

wkof.include('Apiv2, ItemData');
wkof.ready('Apiv2, ItemData').then(fetch_data);

function fetch_data() {
	var options = {
		last_update: '2018-04-01T12:00:00Z' // Any date format accepted by "new Date(string)"
	};

	wkof.Apiv2.fetch_endpoint('reviews', options).then(process_response);

	wkof.ItemData.registry.sources.wk_items.filters.some_filter= {
		type: 'number',
		label: 'Something',
		filter_func: filter_callback,
		promise: reviewsPromise, // Don't apply filter until this promise is resolved.
	};
}

function process_response(response) {
	reviews = response; // Save for later.
	reviewsPromise.resolve(); // Resolve the promise since we have the data needed for the filter.
}

function filter_callback(filterValue, item) {
	// Cross reference data in the reviews variable.
}

[rfindley]

Rather than a promise in the config, I’m thinking a function, like maybe prepare, that optionally returns a promise.

	wkof.ItemData.registry.sources.wk_items.filters.some_filter= {
		type: 'number',
		label: 'Something',
		filter_func: filter_callback,
		prepare: prep_func, // Optionally returns a promise.
	};

function prep_func() {
    return wkof.Apiv2.fetch_endpoint(...)
    .then(function(data){
        // TODO: Do some prep work with the /reviews data
    });
}

The function would allow you to generate a promise when needed, or just return the promise that you’ve already fulfilled earlier.

[edit: And you could cache a partial endpoint via wkof.file_cache.save() and load(). It would be up to you to decide what data you want to keep, and how to throw out old data, etc]

[seanblue]

I see. That would work for my use and other use cases as well. Please @ me when this feature is available.

[rfindley]

@seanblue,

The prepare() function is added. It works exactly as described above. Just make sure you actually return a promise, or it won’t wait for completion.

I also fixed the error where Self-Study thought you were on Level 39. At startup, the Apiv2 module loads the contents of the /user endpoint at wkof.user, and I was using wkof.user.level. The problem was that I wasn’t fetching /user every time, and it was pulling a copy from cache.

Note: I somewhat recently added framework version checking, so if your script requires a minimum version of the framework, you can enforce it. Please read [here].

[seanblue]

@rfindley Can you add Initiate (srs stage 0) to your SRS Level filter?

[rfindley]

I’ll add it tonight.

[rfindley]

@seanblue

@rfindley Can you add Initiate (srs stage 0) to your SRS Level filter?

Done

[rfindley]

@seanblue,
FYI, I added ‘Locked’ to the SRS Level filter. It’s value is -1.
Initiate is basically items in the Lesson queue.
Locked is items that aren’t even available for lessons yet.

[valeth]

Regarding filters

@valeth

I generally find the filter options a bit confusing at points, like - and + are seemingly used for both arithmetic operations and ranges.

I would propose changing the range filters to use a range notation that is already common in a lot of languages:

1..3 or 1...3
I’ve seen and used this kind of notation in Rust and Ruby for example.

This would remove the ambiguity for -.

[rfindley]

I’ll add support for:

• 1-3
• 1..3
• 1...3
• 1 to 3

It’s only a single line of code. (Regex is fantastic!)

[rfindley]

Heh… while testing, I discovered that there are exactly 666 Reading and Meaning questions in levels 1…3.
Maybe a message from the Crabigator to those who don’t subscribe??

[rfindley]

Updated on github and greasyfork.

[seanblue]

I created a filter that checks the current time. I call Date.now() once in the prepare method so that all items check against the same “now” in the filter. I’m wondering if that’s something the Open Framework should initialize and expose so that all filters can use the same value for “now” when they are all applied. What do you think?

[rfindley]

I know what you mean in general, but can you give me a specific example of the where it’s needed? That would help me understand how it might be implemented.

On the wkstats.com site, I do a single Date.now() before running all of my calculations. But, of course, all of the code is working together for a single purpose. I’m trying to think of a similar scenario with wkof.