Wanikani Open Framework [developer thread]

@valeth, @DaisukeJigen, @hitechbunny, @acm2010, @seanblue, @Subversity, @irrelephant

ItemData module

I’ve pushed some updates. The ItemData modules is now online. I haven’t written up the documentation yet, but it’s essentially just a single function if all you want to do is fetch data. See the sample_client.js for an example.

Using ItemData

First, create a configuration object that specifies what you want to fetch, and what filters you want to apply.

Here’s the bare minimum, which will fetch all Wanikani items from the /subjects endpoint:

wkof.include('ItemData');
wkof.ready('ItemData').then(fetch_items);

function fetch_items() {
	var items_config = {
		wk_items: {}
	};

	wkof.ItemData.get_items(items_config)
	.then(process_items);
}

function process_items(items) {
	// TODO: Do something with your retrieved items.
	console.log('Our data is ready!');
}

Data sources

In the example above, “wk_items” is the data source that we are drawing from, and represents all item data from Wanikani, and is built into ItemData. However, ItemData is designed to be extended by other scripts, so someone could, for example, write a script that supplies a set of Core10k vocabulary, and registers it with ItemData. That data source could then be found by other scripts (like a quiz script) by looking in wkof.ItemData.registry.

Here’s a look at the top-level contents wk_items in the registry:
image

The main components to note are:

  • fetchera function that fetches the data for this particular registered data source
  • filterspre-defined filters for selecting data returned by the fetcher
  • optionswhatever options this data source supports

Options

The wk_items data source currently supports the following options:

  • assignments (bool) - fetch the /assignments endpoint and cross-link it with /subjects
  • review_statistics (bool) - fetch the /review_statistics endpoint and cross-link it with /subjects
  • study_materials (bool) - fetch the /study_materials endpoint and cross-link it with /subjects

If you explore these options, you’ll see that they contain meta-data useful for creating a user interface for selecting options, such as data type, label, and default value.

Filters

The wk_items data source currently supports the following filters:

  • level (string) - A comma-delimited list of levels and level ranges of items to include in the result.
  • item_type (array/multi) - The item types to include (‘rad’, ‘kan’, ‘voc’).
  • srs (array/multi) - The SRS levels to include (‘appr1’,‘appr2’,…,‘burn’).
  • has_burned (bool) - Items that have been burned (including resurrected ones).

I plan to add more filters as needed, so feel free to make requests. Keep in mind, though, that your own scripts can also add their own filters directly into the registry if you want them to be available to users and other scripts… like maybe a Leech filter. Or, if your filter doesn’t need to be public, simply use Array.filter() on the returned items.

Like the options, the filters also have meta-data that is helpful for creating a user interface for interactively configuring your data selection.

Using options and filters

Let’s look at an example that makes use of the options and filters:

	var items_config = {
		wk_items: {
			options: {
				assignments: true, // API endpoint needed for filtering by SRS level
			},
			filters: {
				item_type: {value:['voc']}, // Include only vocabulary
				level: {value:'1 - -1'}, // Include items from levels 1 to [current level - 1]
				srs: {value: ['appr1','appr2','appr3','appr4']} // Include only Apprentice items
			}
		}
	};

You can also invert the result of any filter:

	srs: {value: ['enli','burn'], invert: true} // Include all except burned and enlightened

Structure of returned item data

The returned data is an array of items identical to the structure returned by data field of the /subjects endpoint:

[{item1}, {item2}, {item3}, ...]

Where each {item} is:

{
	"id": 1,
	"object": "radical",
	"url": "https://www.wanikani.com/api/v2/subjects/1",
	"data_updated_at": "2017-06-12T23:21:17.248393Z",
	"data": {
		"level": 1,
		"created_at": "2012-02-27T18:08:16.000000Z",
		"slug": "ground",
		"document_url": "https://www.wanikani.com/radicals/ground",
		"character": "一",
		"character_images": [],
		"meanings": [{
			"meaning": "Ground",
			"primary": true
		}]
	}
}

Suppose the above radical was {item2} in our array of items:

item[1].data.meanings[0].meaning
// --> 'Ground'

Other endpoints

In the options field, you can include the following endpoints in your fetch:

  • /assignments
  • /review_statistics
  • /study_materials

These endpoints contain user-specific supplemental data for each item found in the /subjects endpoint.
When you include them via the options field of the items_config, they will be automatically cross-linked to the returned data. For example:

{
	"id": 1,
	"object": "radical",
	"url": "https://www.wanikani.com/api/v2/subjects/1",
	"data_updated_at": "2017-06-12T23:21:17.248393Z",
	"data": { ... },  // <-- Data from the /subjects endpoint
	"assignments": { ... },  // <-- Data from the "/assignments" endpoint
	"review_statistics": { ... },  // <-- Data from the "/review_statistics" endpoint
	"study_materials": { ... },  // <-- Data from the "/study_materials" endpoint
}

The format of each sub-object is identical to the data sub-object of the corresponding endpoint.
So, for example, if you want to access your meaning_notes for the above radical:

item[1].study_materials.meaning_synonyms
// --> ["One"]
2 Likes

So let’s say I want to find out how many lessons of each type I have. Can I do that with your framework at this point? I haven’t followed the v2 discussion much at this point, so I’m not even sure if this is directly supported.

@seanblue,

Here’s a complete script to count the number of each item type in your current lessons.
I opted to use wkof.Apiv2.get_endpoint() instead of wkof.ItemData.get_items() because you’re task doesn’t need any cross-linking between the /subjects endpoint and the other endpoints (like /assignments)

	wkof.include('Apiv2, ItemData');
	wkof.ready('Apiv2, ItemData').then(fetch_data);

	function fetch_data() {
		console.log('Fetching Data...');

		// Start fetching the /summary and /subjects endpoints.
		// Store the returned promises in an array.
		var promises = [];
		promises.push(wkof.Apiv2.get_endpoint('summary'));
		promises.push(wkof.Apiv2.get_endpoint('subjects'));

		// When all fetches are done, process the data.
		Promise.all(promises).then(process_data);
	}

	function process_data(results) {
		console.log('Processing Data...');

		// Grab the fetched results (same order as the promises array above).
		var summary = results[0];
		var subjects = results[1];

		// Initialize the type counts.
		var count = {radical: 0, kanji: 0, vocabulary: 0};

		// Pull the list of subject_ids from the lesson list in 'summary'.
		var lesson_subject_ids = summary.lessons[0].subject_ids;
		lesson_subject_ids.forEach(function(subj_id){
			var item = subjects[subj_id];
			count[item.object]++;
		});

		console.log('Lessons available');
		console.log('-----------------');
		console.log('Radicals  : '+count.radical);
		console.log('Kanji     : '+count.kanji);
		console.log('Vocabulary: '+count.vocabulary);
	}

@seanblue,
Pinging you to let you know I updated the script above.
The first time you run it, it may take 10-15sec to fetch and cache all of the /subjects data. After that, it only takes a fraction of a second.

The next thing on my agenda is to add a global progress bar for data fetches, so users will know what’s going on, and client scripts won’t even have to think about it. It should only show up when the user’s cache is starting out empty.

1 Like

Thanks, I was able to get my script working with that as the starting point. I’ll wait until the framework is officially released before distributing it.

I have one question though: did you decide to go with the stand-alone install for the framework as the long term approach? Or is that just for this early access period?

I’m hoping I can use that long-term. I think it’s the better solution. We just need to see if users have a problems with it. And if they do, attempt better documentation first, then maybe resort to an alternative.

By the way, I may pop back in with a suggestion for a change from the code I gave you above.
I had opted for wkof.Apiv2.get_endpoint() because it was easier. But there are some down-sides.

Since then, I’ve added a function wkof.ItemData.get_index() that makes it easier to use wkof.ItemData.get_items(). And the result plays nicer with simultaneous scripts.

But I haven’t pushed that update to github yet, because I want to write a few more index functions.
Rignt now, wkof.ItemData.get_items() returns an array, and you need to iterate over it to find something. But the get_index() functions will take that array and create an object with keys based on your choice of field inside the item. E.g.: wkof.ItemData.get_index(items, ‘slug’), which will return an object like this:

{
	"市立": {  <item info> },
	"上手": {  <item info> },
	...
}

Hopefully it eventually becomes like installing a script manager and becomes something people just know to do when you install your first script. But of course you’ll still have people who don’t know to install it and you’ll probably have devs that forget to mention it in their forum posts sharing their scripts.

I’m just going to keep it as is for now, especially since I’m not sharing it yet. I’ll update the script to a more optimal approach once the framework is more stable.

Thanks again :slight_smile:

1 Like

I’m thinking about encouraging scripters to put a tiny piece of code at the top of their scripts:

if (!wkof) window.location.href = “forum_url_telling_users_to_install_the_framework”;

3 Likes

I’m fleshing out the documentation on Github [here]

I’m currently populating everything into the top-level README.md because it’s easier to write the documentation that way. If anyone has feedback on documentation style or details, please let me know.

For now, I’m leaving the /docs subfolder intact because it contains more information, though it’s extremely rough. As I progress on the main README.md, I plan to delete the contents of the /docs folder incrementally.

1 Like

It’s amazing to see you still working on things past level 60. I feel like after I burn most stuff I’ll not be hanging around so much. Thanks for all the hard work!

@rfindley When I run my script using your framework on a page without the user menu (like the lesson page), I get an error in the console saying Uncaught (in promise) Couldn't extract username from user menu!. It looks like you are rejecting the promise if you can’t find the username from the user menu, but wouldn’t that mean that no scripts can use the framework on pages that don’t display the user menu?

Good catch, thanks. I’ve mostly just been testing while on the dashboard since I haven’t implemented any practical applications yet.

I’ll fix that ASAP.

[Edit] The user and apikey verification has a lot of corner cases that can go wrong… like:

  • Two people using the same pc
  • Someone changing their apikey
  • A user that has never generated their apikey
  • Developers temporarily overriding their own apikey

Anyway, I’m working to make that as transparent as possible for client scripts, so you’ll never have to worry about apikey again.

1 Like

@seanblue,
By the way, the Framework supports overriding your apikey, which is helpful for testing a script with someone else’s API key.
And since it doesn’t check username when using an override key, you can override with your own apikey to get it working on the Lessons page until I fix the username issue.

To set an override key, run this in the javascript console:

localStorage.apiv2_key_override = "put an apiv2 key here"

Then refresh.

The script I wrote doesn’t even need to work on the lessons page, I just happened to notice the console error there. The script requires the “Lessons” bubble to be visible, so I have it running on every page since most pages show that bubble. Thanks for the information though. Still good to know.

Why does the script try to grab the username and API key every time? Is it because of those edge cases you described earlier?

The fix is live on github now. To update, you can clear Apiv2.js file from file_cache, then refresh:

wkof.file_cache.delete(/Apiv2.js/)

[Edit: After first release, updates will happen automatically. TamperMonkey will detect that Core.js was updated, and all of the modules will be on versioned URLs, so they will be loaded into cache upon first access. Old modules will expire from cache.]

1 Like

Working on the global “Loading” dialog for API (and other) data

image

The current concept is:

  • If it takes longer than 1sec to load anything, the dialog will pop up and start showing progress.
  • Each API endpoint has its own progress indicator, since they load in parallel.
  • It shows only the endpoints that are actually being requested.
  • If new endpoints are requested while the dialog is already up, they will be inserted in alphabetical order.
  • The dialog will close when all data has completed, or if the user closes it manually.

I think after this is working, I’ll do an official first-release.

3 Likes

:point_right: [v1.0.0] - Initial release
:point_right: [v1.0.1] - Fix repeating progress timer

Releases will be on greasyfork.
Development will be on github.

I still have a lot of work to do on the documentation (on github), but I’m going to take a break from it to convert my Burn Manager script to use the framework since Burn Manager is currently broken. In the meantime, a lot of questions can probably be answered by looking at the sample_client.js on github. Beyond that, I’ll be happy to answer specific questions here in the forums.

3 Likes

I was thinking I’d use the framework in a script for which I need only the number of items in each SRS level and may have a few questions. Is the best way to get these numbers just retrieving the list of items in each SRS level and checking their lengths?

Since it’s so fast to fetch all items (after it has cached the first time), just grab them all, index them by srs level, and get the length of each srs category.

wkof.ItemData.get_items('subjects,assignments').then(function(items){
    var by_srs = wkof.ItemData.get_index(items,'srs_stage_name');
    Object.keys(by_srs).forEach(function(srs_name) {
        console.log(srs_name+' => '+by_srs[srs_name].length+' items');
    });
});

If you prefer numeric srs levels, use ‘srs_stage’ instead of ‘srs_stage_name’.

It’s also worth noting that any other script that fetches all items will actually be accessing the same underlying objects, so the retrieval process is only done once no matter how many scripts do this. (The index is created dynamically, though, and will be unique to your script).

1 Like