Wanikani Open Framework [developer thread]

DaisukeJigen · March 6, 2018, 1:56pm

Sounds like that’ll be more than plenty. Thanks.
Probably won’t get a chance to play around until Friday though.

valeth · March 8, 2018, 7:02pm

@rfindley The framework shaping up nicely.
I already started porting my script to it and it removed quite a bit of boilerplate code.

There are a few things that would be nice to have:

Scoping for settings keys

Currently the settings generated by defining the menu seem to be only a flat one dimensional hash.
It would be nice if settings could be scoped, either by page, tab or group.
So if I had something like this:

let settings_dialog = new wkof.Settings({
  script_id: "wklc"
  title: "WaniKani Lesson Cap"
  on_save: save_setting
  settings: {
    general: {
      type: "page",
      label: "General",
      content: {
        max_score: { type: "number", label: "Max score", default: 100 }
      }
    },
    scores: {
      type: "page",
      label: "Scores",
      content: {
        apprentice1: { type: "number", label: titleize("apprentice1"), default: 6.0 },
        apprentice2: { type: "number", label: titleize("apprentice1"), default: 3.0 }
      }
    }
  }
});

I would like to get a structure like this:

"wklc": {
  "general": {
    "max_score": 100
  },
  "scores": {
    "apprentice1": 6.0,
    "apprentice2": 3.0
  }
}

This would make it easier to iterate over certain collections of settings for example.

Multi select options

Not sure if this is already in the framework, didn’t look through all of the code yet, but having multiselect as an option would be nice for some settings. The other alternative is to create a lot of checkboxes, which is a bit tedious.

I also noticed that input fields seem to be always of type="text", despite the type being specified as "number" in the settings setup.

Well, that turned out to be a longer post than expected

rfindley · March 8, 2018, 10:47pm

Great feedback… thanks!

I’ve already added some of the changes/additions you mentioned, though I haven’t uploaded them yet since I seem to be making several changes per day.

I’ve added a ‘list’ type (though I may rename that… maybe ‘select’?), which is a non-dropdown <select>. It has support for ‘multi_select:true’, but I still need to test it.

I’ve also added ‘section’, which creates a section header in the dialog (I’ll post pics when I’m back at my PC), and ‘divider’, which is <hr> (Should I just call that type ‘hr’?)

I added an ‘html’ type, which allows you to do advanced or custom stuff beyond what the basic settings can do. It has a ‘pre_open’ callback that you can use to attach event handlers.

Like you, I’ve also thought about structured settings. I definitely don’t want to tie the settings data structure to the dialog structure, because I have several dialogs that I know won’t work well that way. I’ve thought about using a dotted field name (e.g. ‘general.max_score’). Currently, you can actually do that and it will do something like:

setting.myscript[‘general.max_score’]

Then you could iterate via:

var general_settings = Object.keys(settings.myscript).filter(name => name.match(/^general\./) !== null);

But that’s messy. So, I’m trying to think of a good way to parse a nested field name without the code getting overly complicated. I have some ideas, and will probably work on that tonight.

Oh, and ‘number’ being ‘text’ was a total oversight. I’ll fix that tonight. And maybe I’ll add type:'input' that accepts a subtype (e.g. subtype:'password'), so I can support all the other less common types without bloating the code.

rfindley · March 13, 2018, 10:21am

I’m still working on this, and will hopefully be done in the next day or two. Adding the scoping really messed up the code, but I’ve got it mostly straightened up again, and I think it’s going to be a lot more powerful now. I added a “path” parameter that overrides the default flat scoping.

Using your example, path can be something simple like “@scores.apprentice1”, where ‘@’ is shorthand for wkof.settings.your_script.

But you can also use nearly any expression in the path, which opens up some interesting possibilities.
In the example below, I have a dropdown that lets you choose between some presets, and two checkboxes that modify the values inside the active preset. (i.e. presets[active_preset].some_setting)

wkof.settings.my_script = {
    active_preset: 0,
    presets: [
        {reading2meaning: true, meaning2reading: false},
        {reading2meaning: false, meaning2reading: true},
        {reading2meaning: true, meaning2reading: true}
    ]
}

var config = {
    active_preset: {type: 'dropdown', refresh_on_change: true, content: {
        '0': 'JP to EN',
        '1': 'EN to JP'
        '2': 'Both'
    },
    reading2meaning: {
        type:'checkbox',
        path:'@presets[@active_preset].reading2meaning'
    },
    meaning2reading: {
        type:'checkbox',
        path:'@presets[@active_preset].meaning2reading'
    }
}

The ‘refresh_on_change: true’ tells it to update all of the controls when the preset is changed. All automatic!

Kumirei · March 13, 2018, 12:00pm

I really appreciate the work you’re putting into this

rfindley · March 15, 2018, 11:03pm

@valeth, @Kumirei, @seanblue, @acm2010, @DaisukeJigen,

I’m ready to push an update for the wkof.Settings module, and want to make sure I don’t break anyone’s script. To the best of my knowledge, I think the only breaking change is:

The parameters passed to any “on_change” functions.
- before: on_change(value, config)
- after: on_change(name, value, config)
  - name - The name of the particular setting’s node in the configuration object.
  - value - The new value of the setting.
  - config - The contents of the of the particular setting’s node in the configuration object.

I could have just put the name parameter at the end to avoid breaking the interface, but having it first makes it consistent with the rest of the interface, and as far as I’m aware, no one has a released script that uses the on_change callbacks yet.

…which brings me to my next point. It would be great to have a registry of scripts that use the framework, so I can look at people’s code before making changes to make sure I don’t break anything. And, in some cases, maybe I can do some testing just to make sure.

So, if you’re using the framework, would you all mind posting a link to any of your released scripts? Also, if I have any big updates coming up, I’ll post here beforehand if I think there’s much risk of anything breaking.

Anyway, I haven’t uploaded the changes yet. I need to do some “quality control” checks beforehand. But I’ll be posting it within the next 18 hours or so.

Okay, so some of the additional changes to the Settings module:

Settings dialog background

I noticed that my (existing) Ultimate Timeline tends to grab certain mouse events when trying to resize a Settings dialog. So, I made a semi-transparent (i.e. tinted) full-screen background that will catch any mouse actions that fall through while resizing or moving a dialog. The background essentially makes the settings dialogs modal. The background is optional (ON by default), and can be disabled by adding background:false to your Settings configuration object. The background also uses a reference counter, so if more than one dialog is open, the background will stay open until the counter drops back down to zero.

Structured wkof.settings.my_script (i.e. “path” parameter)

Originally, all settings saved by the Settings dialog would be placed directly under your script’s sub-object:

wkof.settings.my_script = {
    setting1: "value1",
    setting2: "value2",
    ...
}

Now, each setting in your dialog configuration can add a path field, which can evaluate complex expressions.
Suppose you want a non-flat structure in your script’s settings:

wkof.settings.my_script = {
    active_preset: '0',
    presets: [
        '0': {name:'Preset #1', setting1:'value1', setting2: 'value2'},
        '1': {name:'Preset #2', setting1:'value1', setting2: 'value2'},
    ]
}

You can now add a path expression to any setting your dialog configuration:

    setting1: {
        path: '@presets[@active_preset].setting1'
        ...
    }
});

The “@” is shorthand for “wkof.settings.my_script.”, where “my_script” is your script name, of course.

First, I run a path.replace() on the “@”, then I do an eval(). So, you can technically use any valid left-hand side of an assignment expression. Other examples might include:

    path:'@group1["setting_"+@index]'

which translates to:

    wkof.settings.my_script.group1["setting_"+wkof.settings.my_script.index]

New types

There’s now support for:

A ‘list’ type with a multi:true for selecting mulitple values.
A ‘section’ type that draws a section header bar.
A ‘divider’ type that is an <hr> tag.
An ‘input’ type, with ‘subtype’ support, so you can do things like <input type="password">
An ‘html’ type that lets you add arbitary inline html. You can then use the new global “pre_open” callback to attach any event handlers or whatever.

Automatic refresh

If you have a setting that affects other settings, the new “refresh_on_change:true” can be used to trigger the dialog to update based on the current contents of wkof.settings.my_script.

// Somewhere Inside your dialog configuration:
var dialog = new wkof.Settings({
    ...
    active_preset: {type:'dropdown', refresh_on_change:true},
    setting1: {type: 'checkbox', path:'@presets[@active_preset]'},
    ...
}

In this example, when you change the active preset, the “refresh_on_change” will cause "setting1" to be re-evaluated based on the new preset.

In the example, “setting1” changed because its “path” referred to the active_preset. But you could also use an "on_change" event to modify other settings in a more complex way.

DaisukeJigen · March 15, 2018, 11:10pm

Damn man, you’re killing it.
The only script I have going with Open Framework so far isn’t published anywhere. I’m on a different machine right now, so I can’t double-check how I have it set-up, but since nobodies using it, feel free to break it on me

seanblue · March 15, 2018, 11:29pm

I’m not using that function, so I’m good on my end. And here is a link to my script’s forum post: [Userscript] WaniKani Lesson Hover Details

How are you planning to manage breaking changes in the long run? If your framework becomes even remotely popular, breaking changes could cause some real headaches for users. I think it’ll be especially problematic because users can only have one version of the framework and devs will respond to breaking changes at different rates, meaning it’s unlikely all of a user’s scripts will be compatible with the same version of the framework for a period of time.

rfindley · March 15, 2018, 11:52pm

Mostly, I’m hoping there will be very few breaking changes, but I’ve also taken some precautions to protect backward compatibility. For example, in most of the complex functions, you pass in all of the arguments inside a single object, rather than as individual arguments. So, if new parameters are needed, you just add them to the object, and argument order is irrelevant.

Obviously, I didn’t do that in this particular case. Maybe I should. I’ll think about it before I push the update.

Another possible method is to make a xxxxxx_v2() version of a function if there’s a significant break to a commonly-used function. I’m hoping I won’t have to do that, but it’s a model that works.

rfindley · March 20, 2018, 11:09pm

The changes to the Settings module are now live on greasyfork and github.

I’ve created an examples/ folder on github, and will start posted demo client code there. Currently, there’s the original sample_client.js, and now a new settings_demo.js.

The settings_demo.js shows a lot of the features of the Settings module. I still need to add some of the supported features to the demo (I should have that up soon). That will serve as initial ‘documentation’ until I can spend the time to update the official docs.

rfindley · March 22, 2018, 5:42am

I’ve added documentation on github for the Apiv2, Menu, and Progress modules.
[Documentation]

The only documentation still missing is the Settings module. It’s the most complex module, but it’s also the most well ‘documented’ via [demo code]. I will probably delay documenting it until I finish updating my Self-Study script, because that script is triggering a lot of additions to the Settings module.

rfindley · March 24, 2018, 4:58pm

I’m getting excited to finish the Self-Study Quiz script.
It has contributed a lot of nice features to the Open Framework!
I’ll be uploading some Framework updates within the next few days. No breaking changes… just new features and a few bug fixes.

rfindley · March 26, 2018, 11:07pm

I’ve pushed some updates:

ItemData module:

Fix typo in srs filter on get_items() – Apprentice 4 was changed from app4 to appr4.
Change default item_type filter from ['rad','kan','voc'] to [] (does not affect any existing scripts)
Change default level filter from "1-60" to "" (does not affect any existing scripts)
Added hover_tip to filters, for use in user interfaces when selecting filters.
Added set_options to filters automatically include endpoints that are required by a filter.

Settings module:

Added hover_tip support to the Settings() config, so all input fields and tab pages in the dialog can have hover-tip help.
Added support for placeholder for text inputs.
Added a tabset type, so tabs can be used other than just at the top level.
Added id attributes to group and page objects (useful for CSS selectors)

Apiv2:

Added support for a wider variety of date formats in the updated_after and last_update filters. Now you can use a Date object, or any timestamp string that is understood by the Date() constructor.

CSS:

Improved some CSS for handling narrow screens (mostly on Settings dialogs).

Docs:

Improved the guidance for which function to use to retrieve a particular set of API data. The verbage is included on the following functions:
- wkof.ItemData.get_items()
- wkof.Apiv2.get_endpoint()
- wkof.Apiv2.fetch_endpoint()

rfindley · March 27, 2018, 6:12pm

I pushed a small change today.

Since the framework shares certain URLs for CSS and jquery_ui, I decided to consolidate them into the Core.js module:

wkof.support_files[file_id]

So, suppose you want to use jqueryUI in your own script, but you aren’t sure if its already been loaded by the framework or another script, you can do this:

wkof.load_script(wkof.support_files['jquery_ui.js'], true /* use_cache */)
.then(do_something);

This will automatically detect if the file has already been loaded, and only load it if necessary.

Currently, there are only two files:

jqui_wkmain.css - A CSS file used for styling the Settings dialogs and Progress bar dialog
jquery_ui.js - Just the standard jQuery UI, drawn from google’s speed libraries.

rfindley · March 27, 2018, 10:03pm

Added auto-cleanup of the wkof.file_cache.
Mainly, the intent is to remove old versions of files. Up to now, every time I bumped a script rev, it has been accumulating old versions in cache.

The cleanup algorithm is experimental, and may need some refinement. Currently, if a file hasn’t been accessed in two weeks, it will be deleted. I think that will catch most use-cases while minimizing unwanted flushes. After all, if someone hasn’t been on WK in two weeks, a 10-15 second re-population of the API endpoints is the least of their worries But if needed, I can exclude the API endpoints from cache cleanup since the filenames aren’t likely to change (meaning no accumulation of defunct data),

Please let me know if you think of any corner cases that I may want to consider.

DaisukeJigen · April 2, 2018, 4:12pm

We can set defaults on setting values, but it appears the settings don’t actually exist until the popup is opened, and then saved. Can always check for undefined and adjust accordingly, but would be nice not to have to do that. Both when first installing the script, and when/if I add a setting after someones already running said script.
Probably just missing something though…

rfindley · April 2, 2018, 5:18pm

You can pre-load your settings via:

wkof.Settings.load('my_script');

That allows you to create the dialog only when needed.
You can also use:

wkof.Settings.save('my_script');

if you change any settings outside of the dialog. That will allow the framework to notify any other interested scripts (such as a future settings-syncing script) to know when something has changed in wkof.settings.*

If you have already created your settings dialog, you can also use:

dialog.load();
- or -
dialod.save();

Those are just shortcuts for the other functions above.

rfindley · April 2, 2018, 5:31pm

By the way, saving and loading are asynchronous and return a promise. You’ll need to make sure the action is complete before accessing the settings:

wkof.Settings.load('my_script').then(do_something);

DaisukeJigen · April 2, 2018, 5:46pm

I have my settings. Opened dialog and saved. Then I edit the script to have a new setting ‘test’. Reloading the page, test (with its default) is not loaded with the settings. Nor were the defaults loaded when a user runs the script for the first time.

Summary

...
        wkof.include('Apiv2, Menu, Settings');
        wkof.ready('Menu').then(install_menu);
        wkof.ready('Settings').then(install_settings);
...
    function install_settings() {
        settings_dialog = new wkof.Settings({
            script_id: 'fawa',
            name: 'fawa',
            title: 'Fast Abridged Wrong/Multiple Answer',
            on_save: process_settings,
            settings: {
                'alwaysShow': {type:'checkbox',label:'Always Show Correct Answers',default:defaults.alwaysShow},
                'alwaysShowOnlyMultiple': {type:'checkbox',label:'&nbsp;&nbsp;&nbsp;(Only if multiple answers)',default:defaults.alwaysShowOnlyMultiple},
                'dontShowWrong': {type:'checkbox',label:'&nbsp;&nbsp;&nbsp;(Don\'t show wrongs)',default:defaults.dontShowWrong},
                'test': {type:'checkbox',label:'&nbsp;&nbsp;&nbsp;(Don\'t show wrongs)',default:true}
            }
        });
        settings_dialog.load();
    }
...

wkof.settings.fawa
{alwaysShow: true, alwaysShowOnlyMultiple: true, dontShowWrong: false}
alwaysShow: true
alwaysShowOnlyMultiple: true
dontShowWrong: false
__proto__: Object

But, $.extend can get me around this:

Summary

    var defaults = {
        alwaysShow: false,
        alwaysShowOnlyMultiple: false,
        dontShowWrong: false,
        test: true
    };
settings_dialog.load().then(function(){
            wkof.settings.fawa = $.extend(defaults,wkof.settings.fawa);
            settings_dialog.save();
        });

rfindley · April 2, 2018, 5:59pm

The load() function doesn’t populate defaults because it works independently from the dialog settings since you can load settings without creating a dialog first. In my own scripts, I do exactly what you did with $.extend(), except I used the deep-copy flag:

wkof.settings.fawa = $.extend(true, {}, defaults, wkod.settings.fawa);

That also leaves defaults intact (if that matters) by merging it into the empty {} instead of merging wkof.settings.fawa into defaults.

EDIT: I was going to look into pre-populating the defaults when you call load() from the dialog object… but I realized it’s not always going to be possible since the path parameter allows some settings to be reliant upon other settings (e.g. path:'@presets[@active_preset]'). So, I think $.extend() is still the best option.