Wanikani Open Framework [developer thread]

With this configuration:

Settings

I get this result.

result

In other words, if I use Japanese commas I still get garbled results. But latin commas are OK.

It is the same in both Item Inspector and Self Study Quiz. This is not a script issue.

I have verified that the code you propose for split_list produces the correct string. This proposed code is OK to use. The problem is elsewhere.

I have verified that the .kanji_list.indexOf(item.data.slug) call used in the kanji filter locates the slug at the correct index.

I have verified that the slug contains the correct character.

I have verified that you use the correct unicode characters.for Japanese in split_list(str).

I have placed some console.log() command in the filter functions to get more information on what is going on. There is no console output as if the filter was never called. But this is absurd. The filter is obviously called because I see the items being filtered and all other filters are disabled.

I have copied your code in one of my test scripts where I know console output works. Guess what? I get console output and I can’t reproduce the problem there. Using Japanese punctuation works correctly in the test program.

So I investigated the greasyfork directives in your script to see whether the explanation is there. I removed the @run-at document-end directive. This is the culprit. Without this directive I get console output and the filter works fine.

You have to replace split_list() in the “Item List Filter” script and wkof’s ItemData module. (And also in wkof’s Core module when using Japanese commas or spaces in wkof.ready() ). And then you also need to clear or disable wkof’s cache to make sure you’re loading the updated version of the ItemData module.

I was able to get it working. (I’ll upload when I can. Having some problems with my editor at the moment).

This is obvious. I did that and still got the problem. I traced it to the @run-at document-end directive as I explained. Your code is fine but it doesn’t work for me until this directive is removed.

Edit: I just tested it on Firefox. This is a browser specific thing. The problem occurs on Chrome, not on Firefox.

I don’t think it’s a browser issue, and at most it’s maybe tangentially related to @run-at. I don’t doubt your results, but there’s no evident logic to them yet.

It works fine on my Chrome without modifying @run-at, as I would expect. I was getting bad results before changing split_list() in:

  • wkof ItemData module
  • wkof Core module
  • Item List Filter script

I modified those, then cleared wkof.file_cache.

After that, with the filters configured as you showed, all of the expected items (and nothing else) were showing up in the quiz.

1 Like

I am unable to reproduce your result. I changed split_list() in

  • wkof ItemData module
  • wkof Core module
  • Item List Filter script

I cleared wkof.file_cache. I did the whole thing twice to make sure there was no mistake. I still have the problem and there is no console.log() output from the traces I have put in the filters. But as soon as I removed the @run-at directive the problem disappeared and the console.log() output appeared.

I don’t understand the logic behind your approach. If the wkof modules and the cache were causing the interference, why would removing the directive work? It is the same wkof and the same cache running. Also why would the framework have an impact on console.log() output?

Besides I was able to get the filter running without changing the modules and clearing the cache when I removed the directive. I think this shows there is no interference from wkof.

An additional point: the current wkof (with the old split_list()) works in Firefox for me. There is no need to upgrade wkof on Firefox.

I don’t doubt you got the result you stated but it doesn’t work like this for me.

Clearing the cache is a drastic action. I lost all my settings. Are you sure you want to roll out a solution like this?

Edit: I think we are facing a bug involving the directive when running on Chrome. This would explain the inconsistent behavior and why the problem doesn’t appear on Firefox. There is no logic in bugs.

That’s where we differ. There is always logic in bugs, just not always evident until you’ve found the root cause.

Your screen-shots show exactly the result I would expect if the split_list() is not processing Japanese commas, and is exactly what I got before updating it (though I am not using your script to get results, I am using Self-Study Quiz). ground and gambler radicals are properly recognized due to latin commas, and ookii has a latin comma separating it from the rest of the line, including the immediately following Japanese comma. So, logically, split_list() isn’t being update somewhere. It exists in the three places I stated. Do you have another copy in your script? I haven’t installed it.

So, I fixed split_list()in those 3 locations, cleared cache to ensure the new version loads (which will not be necessary after I change the version number while publishing), and Self-Study Quiz gets the right results: 12 hits in the quiz… two each for kanji and vocab (reading and meaning) and one each for radicals.

So, from my perspective, the issue is resolved for wkof and self-study quiz. Any remaining issues are likely in your script, which I don’t have installed. Maybe you do have a timing issue in your script, but until you understand the root cause, blindly changing @run-at isn’t the step I would take. Try looking at the order you are intending to do things (e.g. load filters into the registry, then execute code based on them, etc) and think about what mechanisms are ensuring that things are really happening in that order.

2 Likes

I must disagree. My reasons:

  • wkof with the old split_list() works in firefox. This is not an issue of wkof not being up-to-date. Please test it and see for yourself.
  • The console.log() output appears and disappears depending on whether the @run-at directive is present. Something other than split_list() is happening.
  • My script loads filters using code I have borrowed from Self Study Quiz. There is no special check for the order of handling filters in this code. It just iterates over the registry. So if there is an issue with that we both have it.
  • I don’t use split_list() in my script.

Note: The console.log() out disappearance occurs when the console.log() command is located in the filtering function. It doesn’t happen when outside this function.

Is there a way to hide the progress popups that wkof automatically creates for certain api requests?

Not currently, but I could add an override for the timer.
Currently, it pops up if requests take longer than 1sec. That was more than enough time for most requests back when Apiv2 first started, but maybe it’s a little short now.
Next time I’m working on wkof (and if I remember), I’ll add a localStorage setting for overriding the default timer.

2 Likes

Cool. The use case I had in mind was that sometimes, a script wants to keep something up to date in the background. However, the user isn’t actually expecting/waiting for anything to complete so the progress bar is just distracting in that scenario. Ideally, this could be configured on a per-call basis so that for other scenarios, it can still be shown. Not sure how much work that would be.

There is your first significant clue: It’s not calling the code that you think it’s calling.
Do a “sanity check”. At the command-prompt, type:

wkof.ItemData.registry.sources.wk_items.filters.kanji_list.filter_func

and hit enter. It should show you the function contents (at least in Chrome, not sure about Firefox). Does that copy of code contain your console.log()? If not, that proves you are not running what you think you are. If it does contain your console.log(), then there are a few possibilities:

  • It’s the right code, but it’s not running through the code path that you thought it was.
  • You’re hot-swapping the function somehow with a different instance. Maybe there are two copies of the Item List filter: one installed directly in TamperMonkey and another included via @require. That could create a race condition that @run-at might seem to fix.

Of course, there may be other causes… I’m just throwing some ideas out there. But the main thing I wouldn’t suspect is that console.log() is malfunctioning. Replaced maybe. Not-called maybe. But malfunctioning? Not likely.

I know you’re focused on @run-at because things started working when you changed it. But that’s only a clue, not a solution. Until you can understand why it appears to fix the problem, I would hesitate to think that it is anything more than a clue.

How could it, since the old split_list() doesn’t recognize Japanese commas? Or did you removed the Japanese commas after your screen-capture above? Admittedly, I am somewhat skimming through thread posts since I am working right now.

2 Likes

Good idea. Looks like it would be easy to add an option to the fetch request. Something like {show_progress: false}

2 Likes

This is because wkof applies split_list() only to module lists, lists of end points and the built-in filters. Japanese commas are not involved there. I have checked the code.

The relevant split_list() calls are in the filter filter_value_map functions defined in the Item List filters. These calls are not performed by wkof.

function vocabulary_filter_map(filter_value) {vocabulary_list = split_list(filter_value);};

I will investigate your other suggestions.

You nailed it, congratulations! :+1: There were @require directives causing race conditions. Removing them solved the problems. Both console.log() and filtering. You were right on insisting to continue the troubleshooting. Thanks for your help.

This is working with the wkof with the old split_list(). You don’t need to update wkof, just the Item List filters will do.

There is an @require directive in the test copy of Self Study Quiz I supplied you. If you have it installed it may cause you grief in your tests.

2 Likes

@rfindley I am reporting on the work I have made for the html type setting. I have been successful beyond my wildest expectations. This will be long. I have a lot to report on.

I have made changes to the Settings.js module and to the css. I have posted the changes on github. For css I didn’t change any existing rule. I just appended new rules at the end of the file to style html type settings. More about this below.

I have prepared a test set to let you verify how what I did works. This will come handy to test changes. You are a much better programmer than I am. I expect that you will find things to improve on and disagree with some of what I did. This is part of the process. I will adapt to what you say.

The test set is on greasyfork.

The test set does two things. First it installs this.

testset

It will open a dialog where a series of tests for html type settings are displayed. I have done a lot of testing to make sure my proposal is complete, robust and doesn’t break anything.

The other thing is it install a series of test filters. I am interested in html types for developing filters so I tested that I can successfully write a filter.

I have updated the test version of Self-Study Quiz I have supplied you to support html type filters. You may use it to verify the test filters. It is on greasyfork.

As an aside I no longer intend to include all filters in @require directives. I made some research on how these directive work and I didn’t like what I have found. I will load the filters with wkof.load_script() instead. If you are interested I will provide you this code for Self-Study Quiz when it is ready.

Here is an account of what I have done.

Insertion of html types in the dialog

I have modified the code for inserting html type element to this. This is in the config_to_html(context) function.

            case 'html':
                    context.config_list[name] = item;
                    var content = item.content;
					for (cname in content){
                        context.config_list[cname] = content[cname];
                        populateDefaults(context, base, cname, content[cname]);
                    }
					html = make_label(item, item.hover_tip);
					html += wrap_right(item.html);
                    html = wrap_row_html(html, name, item.shade);

I have found that including the html in a standard (for the dialog) .row .left .rigth series of div works better than in-line insertion. It sets aside dialog screen estate in a manner that is consistent with the rest of wkof. It is a bit easier for the developer to craft html that doesn’t break the overall layout of the dialog. But the most important advantage is it creates a structure that css rules and Javascript code can rely on. There will be some examples below. The alternative would be to impose some standard on how the html is crafted, but if this is required it is better to automate it.

The function wrap_row_html() is like this:

function wrap_row_html(html, name, shade) {return '<div class="row full html_type'+(shade?' html_shade':'')+'" name="'+name+'">'+html+'</div>';}
  • The class html_type identifies the row as an html type setting. It is used for limiting the scope of css rules to html types.
  • The html_shade class is used to create an optional border around the inserted html and set the background color to make the whole html setting standout visually.
  • I force the row to be full width. I toyed with the idea of making it optional but I couldn’t find a html layout that makes sense in this configuration. This is not a real limitation. The crafter of the html can make a half width layout if they want.
  • The name attribute is defined in the row because Item Inspector and Self-Study quiz need to locate the name of the setting when initializing or toggling the enabling checkbox of a filter. They currently do this with row.find('.setting').but in an html element there is no guarantee that there is an element with class setting. A html filter may take no parameter and display only text. Storing the name in the row guarantees that there is a place to find it.

The make_label(item, item.hover_tip) has been modified to take an extra hover_tip parameter. We can place a hover_tip on the label of a html type setting. The new make_label() is functionally equivalent to the old one when the hover_tip parameter is absent.

The html insertion code also populates context.config_list with configuration subobjects. When testing tagging I have found that the code all over the place expects to find a configuration subobject for the item being processed in context.config_list.

This applies to the individual settings that comprise the html type element. A configuration subobject for the whole html type doesn’t do because each setting is tagged with a different name and context.config_list is accessed by this name…

I didn’t want to make the code work in absence of a configuration subobject. I preferred to supply the data. I require each html type to come with a context: subobject that contains the individual configuration for each setting and button that must be managed by the framework. This is the purpose of the for (cname in content) loop in the html insertion code. No html is generated from this. It just make the code happy. And it works.

I also add the configuration subobject of the whole html in context.config_list. This is needed to support pre_open callbacks for filters. There is in the test filters a real-life example of a filter that needs it. More about this later.

The configuration subobject may contain a default value for the subobject. The for loop initializes this as well. I have moved all the default initialization code of config_to_html(context) in a separate function named populateDefaults() and call this everywhere. This permits to call this code from the html insertion routine.

WKOF functionality in html type setting

The decision to require configuration subobjects has been hugely successful. Most of the time the individual setting is treated by the code the same (meaning the same code path) as if it were outside of an html type. This is because the data is in config_to_html(context) so the code doesn’t see a difference between html and non-html type settings.

The following features work in html type settings with no change in the code. It is sufficient to request them in the configuration.

  • on_change' call backs
  • on_click' call backs
  • refresh_on_change flags
  • path

I now realize that I forgot to test path outside the context of a filter. But I have tested it within filters and it works.

The following features are ignored if they are set .

  • label
  • hover_tip
  • placeholder
  • full_width

These options are implemented by generating the html accordingly. Since there is no html generation for html type these features are ignored. This is not a real limitation. The crafter of the html may incorporate them by hand in their html.

Built-in validation and the validate callback work too but there was a snag. The error message was displayed below the whole html type element instead of below the individual setting that triggered it. This is because the code issue a elem.closest('.right') call to locate the div below which the message should be displayed. In a html type this locates the .right that encloses the entire html. My solution is that the class msg should be used for this purpose. The validation code now issues a elem.closest('.msg'). I have modified wrap_right() that is responsible to create .right div to always include the msg class so the new code is functionally equivalent to the old when used outside of an html type. But within an html type the crafter has the option to enclose his setting in a div of class msg to position the error message at the correct place.

textarea

<textarea> types work great in this solution. Their configuration subobject uses type ‘others’ or ‘textarea’, it doesn’t matter which one. This falls under the default: clauses of the relevant switch statements where and'elem.val() is called. This works as you intended. I didn’t believe it would but I misread your code. I am sorry about that.

The following wkof features work for textarea. If suffices to request them in the configuration. The reason is the relevant code does not test for item.type. The code is executed regardless of the type.

  • validate callback (with the change mentioned above)
  • on-change callback
  • refesh_on_change
  • path (tested only in filters)

Tagging

Tagging the settings is required as you indicated. This process is error prone and debugging may require to place traces in the wkof Setting module. There is no error message. The dialog just fails to synchronize with the data in the settings without giving a clue about what goes wrong. Debugging this is a hassle.

There are rules to follow. Any departure from the rules result in the dialog not synching with the settings. Reverse engineering what the rules are was painful and tedious. We must document them somewhere. Debugging the tags is much easier when we know what we must look for.

The rules are:

Any dialog element that needs to synch with the settings and every button that needs to use some wkof feature must have:

  • The setting class set
  • A name defined in the name attribute that is unique throught the entire dialog.
  • The id attribute must have the form script_id_+'name' where the script_id is the same script_id that is given in the global dialog configuration and ‘name’ is the same name that was given in the name attribute.
  • Labels for attribute must be set to the id attribute of the input they correspond to.
  • In the case of a filter the'script_id must not contain an underscore. Item Inspector and Self Study Quiz have to html.replaceAll() this script_id with their own script_id when making their settings dialog to make this dialog compliant with the rules.

CSS

I have defined a number of classes for styling the html. It suffices to apply the class to an element and the relevant css is applied. I have found that without this crafters of the html would constantly reinvent the wheel. You will find a description of the classes and what they do in the HTML and CSS tests in both the Test Set dialog and in the test filters.

The classes provides the following:

  • A look and feel that is consistent with the styling of setting dialogs.
  • They make styling easy. it is just a game of applying the right classes.
  • They cover the most frequent layouts.
  • They are not mandatory. Crafters are free to add custom styling for special needs, or they may choose not to use them at all.
  • They resize well.
  • They handle the .narrow mode built into the current dialog styling.
  • All css selectors are guarded with a reference to the html_type class that is inserted in the top .row div of every html type. This ensures they don’t spill over unintended elements.
  • No modification to the existing css rules of the framework were made. Conflicts were resolved by making the html type rule more specific. This makes for rather long selectors.

Filters

Html type usually have more than one setting but filter related code assumes there is only one. This include wkof.itemData.get_items() that accepts a configuration object as input and passes the filter value to filter_func and filter_value_map. But wkof.itemData.get_items() is a passthrough. It doesn’t care about the filter value datatype. I chose to arrange the html type filters to be mapped with a path to the root of an object and all the individual settings are properties of this object. This enable to pass all the settings through wkof.itemData.get_items() without changing this code. For wkof.itemData.get_items() an object is a single data as is expected. It works.

In Item Inspector and Self-Study Quiz html type are handled with this code.

                   case 'html':
                        flt_content[src_name+'_flt_'+flt_name] = {
                            type:flt.type,
                            label:flt.label,
                            html:flt.html.replaceAll(/\sid=".*?_/g, ' id="'+scriptId+'_').replaceAll(/\sfor=".*?_/g, ' for="'+scriptId+'_'),
                            path:'@ipresets[@active_ipreset].content["'+src_name+'"].filters["'+flt_name+'"].value',
		                 	hover_tip:flt.hover_tip,
                            pre_open:flt.pre_open,
                            content:flt.content,
                        };
                        settings.filters[flt_name].value = {};
                        for (var cname in flt.content){
                            settings.filters[flt_name].value[cname] = flt.content[cname].default;
                            flt.content[cname].path = '@ipresets[@active_ipreset].content["'+src_name+'"].filters["'+flt_name+'"].value.'+cname;
                        };
                        break;

This is replicating the options in the registry into the dialog configuration subobject.

  • The paths must be tailored to handle presets. They are not set by the filter.
  • There is a path for the whole html which is used to send the settings to wkof.itemData.get_items().
  • There are individual paths for each setting that are used to synch the dialog and the setting.
  • Default values are gathered for initialization.
  • The html is replaceAll() to include the correct script_id in the tags to comply with tagging rules.

There were some changes to the Item Inspector and Self-Study Quiz css. These rules are meant to style non-full width one setting filters controled by a checkbox. They interfere with the styling of full width multiple settings html types. I have guarded these rule with negative selectors on the html_type class to remove the inteference.

I have added one css rule to force the label of a html type to stay on the same line as the checkbox and occupy the full width of the line.

I have written a real life filter that I wanted to do. The goal was to make sure I could actually use what I came up with. The filter is like this:

This is an Item List filter with a better user interface. The item lists are in textareas for ease of editing and users can save their lists in a file and read them back later. This enables users with many lists to swap them without having to retype them afresh.

One snag is that the filter has no knowledge of the path where the data is in the settings. This is determined by the sript that uses the filter. There is a workaround. Upon upload of a file I set the content of the textareas with an elem.val(list) call. Then I force synching with the setting with an elem.change(). The framework built-in onchange() handler does the rest.

Then there is the download. The button is actually a link with a data: url. I have on_change handlers that update the href attribute every time there is a change. This works excepts for one thing. When the dialog is first started the link is not initialized because there has been no call to an on_change handler yet. For this I need a pre-open call back.

I know there is a download API that I could use. I have rejected it because this is not supported in Safari.

It is not possible to use the global pre_open callback because the filter is not setting up the dialog. A filter specific callback should be supported by the framework, otherwise I am out of luck. The filter users will have to make a spurious edit to initialize the download.

My solution is to support a pre_open callback in the item configuration on top of the global one, I have introduced this code in the open(contect) function just after the regular pre_open call.

        for (var name in context.config_list) {
            var item = context.config_list[name];
            if (typeof item.pre_open === 'function'){
                var base = 'wkof.settings["'+context.cfg.script_id+'"]';
		        var path = (item.path || base+'["'+name+'"]');
		    	path = path.replace(/@/g, base+'.');
                var elem = $(dialog).find('div[name="'+name+'"]')[0];
                item.pre_open(elem, path);
            };
        };

This iterates over context.config_list to locate item specific pre_open callbacks and call them.

There is another snag. pre_open occurs before the dialog is refreshed. This is how it should be because we don’t know how pre_open will affect the settings. I can’t get the information from the dialog because it is not initialized. I need the path to get the information from the settings. This code fully evaluate the path and passes it as a parameter. Then I have all I need to initialize the download link.

This concludes this rather long explanation. I hope I have forgotten nothing.

to the rescue!

1 Like

I have a request for @rfindley @Kumirei @Saimin about filters. Could you please issue a wkof.set_state('yourScriptId', 'ready') call when you are done registering your filters? This way I can wait for the registration with wkof.wait_state('yourScriptId', 'ready'). @rfindley the wkof built-in filters are fine but I need this in Item Lists and Part Of Speech please.

The reason is I want to load filters programmatically at startup using wkof.load_script() based on a user setting. When I do this the promise is resolved when the script is loaded, not when the script is done executing. I end up with Item Inspector starting up before the filters are done registering and problems ensue.

I have tested with my own filters and wkof.set_state() / wkof.wait_state() seems to work fine.

@seanblue this request is not addressed to you because I don’t intend to load your script programmatically. There are many people having it installed in Tampermonkey. Loading it programmatically will cause it to be loaded twice way too often. But if you want to support wkof.set_state() / wkof.wait_state() anyway you are welcome.

1 Like

JLPT, Joyo, and Frequency Filters actually already had a ready flag, but I changed the flag key to JJFFilters. For visually similar kanji I had to add the flag, with the key visually_similar_kanji_filter.

1 Like

Thanks. I appreciate the help. :+1:

I needed to use the full strings 'wkof.Kumirei.JJFFilters' and 'wkof.Kumirei.VSKFilter' because this is what you use in your wkof.set_state() calls. This is fine. I no longer have startup problems with your scripts.

1 Like

I changed this one to just JJFFilters.

Lol, I didn’t even see this one, apparently I already had a flag in that filter as well. I added a new flag with the key I mentioned above, haha. I will be removing the new one and changing the old one to VSKFilter for symmetry

1 Like