[Userscript] ConfusionGuesser

Version 1.11 patch-notes:

Bugfix

Fixed a bug that happened when Guess only learned items was disabled, the spoiler warning feature was enabled, and the guess list contained an item that was not yet learned.


@Faun Sorry that I could not help with your problem. In a last attempt I have downloaded version 1.9 and 1.10 from Greasy Fork as files and compared them in a binary diffViewer to maybe uncover some character encoding problems, but I could not find anything. Maybe I should not have used surrogate pairs in my code, and these characters are causing problems only on your specific machine. But since the change from 1.9 to 1.10 didn’t even touch the code parts where I’ve used surrogate pairs, this also seems highly unlikely. If you still want to try, here is a version where I have removed the surrogate pairs (which causes some minor functionality loss):

Code
// ==UserScript==
// @name         ConfusionGuesser
// @namespace    confusionguesser
// @version      1.10
// @description  If you give a wrong answer during reviews, it tries to guess which other WaniKani item you confused it with.
// @author       Sinyaven
// @match        https://www.wanikani.com/review/session
// @match        https://preview.wanikani.com/review/session
// @grant        none
// ==/UserScript==

(async function() {
	"use strict";
	/* global $, wkof */
    /* eslint no-multi-spaces: "off" */

	if (!window.wkof) {
		alert("ConfusionGuesser script requires Wanikani Open Framework.\nYou will now be forwarded to installation instructions.");
		window.location.href = "https://community.wanikani.com/t/instructions-installing-wanikani-open-framework/28549";
		return;
	}

	wkof.include("ItemData,Menu,Settings");
	await wkof.ready("document,ItemData,Menu,Settings");

	const PROBABILITY_SCALING_RENDAKU = 0.4;
	const PROBABILITY_SCALING_GEMINATION = 0.4;
	const MIN_KANJI_SIMILARITY = 0.1;
	const LEVENSHTEIN_TRANSPOSITION_COST = 0.3;

	const defaultColors = {
		vissimColor: "#FFD700",
		ononColor: "#00AAFF",
		onkunColor: "#00AAFF",
		kunonColor: "#5571E2",
		kunkunColor: "#5571E2",
		naonColor: "#FFA500",
		nakunColor: "#FFA500",
		onnaColor: "#FFA500",
		kunnaColor: "#FFA500",
		nanaColor: "#FFA500",
		specialColor: "#555555",
		textColor: "#FFFFFF"
	}

	const idsEqualities = {
		囗: "口",
		厶: "ム",
		亻: "イ"
	}

	// approximate mappings
	const radicalIdToChar = {
		8761: "丨",  // stick
		8762: "A",  // gun
		8763: "丆",  // leaf
		8764: "人",  // hat === person
		8765: "⺌",  // triceratops
		8766: "丂",  // beggar
		8767: "丷",  // horns
		8788: "丷",  // explosion === horns
		8768: "业",  // spikes
		8770: "A",  // kick
		8771: "之",  // hills
		8772: "爫",  // cleat
		8773: "A",  // pope, only in 盾
		8774: "A",  // chinese
		8775: "龷",  // blackjack
		8776: "A",  // trash
		8777: "A",  // bear, 㠯 is only in 官
		8779: "A",  // spring
		8780: "A",  // cape
		8781: "A",  // creeper
		8782: "㦮",  // bar
		8784: "袁",  // zombie
		8785: "㑒",  // squid
		8787: "廿",  // yurt, more distinctive part
		8790: "俞",  // death-star
		8793: "A",  // morning
		8794: "丞",  // coral
		8797: "鬯",  // psychopath, more distinctive part
		8798: "A",  // satellite, more distinctive part
		8799: "耳",  // elf === ear
		8819: "龹",  // gladiator
		8783: "A",  // grass
		8769: "A",  // viking === grass
//		8778: "④",  // tofu
//		8792: "⑤",  // comb
//		8796: "⑥",  // cactus
	}

	const jokeMeaningFilter = {
		3237: ["The Answer"],
		5633: ["Nic Cage"]
	}

	const typeDescription = {
		onyomionyomi     : "Wrong on'yomi",
		onyomikunyomi    : "Used on'yomi but needed kun'yomi",
		kunyomionyomi    : "Used kun'yomi but needed on'yomi",
		kunyomikunyomi   : "Wrong kun'yomi",
		nanorionyomi     : "Used nanori but needed on'yomi",
		nanorikunyomi    : "Used nanori but needed kun'yomi",
		onyominanori     : "Used on'yomi but needed nanori",
		kunyominanori    : "Used kun'yomi but needed nanori",
		nanorinanori     : "Wrong nanori",
		onyomiundefined  : "Used on'yomi but needed special reading",
		kunyomiundefined : "Used kun'yomi but needed special reading",
		nanoriundefined  : "Used nanori but needed special reading",
		visuallySimilar  : "Visually similar"
	}

	let similarityCache = {};
	let oldSettings = {};

	// variables that will be initialized by settingsChanged()
	let idsHash = {};
	let byReading = {};
	let byCharacters = {};
	let byMeaning = {};
	let srs = [];

	// inject the overlay into the DOM
	let fFooter = document.querySelector("#reviews footer");
	let dOverlay = document.createElement("div");
	let dCollapsible = document.createElement("div");
	dCollapsible.classList.add("collapsed");
	dOverlay.id = "confusionGuesserOverlay";
	dOverlay.appendChild(dCollapsible);
	fFooter.parentElement.insertBefore(dOverlay, fFooter);

	let iShowLess = null;
	document.addEventListener("keydown", ev => {
		if (ev.target.nodeName === "INPUT" || dCollapsible.classList.contains("collapsed")) return;
		if (iShowLess && ev.key.toLowerCase() === wkof.settings.confusionguesser.hotkeyExpand && !ev.ctrlKey && !ev.shiftKey && !ev.altKey && !ev.metaKey) iShowLess.checked = !iShowLess.checked;
	});

	// add entry to hotkey list
	let tHotkeyEntry = document.createElement("tr");
	let sHotkeyKey = document.createElement("span");
	let tHotkeyKeyDisplay = document.createElement("td");
	let tHotkeyDescription = document.createElement("td");
	tHotkeyDescription.innerText = "Expand/collapse guesses";
	tHotkeyKeyDisplay.appendChild(sHotkeyKey);
	tHotkeyEntry.appendChild(tHotkeyKeyDisplay);
	tHotkeyEntry.appendChild(tHotkeyDescription);
	document.querySelector("#hotkeys tbody").appendChild(tHotkeyEntry);

	installCss();
	setupMenu();

	// create observer to react to wrong answers
	let iUserInput = document.getElementById("user-response");
	let fObserverTarget = document.querySelector("#answer-form fieldset");
	let observer = new MutationObserver(m => m.forEach(handleMutation));
	observer.observe(fObserverTarget, {attributes: true, attributeFilter: ["class"]});

	function handleMutation(mutation) {
		if (mutation.target.classList.contains("incorrect")) {
			showGuesses();
		} else {
			hideGuesses();
		}
	}

	async function showGuesses() {
		// clear cache
		similarityCache = {};

		let guesses = [];
		let currentItem = $.jStorage.get("currentItem");
		let expectedReading = currentItem.emph === "onyomi" ? currentItem.on : (currentItem.emph === "kunyomi" ? currentItem.kun : (currentItem.nanori || currentItem.kana));
		let question = radicalIdToChar[currentItem.id] || currentItem.rad || currentItem.kan || currentItem.voc;

		// ensure that idsFile finished loading
		idsHash = await idsHash;

		switch ($.jStorage.get("questionType")) {
			case "reading": guesses = guessForReading(iUserInput.value, expectedReading, question); break;
			case "meaning": guesses = guessForMeaning(iUserInput.value, question); break;
		}

		// for each item-reading combination find the guess with the highest rating
		let itemIdBestGuess = {};
		guesses.forEach((g, i) => {let id = g.item.id + (g.reading || ""); itemIdBestGuess[id] = (itemIdBestGuess[id] !== undefined && g.probability <= guesses[itemIdBestGuess[id]].probability) ? itemIdBestGuess[id] : i});
		//  pass on "show" property to this item's guess with the highest rating
		guesses.forEach(g => {if (g.show) guesses[itemIdBestGuess[g.item.id + (g.reading || "")]].show = true});
		// remove duplicate guesses suggesting the same WK item with the same reading (only keep the guess with highest probability)
		guesses = guesses.filter((g, i) => itemIdBestGuess[g.item.id + (g.reading || "")] === i);
		// if the list is long, remove guesses with probability 0
		// remove the guesses for kanji that the user got correct
		guesses = guesses.filter(g => g.type !== "correct");
		// sort descending by probability; also put guesses that are to be shown to the front
		guesses.sort((g1, g2) => g1.show === g2.show ? g2.probability - g1.probability : (g1.show ? -1 : 1));

		// remove old guesses
		while (dCollapsible.firstChild) {
			dCollapsible.removeChild(dCollapsible.firstChild);
		}
		if (guesses.length === 0) return;
		// display new guesses
		let dGuesses = document.createElement("div");
		if (guesses.some(g => !g.show)) addShowMoreOption(dCollapsible);
		guesses.forEach(g => { if (isSpoiler(g)) dGuesses.appendChild(spoilerButton(g)); dGuesses.appendChild(guessToDiv(g)); });
		dGuesses.id = "guesses";
		dCollapsible.appendChild(dGuesses);
		dCollapsible.classList.toggle("collapsed", false);
	}

	function hideGuesses() {
		dCollapsible.classList.toggle("collapsed", true);
	}

	function guessForMeaning(answer, question) {
		answer = answer.toLowerCase();
		let result = searchByMeaning(answer).map(item => ({type: "visuallySimilar", item, probability: rateSimilarity(item, question)}));
		// for each guess, select the meaning that is closest to the answer, and also adapt its probability by the meaning similarity
		result.forEach(r => {let best = meaningsOfItem(r.item).reduce((best, m) => {let rating = rateSimilarity(m.toLowerCase(), answer); return rating > best.rating ? {rating, meaning: m} : best}, {rating: -1}); r.meaning = best.meaning; r.probability *= best.rating});
		result.sort((a, b) => b.probability - a.probability);
		if (result.length > 0) result[0].show = true;
		return result;
	}

	function guessForReading(answer, expected, question) {
		expected = ensureArray(expected);
		let guesses = expected.flatMap(e => guessForReading_splitByOkurigana(answer, e, question));
		return guesses.concat(guessForReading_wholeVocab(answer, question));
	}

	function guessForReading_wholeVocab(answer, question) {
		let result = searchByReading(answer).filter(item => item.object === "vocabulary").map(item => ({type: "visuallySimilar", item, reading: answer, probability: rateSimilarity(item, question)}));
		result.sort((a, b) => b.probability - a.probability);
		if (result.length > 0) result[0].show = true;
		return result;
	}

	function guessForReading_splitByOkurigana(answer, expected, question) {
		question = replaceDecimalWithJapanese(question);
		let parts = question.split(/([\u301c\u3041-\u309f\u30a0-\u30ff]+)/);
		if (parts.length === 1) return guessForReading_splitByKanji(answer, expected, question);

		// if question contains okurigana, separate at these positions using regexp (assumption: can be done unambiguously - seems to work for every WK vocab's correct reading)
		let regex = new RegExp("^" + parts.map((p, idx) => idx & 1 ? p : (p ? "(.+)" : "()")).join("") + "$");
		answer = answer.match(regex);
		expected = expected.match(regex);
		// if okurigana does not match answer return no guesses
		if (!answer) return [];

		answer.shift();
		expected.shift();
		return answer.flatMap((a, idx) => guessForReading_splitByKanji(a, expected[idx], parts[idx * 2]));
	}

	function guessForReading_splitByKanji(answer, expected, question) {
		if (!question) return [];
		question = question.replace(/(.)々/g, "$1$1");
		let splitAnswer = possibleSplits(answer, question.length);
		let splitExpected = possibleSplits(expected, question.length);
		// try to split the expected reading to on/kun of each kanji; if not successful then return no guesses
		let kanjiReadings = splitExpected.filter(s => s.every((part, idx) => validReading(question[idx], part))).pop();
		// if it was not possible to validate the reading for every kanji, try to find a split where at least every second kanji has a validated reading
		kanjiReadings = kanjiReadings || splitExpected.filter(s => s.reduce((state, part, idx) => state === 2 ? 2 : (validReading(question[idx], part) ? 0 : ++state), 0) !== 2).pop();
		// if there was still no solution found, give up
		if (!kanjiReadings) return [];

		// for each split (of the answer entered by the user), guess for each part => array of arrays of arrays
		let guesses = splitAnswer.map(s => s.map((part, idx) => guessForReading_singleKanji(part, kanjiReadings[idx], question[idx])));
		// remove splits where at least one part resulted in no guesses
		guesses = guesses.filter(forSplit => forSplit.every(forPart => forPart.length > 0));
		// calculate probability of each guess
		guesses.forEach(forSplit => forSplit.forEach((forPart, idx) => forPart.forEach(guess => {guess.probability *= calculateGuessProbability(guess, question[idx])})));
		// sort guesses for each part descending by probability
		guesses.forEach(forSplit => forSplit.forEach(forPart => forPart.sort((a, b) => b.probability - a.probability)));
		// calculate probability of each split by multiplying the highest probability of each part together
		let splitProbability = guesses.map(forSplit => forSplit.reduce((totalProb, forPart) => totalProb * forPart.reduce((highestProb, guess) => Math.max(highestProb, guess.probability), 0), 1));
		// scale the probability of each guess with the highest probabilities for the other parts in the split (overall split probability, but instead of the best guess for the current part, take the current guess)
		guesses.forEach((forSplit, idx) => forSplit.forEach(forPart => forPart.forEach((guess, i) => {if (i !== 0) guess.probability *= splitProbability[idx] / forPart[0].probability})));
		guesses.forEach((forSplit, idx) => forSplit.forEach(forPart => {forPart[0].probability = splitProbability[idx]}));
		// find the split with the highest probability
		let idx = splitProbability.reduce((best, probability, idx) => probability <= best.probability ? best : {probability, idx}, {probability: 0}).idx;
		// mark the best guess (sorted, therefore at index 0) for each part of the best split with "show"
		(guesses[idx] || []).forEach(forPart => {forPart[0].show = true});
		return guesses.flat(2);
	}

	function guessForReading_singleKanji(answer, expected, question) {
		let reading = answer;
		if (answer === expected) return [{type: "correct", item: getItem(question), reading, probability: 1}];

		let                                  guesses =                searchByReading(reading).filter(s => s.object === "kanji").map(item => ({type: "visuallySimilar", item, reading, probability: 1}));
		reading = removeRendaku1(answer);    guesses = guesses.concat(searchByReading(reading).filter(s => s.object === "kanji").map(item => ({type: "visuallySimilar", item, reading, probability: PROBABILITY_SCALING_RENDAKU})));
		reading = removeRendaku2(answer);    guesses = guesses.concat(searchByReading(reading).filter(s => s.object === "kanji").map(item => ({type: "visuallySimilar", item, reading, probability: PROBABILITY_SCALING_RENDAKU})));
		reading = removeGemination1(answer); guesses = guesses.concat(searchByReading(reading).filter(s => s.object === "kanji").map(item => ({type: "visuallySimilar", item, reading, probability: PROBABILITY_SCALING_GEMINATION})));
		reading = removeGemination2(answer); guesses = guesses.concat(searchByReading(reading).filter(s => s.object === "kanji").map(item => ({type: "visuallySimilar", item, reading, probability: PROBABILITY_SCALING_GEMINATION})));
		reading = removeGemination3(answer); guesses = guesses.concat(searchByReading(reading).filter(s => s.object === "kanji").map(item => ({type: "visuallySimilar", item, reading, probability: PROBABILITY_SCALING_GEMINATION})));
		reading = removeGemination4(answer); guesses = guesses.concat(searchByReading(reading).filter(s => s.object === "kanji").map(item => ({type: "visuallySimilar", item, reading, probability: PROBABILITY_SCALING_GEMINATION})));
		// change the type to onyomionyomi, onyomikunyomi etc. where applicable
		guesses.filter(g => g.item.data.characters === question).forEach(g => {g.type = validReading(question, answer) + validReading(question, expected)});
		return guesses;
	}

	function calculateGuessProbability(guess, question) {
		switch (guess.type) {
			case "visuallySimilar": return rateSimilarity(guess.item, question);
			case "correct":         return 1;
			default:                return 1; // onyomikunyomi etc.
		}
	}

	function getItem(characters, preferred = ["kanji", "vocabulary", "radical"]) {
		return (byCharacters[characters] || []).reduce((result, item) => result ? (preferred.indexOf(item.object) < preferred.indexOf(result.object) ? item : result) : item, undefined);
	}

	function ensureArray(listOrEntry) {
		return Array.isArray(listOrEntry) ? listOrEntry : (listOrEntry === undefined ? [] : [listOrEntry]);
	}

	function searchByReading(reading) {
		return byReading[reading] || [];
	}

	function searchByMeaning(meaning) {
		return byMeaning.get(meaning);
	}

	function possibleSplits(reading, partCount) {
		if (partCount === 1) return [[reading]];

		let result = Array.from({length: reading.length - partCount + 1}, (val, idx) => ({start: reading.substr(0, idx + 1), end: reading.substr(idx + 1)}));
		result = result.flatMap(r => possibleSplits(r.end, partCount - 1).map(s => [r.start].concat(s)));
		result = result.filter(r => r.every(part => !"ぁぃぅぇぉっゃゅょゎん".includes(part[0])));
		return result;
	}

	function validReading(kanji, reading) {
		let item = getItem(kanji);
		return item.data.readings.reduce((type, r) => type || (r.reading === reading || applyRendaku1(r.reading) === reading || applyRendaku2(r.reading) === reading || applyGemination(r.reading) === reading ? r.type : undefined), undefined);
	}

	function applyRendaku1(reading) {
		let idx = "かきくけこさしすせそたちつてとはひふへほ".indexOf(reading[0]);
		return idx >= 0 ? "がぎぐげござじずぜぞだぢづでどばびぶべぼ"[idx] + reading.substr(1) : undefined;
	}

	function applyRendaku2(reading) {
		let idx = "はひふへほち".indexOf(reading[0]);
		return idx >= 0 ? "ぱぴぷぺぽじ"[idx] + reading.substr(1) : undefined;
	}

	function applyGemination(reading) {
		// definitely not based on any grammar rules, but it works good enough for WK vocab
		let replacementGemination = "ちつくき".includes(reading.substr(-1));
		return replacementGemination ? reading.substr(0, reading.length - 1) + "っ" : reading + "っ";
	}

	function removeRendaku1(reading) {
		let idx = "がぎぐげござじずぜぞだぢづでどばびぶべぼぱぴぷぺぽ".indexOf(reading[0]);
		return idx >= 0 ? "かきくけこさしすせそたちつてとはひふへほはひふへほ"[idx] + reading.substr(1) : undefined;
	}

	function removeRendaku2(reading) {
		return reading[0] === "じ" ? "ち" + reading.substr(1) : undefined;
	}

	function removeGemination1(reading) {
		return reading.substr(-1) === "っ" ? reading.substr(0, reading.length - 1) : undefined;
	}

	function removeGemination2(reading) {
		return reading.substr(-1) === "っ" ? reading.substr(0, reading.length - 1) + "ち" : undefined;
	}

	function removeGemination3(reading) {
		return reading.substr(-1) === "っ" ? reading.substr(0, reading.length - 1) + "つ" : undefined;
	}

	function removeGemination4(reading) {
		return reading.substr(-1) === "っ" ? reading.substr(0, reading.length - 1) + "く" : undefined;
	}

	// where is removeGemination5() with き? I don't know. I'm also dumbfounded.
	// (dumbfounded 呆気 seems to be the only WK vocab which turns き into っ, so whatever)

	// ---IDEOGRAPHIC DESCRIPTION SEQUENCE STUFF--- //

	// turns IDS into tree structure
	function getKanjiComponents(kanji) {
		let line = idsHash[kanji];
		if (!line) return [kanji];
		// get all decomposition variations that apply to the Japanese character appearance (I think that's what [J] in ids.txt means)
		let variations = line.split("\t").filter(v => !v.match(/\[[^J\]]+\]/));
		if (variations.length === 0) variations = [line.split("\t")[0]]; // fix for lines such as "U+225BB	A	⿱心夂[G]	⿱心夊[T]" - TODO: research what the letters in [] mean
		return variations.map(v => v === kanji ? (idsEqualities[v] || v) : parseIds(v[Symbol.iterator]()));
	}

	function parseIds(iter) {
		let idc = iter.next().value;
		if (!"⿰⿱⿲⿳⿴⿵⿶⿷⿸⿹⿺⿻".includes(idc)) return getKanjiComponents(idc)[0];
		let node = {idc, parts: []};
		node.parts[0] = parseIds(iter);
		node.parts[1] = parseIds(iter);
		if (!"⿲⿳".includes(idc)) return node;
		node.parts[2] = parseIds(iter);
		return node;
	}

	// turns tree structure to path list
	function componentTreeToPathList(components) {
		return components.parts ? components.parts.flatMap((part, partIdx) => componentTreeToPathList(part).map(path => components.idc + partIdx + path)) : [components];
	}

	function ratePathSimilarity(path1, path2) {
		let array1 = path1.match(/[⿰-⿻][0-9]/g) || [];
		let array2 = path2.match(/[⿰-⿻][0-9]/g) || [];
		if (array1.length === 0 && array2.length === 0) return 1;
		let dist = levenshteinDistance(array1, array2, (node1, node2) => node1[0] !== node2[0] ? 1 : (node1[1] === node2[1] ? 0 : 0.5));
		// both paths lead to the same component, therefore they are always a little similar => array.length + 1
		return (1 - dist / (Math.max(array1.length, array2.length) + 1)) / array1.reduce((total, a) => total * ("⿲⿳".includes(a[0]) ? 3 : 2), 1);
	}

	function rateComponentSimilarity(components1, components2) {
		let pathList1 = componentTreeToPathList(components1);
		let pathList2 = componentTreeToPathList(components2);
		let paths1 = {};
		let paths2 = {};
		// using Array.from(p).slice(-1) instead of e.g. p.substr(-1) to maintain surrogate pairs
		pathList1.forEach(p => {let char = Array.from(p).slice(-1)[0]; paths1[char] = (paths1[char] || []).concat([p])});
		pathList2.forEach(p => {let char = Array.from(p).slice(-1)[0]; paths2[char] = (paths2[char] || []).concat([p])});
		let similarity = pathList1.reduce((total, p1) => total + (paths2[Array.from(p1).slice(-1)] || []).reduce((best, p2) => Math.max(best, ratePathSimilarity(p1, p2)), 0), 0);
		similarity    += pathList2.reduce((total, p2) => total + (paths1[Array.from(p2).slice(-1)] || []).reduce((best, p1) => Math.max(best, ratePathSimilarity(p2, p1)), 0), 0);
		return similarity / 2;
	}

	function rateKanjiSimilarityUsingIds(kanji1, kanji2) {
		let components1 = getKanjiComponents(kanji1);
		let components2 = getKanjiComponents(kanji2);
		// choose the pair with the best similarity rating
		return components1.flatMap(c1 => components2.map(c2 => [c1, c2])).reduce((best, pair) => Math.max(best, rateComponentSimilarity(pair[0], pair[1])), 0);
	}

	function rateSimilarity(itemOrText1, itemOrText2) {
		if (itemOrText1 === itemOrText2) return 1;
		let text1 = typeof itemOrText1 === "string" ? itemOrText1 : (itemOrText1.data.characters || radicalIdToChar[itemOrText1.id]);
		let text2 = typeof itemOrText2 === "string" ? itemOrText2 : (itemOrText2.data.characters || radicalIdToChar[itemOrText2.id]);
		if (!text1 || !text2) return 0;
		let cacheId1 = text1 + "§" + text2;
		let cacheId2 = text2 + "§" + text1;
		if (similarityCache[cacheId1]) return similarityCache[cacheId1];
		if (similarityCache[cacheId2]) return similarityCache[cacheId2];
		let chars1 = Array.from(text1);
		let chars2 = Array.from(text2);

		if (chars1.length > 1 || chars2.length > 1) {
			let dist = levenshteinDistance(chars1, chars2, (char1, char2) => 1 - rateSimilarity(char1, char2));
			// normalize distance to [0;1] and invert it so that 0 is complete mismatch
			return similarityCache[cacheId1] = 1 - dist / Math.max(chars1.length, chars2.length);
		}

		let item1 = typeof itemOrText1 === "string" ? getItem(itemOrText1) : itemOrText1;
		let item2 = typeof itemOrText2 === "string" ? getItem(itemOrText2) : itemOrText2;

		let similarity = item1 && item2 ? MIN_KANJI_SIMILARITY : 0;
		if (wkof.settings.confusionguesser.useWkRadicals && item1 && item2) {
			let radicals1 = item1.data.component_subject_ids || [item1.id];
			let radicals2 = item2.data.component_subject_ids || [item2.id];
			let matchCount = radicals1.reduce((matchCount, radicalId) => matchCount + (radicals2.includes(radicalId) ? 1 : 0), 0);
			similarity = Math.max(2 * matchCount / (radicals1.length + radicals2.length), similarity);
		}
		if (wkof.settings.confusionguesser.useWkSimilarity && item1 && item2) {
			if ((item1.data.visually_similar_subject_ids || []).includes(item2.id) || (item2.data.visually_similar_subject_ids || []).includes(item1.id)) similarity = Math.max(similarity, 0.5);
		}
		if (wkof.settings.confusionguesser.useIds) {
			similarity = Math.max(similarity, rateKanjiSimilarityUsingIds(text1, text2));
		}
		return similarityCache[cacheId1] = similarity;
	}

	// levenshtein distance with restricted transposition of two adjacent characters (so in fact it's not levenshtein distance but optimal string alignment distance)
	function levenshteinDistance(array1, array2, elementDistanceFunction = (e1, e2) => e1 === e2 ? 0 : 1) {
		// initialize distance matrix
		let d = Array.from(new Array(array1.length + 1), (val, i) => Array.from(new Array(array2.length + 1), (v, j) => i === 0 ? j : (j === 0 ? i : 0)));
		// fill distance matrix from top left to bottom right
		d.forEach((row, i) => {if (i > 0) row.forEach((val, j) => {if (j > 0) row[j] = Math.min(d[i - 1][j] + 1, d[i][j - 1] + 1, d[i - 1][j - 1] + elementDistanceFunction(array1[i - 1], array2[j - 1]), i > 1 && j > 1 ? d[i - 2][j - 2] + LEVENSHTEIN_TRANSPOSITION_COST + elementDistanceFunction(array1[i - 2], array2[j - 1]) + elementDistanceFunction(array1[i - 1], array2[j - 2]) : Number.MAX_SAFE_INTEGER)})});
		// result is in the bottom right of the matrix
		return d[array1.length][array2.length];
	}

	// ---DECIMAL TO JAPANESE STUFF--- //

	function replaceDecimalWithJapanese(text) {
		let parts = text.split(/([0123456789]+)/g);
		return parts.map((p, i) => i & 1 ? decimalToJapanese(p) : p).join("");
	}

	function decimalToJapanese(decimal, zero = "零") {
		let groups = Array.from(decimal).reverse().reduce((result, digit, idx) => {result[Math.floor(idx / 4)] = (result[Math.floor(idx / 4)] || []).concat([digit]); return result;}, []);
		let japanese = groups.reduce((result, group, i) => { group = decimalToJapanese_4block(group); return !group ? result : (group + " 万億兆"[i] + result); }, "").replace(" ", "");
		return japanese || zero;
	}

	function decimalToJapanese_4block(array) {
		return array.reduce((result, digit, i) => { digit = decimalToJapanese_digit(digit); return !digit ? result : ((i > 0 && digit === "一" ? "" : digit) + " 十百千"[i] + result); }, "").replace(" ", "");
	}

	function decimalToJapanese_digit(digit) {
		return " 一二三四五六七八九"["0123456789".indexOf(digit)].replace(" ", "");
	}

	async function loadIdsHash() {
		if (wkof.file_cache.dir.ideographicDescriptionSequences) return wkof.file_cache.load("ideographicDescriptionSequences");

		let result = {};
		let idsFile = await wkof.load_file("https://raw.githubusercontent.com/cjkvi/cjkvi-ids/master/ids.txt");
		let lines = idsFile.matchAll(/U\+\S+\t(\S+)\t(.+)/g);
		for (let line of lines) {
			result[line[1]] = line[2];
		}
		wkof.file_cache.save("ideographicDescriptionSequences", result);
		return result;
	}

	// ---FUZZY SEARCH STUFF (N-GRAM)--- //

	function histogram(array) {
		return [...array.reduce((result, element) => result.set(element, (result.get(element) || 0) + 1), new Map())];
	}

	function highestBins(histogram, nrOfBins, minBinHeight) {
		return histogram.sort((a, b) => b[1] - a[1]).filter((h, idx) => idx < nrOfBins && h[1] >= minBinHeight);
	}

	function hashGrams(wkItems, gramSizes = [4, 3]) {
		let result = {};
		wkItems.forEach(item => meaningsOfItem(item).forEach((m, mNr) => {let entry = {item, mNr}; toGrams(m, gramSizes).flat().forEach(g => {result[g] = (result[g] || []).concat(entry)})}));
		result.get = (text, gramSizes = [4, 3]) => gramSizes.reduce((prevResult, size) => prevResult.length > 0 ? prevResult : highestBins(histogram(toGrams(text, [size])[0].flatMap(gram => result[gram] || [])), 10, text.length / 2).map(h => h[0].item), []);
		return result;
	}

	function toGrams(text, gramSizes = [4, 3]) {
		text = "§" + text.toLowerCase() + "§";
		return gramSizes.map(size => Array.from(new Array(text.length - size + 1), (val, idx) => text.substr(idx, size)));
	}

	function meaningsOfItem(item) {
		let meanings = jokeMeaningFilter[item.id] ? item.data.meanings.filter(m => !jokeMeaningFilter[item.id].includes(m.meaning)) : item.data.meanings;
		return meanings.concat(wkof.settings.confusionguesser.includeWhitelist ? item.data.auxiliary_meanings.filter(m => m.type === "whitelist") : []).map(m => m.meaning).concat(wkof.settings.confusionguesser.includeUserSynonyms && item.study_materials ? item.study_materials.meaning_synonyms : []);
	}

	// ---DOM STUFF--- //

	function addShowMoreOption(div) {
		iShowLess = document.createElement("input");
		let lShowMore = document.createElement("label");
		let lShowLess = document.createElement("label");
		iShowLess.id = "showLess";
		iShowLess.type = "checkbox";
		iShowLess.checked = !wkof.settings.confusionguesser.showAllByDefault;
		lShowMore.htmlFor = "showLess";
		lShowLess.htmlFor = "showLess";
		lShowMore.innerText = "+";
		lShowLess.innerText = "-";
		div.appendChild(lShowMore);
		div.appendChild(iShowLess);
		div.appendChild(lShowLess);
	}

	function guessToDiv(guess) {
		let a = document.createElement("a");
		let sJapanese = document.createElement("span");
		let sEnglish = document.createElement("span");
		let sProbability = document.createElement("span");
		let rJapanese = document.createElement("ruby");
		a.href = guess.item.data.document_url;
		a.target = "_blank";
		a.title = typeDescription[guess.type];
		a.classList.add(guess.type);
		a.classList.add(guess.item.object);
		if (guess.show) a.classList.add("show");
		rJapanese.lang = "ja-JP";
		rJapanese.innerText = guess.item.data.characters || "";
		sEnglish.innerText = guess.meaning || guess.item.data.meanings[0].meaning;
		sProbability.innerText = guess.probability.toFixed(2);

		if (!guess.item.data.characters) {
			appendSvgChild(guess.item.data.character_images, rJapanese);
		}
		if (guess.item.data.readings) {
			let rFurigana = document.createElement("rt");
			rFurigana.innerText = guess.reading || guess.item.data.readings[0].reading;
			rJapanese.appendChild(rFurigana);
		}

		sJapanese.appendChild(rJapanese);
		a.appendChild(sJapanese);
		a.appendChild(sEnglish);
		a.appendChild(sProbability);
		return a;
	}

	async function appendSvgChild(character_images, element) {
		try {
			element.innerHTML += await wkof.load_file(character_images.filter(c => c.content_type === "image/svg+xml" && !c.metadata.inline_styles)[0].url);
		} catch(e) {
			let iSvg = document.createElement("img");
			iSvg.src = character_images.filter(c => c.content_type === "image/svg+xml" && c.metadata.inline_styles)[0].url;
			element.appendChild(iSvg);
		}
	}

	function spoilerButton(guess) {
		let b = document.createElement("button");
		b.innerText = "Show spoiler";
		if (guess.show) b.classList.add("show");
		b.addEventListener("click", (e) => e.target.parentElement.removeChild(e.target));
		return b;
	}

	function isSpoiler(guess) {
		if (wkof.settings.confusionguesser.spoilerHandling === "none" || !guess.item.assignments.available_at) return false;
		let nextReview = new Date(guess.item.assignments.available_at);
		let remainingTimeInMs = nextReview - new Date();
		let stage = srs[guess.item.data.spaced_repetition_system_id].data.stages[guess.item.assignments.srs_stage];
		if (!stage) return false; // should not happen
		return remainingTimeInMs < srsStageIntervalInMs(stage) * parseInt(wkof.settings.confusionguesser.spoilerHandling) / 100;
	}

	function srsStageIntervalInMs(stage) {
		let result = stage.interval;
		if (stage.interval_unit === "milliseconds") return result; result *= 1000;
		if (stage.interval_unit ===      "seconds") return result; result *= 60;
		if (stage.interval_unit ===      "minutes") return result; result *= 60;
		if (stage.interval_unit ===        "hours") return result; result *= 24;
		if (stage.interval_unit ===         "days") return result; result *= 7;
		if (stage.interval_unit ===        "weeks") return result;
		return null;
	}

	function fromButtonColor() {
		let button = document.querySelector("#option-kana-chart > span > i");
		let textColor = button ? getComputedStyle(button).color : "rgb(0, 0, 0)";
		let color = "rgb(255, 255, 255)";
		while (button && getComputedStyle(button).backgroundColor.match(/rgba\(.*0\)/)) button = button.parentElement;
		color = rbgToHex(button ? getComputedStyle(button).backgroundColor : color);
		let result = Object.assign({}, defaultColors);
		Object.keys(result).forEach(k => result[k] = color);
		result.textColor = rbgToHex(textColor);
		return result;
	}

	function rbgToHex(rgbString) {
		let rgb = rgbString.match(/\( *([^,]*), *([^,]*), *([^,\)]*)/);
		return rgb.slice(1, 4).reduce((result, c) => result + parseInt(c).toString(16).padStart(2, "0"), "#");
	}

	// ---SETTINGS STUFF--- //

	async function settingsChanged(settings) {
		// find changed settings
		let changes = Object.keys(settings).filter(key => oldSettings[key] !== settings[key]);
		let refetchItems = false;

		changes.forEach(key => {
			switch(key) {
				case "useIds":
					idsHash = settings.useIds ? loadIdsHash() : {};
					if (!settings.useIds) wkof.file_cache.delete("ideographicDescriptionSequences");
					return;
				case "spoilerHandling":
					if (oldSettings.guessOnlyLearnedItems || oldSettings.spoilerHandling !== "none") return;
				case "guessOnlyLearnedItems":
				case "useFuzzySearch":
				case "includeWhitelist":
				case "includeUserSynonyms":
					refetchItems = true;
					return;
				case "showAsOverlay":
					dOverlay.classList.toggle("noOverlay", !settings.showAsOverlay);
					return;
				case "showTypes":
					dOverlay.classList.toggle("hideTypes", !settings.showTypes);
					return;
				case "showRatings":
					dOverlay.classList.toggle("hideRatings", !settings.showRatings);
					return;
				case "highContrast":
					dCollapsible.classList.toggle("highContrast", settings.highContrast);
					return;
				case "fontSize":
					dOverlay.style.setProperty("font-size", settings[key]);
					return;
				case "hotkeyExpand":
					tHotkeyEntry.classList.toggle("disabled", settings.hotkeyExpand === "");
					sHotkeyKey.innerText = settings.hotkeyExpand.toUpperCase();
					return;
				default:
					if (key.endsWith("Color")) dOverlay.style.setProperty("--" + key.substr(0, key.length - 5), settings[key]);
					return;
			}
		});

		if (refetchItems) {
			let config = {wk_items: {options: {study_materials: settings.includeUserSynonyms, assignments: settings.spoilerHandling !== "none"}}};
			if (settings.guessOnlyLearnedItems) config.wk_items.filters = {srs: {value: ["lock", "init"], invert: true}};
			let items = await wkof.ItemData.get_items(config);
			byReading = wkof.ItemData.get_index(items, "reading");
			byCharacters = items.reduce((result, i) => { result[i.data.characters] = (result[i.data.characters] || []).concat([i]); return result; }, {});
			byMeaning = settings.useFuzzySearch ? hashGrams(items) : {get: text => {text = text.toLowerCase(); return items.filter(item => meaningsOfItem(item).some(m => m.toLowerCase().startsWith(text)));}};
		}

		// SRS stage durations are needed if spoiler handling is activated
		if (settings.spoilerHandling !== "none" && srs.length === 0) {
			if (!wkof.file_cache.dir["Apiv2.spaced_repetition_systems"] || new Date() - new Date(wkof.file_cache.dir["Apiv2.spaced_repetition_systems"].added) > 604800000) {
				srs = await wkof.Apiv2.get_endpoint("spaced_repetition_systems");
			} else {
				srs = (await wkof.file_cache.load("Apiv2.spaced_repetition_systems")).data;
			}
		}

		Object.assign(oldSettings, settings);
	}

	function setupMenu() {
		wkof.Menu.insert_script_link({name: "confusionguesser", submenu: "Settings", title: "ConfusionGuesser", on_click: openSettings});

		let defaults = {
			guessOnlyLearnedItems: true,
			spoilerHandling: "none",
			includeWhitelist: false,
			includeUserSynonyms: false,
			useFuzzySearch: true,
			useWkRadicals: true,
			useWkSimilarity: true,
			useIds: false,
			showAsOverlay: true,
			showAllByDefault: false,
			showTypes: true,
			showRatings: false,
			highContrast: false,
			fontSize: "1.12rem",
			hotkeyExpand: "e"
		}
		Object.assign(defaults, defaultColors);
		return wkof.Settings.load("confusionguesser", defaults).then(settingsChanged);
	}

	function openSettings() {
		let fontSizeOptions = {};
		fontSizeOptions["0.8rem"] = "Small";
		fontSizeOptions["1.12rem"] = "Medium";
		fontSizeOptions["1.5rem"] = "Large";
		fontSizeOptions["2rem"] = "Probably too large";
		fontSizeOptions["3rem"] = "Certainly too large";
		let spoilerHandlingOptions = {};
		spoilerHandlingOptions["none"] = "Never";
		spoilerHandlingOptions["0"] = "If in this batch";
		spoilerHandlingOptions["10"] = "If sooner than 10% of the SRS interval";
		spoilerHandlingOptions["20"] = "If sooner than 20% of the SRS interval";
		spoilerHandlingOptions["30"] = "If sooner than 30% of the SRS interval";
		spoilerHandlingOptions["50"] = "If sooner than 50% of the SRS interval";
		spoilerHandlingOptions["75"] = "If sooner than 75% of the SRS interval";
		let dialog = new wkof.Settings({
			script_id: "confusionguesser",
			title: "ConfusionGuesser Settings",
			on_save: settingsChanged,
			content: {
				tabFunctionality:            {type: "page",     label: "Functionality",                content: {
					guessOnlyLearnedItems:   {type: "checkbox", label: "Guess only learned items",     hover_tip: "When enabled, the guess list will only contain items that you have already learned on WaniKani."},
					spoilerHandling:         {type: "dropdown", label: "Hide spoiler guesses",         hover_tip: "Select if guesses for WK items that will soon come up for review should be hidden.", content: spoilerHandlingOptions},
					grpMeaningGuesses:       {type: "group",    label: "Meaning guesses",              content: {
						useFuzzySearch:      {type: "checkbox", label: "Use fuzzy search",             hover_tip: "When enabled, guesses for a wrong meaning also contain non-exact matches. Might increase loading time of the review page."},
						includeWhitelist:    {type: "checkbox", label: "Include hidden whitelist",     hover_tip: "When enabled, guesses for a wrong meaning also consider the hidden whitelist (for example 'boob grave')."},
						includeUserSynonyms: {type: "checkbox", label: "Include user synonyms",        hover_tip: "When enabled, guesses for a wrong meaning also consider your entered synonyms."},
					}},
					grpSimilarityRating:     {type: "group",    label: "Kanji similarity rating",      content: {
						useWkRadicals:       {type: "checkbox", label: "Use WK radicals",              hover_tip: "With this, kanji are considered similar if they share some WK radicals."},
						useWkSimilarity:     {type: "checkbox", label: "Use WK visually similar list", hover_tip: "With this, kanji are considered similar if WK has them listed as visually similar."},
						useIds:              {type: "checkbox", label: "Use IDS",                      hover_tip: "When enabled, a 2MB text file with ideographic description sequences will be downloaded and stored locally to improve kanji similarity ratings."}
					}}
				}},
				tabInterface:                {type: "page",     label: "Interface",                    content: {
					showAsOverlay:           {type: "checkbox", label: "Show as overlay",              hover_tip: "Display the guess list as an overlay to the right of the question or at the bottom of the page. On narrow displays, the list is always at the bottom."},
					showAllByDefault:        {type: "checkbox", label: "Show all guesses by default",  hover_tip: "When enabled, the guess list will be expanded by default."},
					showTypes:               {type: "checkbox", label: "Show guess types",             hover_tip: "Show 丸⬄九 for guesses based on visual similarity, on⬄kun if you used on'yomi but needed kun'yomi, etc."},
					showRatings:             {type: "checkbox", label: "Show ratings",                 hover_tip: "When enabled, a number between 0 and 1 to the right of each guess shows the rating of that guess."},
					highContrast:            {type: "checkbox", label: "High contrast mode",           hover_tip: "When enabled, the overlay will have a dark background. Always active if the guesses are displayed at the bottom of the page."},
					fontSize:                {type: "dropdown", label: "Font size",                    hover_tip: "Select the font size for the guesses.", content: fontSizeOptions},
					hotkeyExpand:            {type: "text",     label: "Hotkey expand guesses",        hover_tip: "Choose a hotkey to expand/collapse the list of guesses.", match: /^.?$/}
				}},
				tabGuessColors:              {type: "page",     label: "Guess colors",                 content: {
					vissimColor:             {type: "color",    label: "丸⬄九",                        hover_tip: typeDescription.visuallySimilar},
					ononColor:               {type: "color",    label: "on⬄on",                       hover_tip: typeDescription.onyomionyomi},
					onkunColor:              {type: "color",    label: "on⬄kun",                      hover_tip: typeDescription.onyomikunyomi},
					kunonColor:              {type: "color",    label: "kun⬄on",                      hover_tip: typeDescription.kunyomionyomi},
					kunkunColor:             {type: "color",    label: "kun⬄kun",                     hover_tip: typeDescription.kunyomikunyomi},
					naonColor:               {type: "color",    label: "na⬄on",                       hover_tip: typeDescription.nanorionyomi},
					nakunColor:              {type: "color",    label: "na⬄kun",                      hover_tip: typeDescription.nanorikunyomi},
					onnaColor:               {type: "color",    label: "on⬄na",                       hover_tip: typeDescription.onyominanori},
					kunnaColor:              {type: "color",    label: "kun⬄na",                      hover_tip: typeDescription.kunyominanori},
					nanaColor:               {type: "color",    label: "na⬄na",                       hover_tip: typeDescription.nanorinanori},
					specialColor:            {type: "color",    label: "Special",                      hover_tip: "Reading exception / WK does not list this reading"},
					textColor:               {type: "color",    label: "Text",                         hover_tip: "Text color"},
					resetColor:              {type: "button",   label: "Reset colors to default", text: "Reset", on_click: (name, config, on_change) => {Object.assign(wkof.settings.confusionguesser, defaultColors); dialog.refresh();}},
					buttonColor:             {type: "button",   label: "From button color", text: "Load", on_click: (name, config, on_change) => {Object.assign(wkof.settings.confusionguesser, fromButtonColor()); dialog.refresh();}, hover_tip: "Use the colors of the buttons below the input box. Useful for dark mode users."}
				}}
			}
		});
		dialog.open();
	}

	function installCss() {
		let css = "#confusionGuesserOverlay { position: absolute; top: 3rem; right: 0; padding-top: 0.6rem; z-index: 100; overflow-x: hidden; pointer-events: none; }" +
			"#confusionGuesserOverlay.noOverlay { position: relative; top: 0; margin-top: 3rem; }" +
			"#confusionGuesserOverlay > div { border-style: solid none; background: linear-gradient(to left, rgba(0,0,0,0.2), transparent 50%, transparent); border-image: linear-gradient(to left, rgba(255,255,255,0.8), transparent 65%, transparent) 1; transition: transform 0.2s; pointer-events: initial; }" +
			"#confusionGuesserOverlay > div.highContrast, #confusionGuesserOverlay.noOverlay > div { background-color: rgba(0, 0, 0, 0.4); padding-left: 0.7rem; }" +
			"#confusionGuesserOverlay > div.collapsed { transform: translateX(100%); }" +
			"#guesses { margin: 0.6rem 0; padding: 0 60px 0.6rem 0; max-height: 13rem; overflow-x: hidden; overflow-y: auto; display: grid; grid-template-columns: auto auto 1fr auto; grid-row-gap: 0.2rem; }" +
			"#confusionGuesserOverlay.noOverlay #guesses { max-height: initial; overflow-y: auto; }" +
			"#confusionGuesserOverlay.hideRatings #guesses > a > *:last-child { display: none; }" +
			"#confusionGuesserOverlay.hideTypes #guesses > a::before { display: none; }" +
			"#confusionGuesserOverlay.hideTypes #guesses > a > *:first-child { border-radius: 0.5rem 0 0 0.5rem; }" +
			"#guesses > a { display: contents; color: var(--text); text-decoration: none; --type: '?' }" +
			"#guesses > a.onyomionyomi     { --gc: var(--onon); --type: 'on⬄on' }" +
			"#guesses > a.onyomikunyomi    { --gc: var(--onkun); --type: 'on⬄kun' }" +
			"#guesses > a.kunyomionyomi    { --gc: var(--kunon); --type: 'kun⬄on' }" +
			"#guesses > a.kunyomikunyomi   { --gc: var(--kunkun); --type: 'kun⬄kun' }" +
			"#guesses > a.nanorionyomi     { --gc: var(--naon); --type: 'na⬄on' }" +
			"#guesses > a.nanorikunyomi    { --gc: var(--nakun); --type: 'na⬄kun' }" +
			"#guesses > a.onyominanori     { --gc: var(--onna); --type: 'on⬄na' }" +
			"#guesses > a.kunyominanori    { --gc: var(--kunna); --type: 'kun⬄na' }" +
			"#guesses > a.nanorinanori     { --gc: var(--nana); --type: 'na⬄na' }" +
			"#guesses > a.onyomiundefined  { --gc: var(--special); --type: 'on⬄special' }" +
			"#guesses > a.kunyomiundefined { --gc: var(--special); --type: 'kun⬄special' }" +
			"#guesses > a.nanoriundefined  { --gc: var(--special); --type: 'na⬄special' }" +
			"#guesses > a.visuallySimilar  { --gc: var(--vissim); --type: '丸⬄九' }" +
			"#guesses > a.radical          { --ic: var(--radical-color, #00AAFF); }" +
			"#guesses > a.kanji            { --ic: var(--kanji-color, #FF00AA); }" +
			"#guesses > a.vocabulary       { --ic: var(--vocabulary-color, #AA00FF); }" +
			"#guesses > a > *, #guesses > a::before { display: flex; align-items: center; padding: 0.2rem 0.5rem; }" +
			"#guesses > a > *:first-child { display: initial; grid-column: 2; }" +
			"#guesses > a::before { content: var(--type); border-radius: 0.5rem 0 0 0.5rem; color: rgba(255, 255, 255, 0.3); grid-column: 1; }" +
			"#guesses > a > *:last-child, #confusionGuesserOverlay.hideRatings #guesses > a > *:nth-last-child(2) { border-radius: 0 0.5rem 0.5rem 0; justify-content: flex-end; border-right: solid var(--ic); }" +
			"#guesses svg, #guesses ruby img { width: 1em; fill: none; stroke: currentColor; stroke-width: 85; stroke-linecap: square; stroke-miterlimit: 2; transform: translateY(0.15em); }" +
			"#guesses > a.show svg, #guesses > a.show ruby img { filter: drop-shadow(2px 2px 3px rgba(0,0,0,0.4)); }" +
			"#guesses > a.show > *, #guesses > a.show::before, #guesses > button.show { font-size: 1.35em; text-shadow: 2px 2px 3px rgba(0,0,0,0.4); background-color: var(--gc); box-shadow: 0.5rem 0.2rem 0.2rem #0000002b; }" +
			"#guesses > button { grid-column: 1 / -1; padding: 0.5rem; color: rgba(255, 255, 255, 0.3); background: none; border: thin dashed rgba(255, 255, 255, 0.3); border-radius: 0.5rem; --gc: var(--vissim); }" +
			"#guesses > button.show { border: none; }" +
			"#guesses > button + a { display: none; }" +
			"#confusionGuesserOverlay > div > input { display: none; }" +
			"#confusionGuesserOverlay > div > label { position: absolute; top: 0; right: 25px; background-color: white; width: 1.4rem; line-height: 1.4rem; text-align: center; border-radius: 0.3rem; font-weight: bold; cursor: pointer; font-size: large; }" +
			"#confusionGuesserOverlay.noOverlay > div > label { border: solid thin rgba(0, 0, 0, 0.4); }" +
			"#confusionGuesserOverlay > div > :checked + label { display: none }" +
			"#confusionGuesserOverlay > div > :checked ~ #guesses > a:not(.show) > *, :checked ~ #guesses > a:not(.show)::before, #confusionGuesserOverlay > div > :checked ~ #guesses > button:not(.show) { display: none; }" +
			"#hotkeys tr.disabled { display: none; }" +
			"@media (max-width: 767px) {" +
			" #confusionGuesserOverlay { position: relative; top: 0; margin-top: 3rem; }" +
			" #confusionGuesserOverlay > div { background-color: rgba(0, 0, 0, 0.4); padding-left: 0.7rem; }" +
			" #confusionGuesserOverlay #guesses { max-height: initial; overflow-y: auto; }" +
			" #confusionGuesserOverlay > div > label { border: solid thin rgba(0, 0, 0, 0.4); }" +
			"}" +
			// firefox workaround (otherwise shrinks grid width when scrollbar appears, leading to line breaks in the cells)
			"@-moz-document url-prefix() { #guesses { overflow-y: scroll; } }";
		let sCss = document.createElement("style");
		let tCss = document.createTextNode(css);
		sCss.appendChild(tCss);
		document.head.appendChild(sCss);
	}
})();

I’m also curious if version 1.11 changes anything for you. Probably not, because the bugfix deals with a completely unrelated issue, but who knows.

1 Like

No worries, I appreciate your efforts all the same ^^

I’ve just been trying a few things to fix the issue myself, and something I did seems to have done it - the corruption has completely stopped :slightly_smiling_face: I’ve also updated to 1.11, restarted browser and line 290 (the line that always got corrupted) was exactly as it should be. So hopefully that’s that. :crossed_fingers:

For my own reference and in case it helps anyone else, here’s what I did:
I reinstalled TamperMonkey, updated my language packs (Japanese and English UK), cleared my browser cache and (after exporting my scripts) I did a factory reset on TamperMonkey

I’m guessing it was the TamperMonkey reinstall or reset that did it, but I try to be thorough

3 Likes

Thanks for your work on this

1 Like

Is it possible to configure ConfussionGuesser to never show similar items automatically?

(the options seems to allow “always” and various ranges based on SRS only)

1 Like

What exactly do you mean by that? The whole purpose of ConfusionGuesser is to show similar items. Do you want that it does not pop up by itself, but only when you click a button? Or what is the intended use case?

2 Likes

Yes, exactly that. Only show pop-up when clicking button or using shortcut.

Why? I’m trying to de-clutter the UI, and quite often after negative answer I know what I mistaken it with. So I would only need the script for the times when I don’t know and then I could call it explicitly.

Sorry that it took a while. I have now added a setting When answer incorrect (in the tab Interface). If you change it to Minimized ‒ show arrow, only an arrow will appear which you can click to show the list of guesses. And if you set it to Show nothing, not even the arrow is displayed (but you can still use the hotkey to open the list of guesses).


Version 1.12 patch-notes:

Guess list now also available for correct answers

Until now, ConfusionGuesser only reacted to incorrect answers. I have now slightly adapted the algorithm to also be able to show “guesses” after a correct answer. I’m not sure if this feature will be all that helpful, but it might be used to compare other items similar to the current one.

For correct meaning answers, I filter out all guesses where the characters are contained in the current item. Example: After answering 崇拝 with “worship”, the guess list will neither contain [(すう) Worship] nor [(はい) Worship]. The intention is to prevent spoiling the reading.

Settings for behavior after correct/incorrect answer

The tab Interface in the settings dialog now contains two new entries: When answer correct and When answer incorrect. The three available options are: Show nothing, Minimized ‒ show arrow and Show list. When Show nothing or Minimized ‒ show arrow is active, no guesses are computed automatically. So it might sometimes happen that you click the arrow but nothing happens, because ConfusionGuesser only starts the search after the click, but no guesses are found.

Please let me know if something doesn’t work.
Link to previous script version for downgrading in case version 1.12 doesn’t work for you

3 Likes

@Sinyaven seems to be working OK. thank you a lot! :partying_face:

1 Like

This script is fantastic - thank you so much @Sinyaven!

What would you think of a setting for the “When Answer Incorrect” dropdown, that amounts to “Show if similarity > X, otherwise minimize”, with X being a configurable value?

For example, right now, I seem to be triggering the script quite often based on things that have a really low similarity rating:

image

I’m concerned that too many false positives will eventually desensitize me to the “real” cases where I was confused. If I could say, for example, “Only pop up when similarity is >= 0.3” - I think it would help the overall effectiveness of the script as a learning aid!

Thanks, I’m glad it’s useful to you! :slight_smile:

Regarding the request: I’m not sure – isn’t it still useful to know whether the reading you used actually exists for any kanji (in WK)?

That’s an interesting point.

I think that, since Japanese is so full of homonyms, odds are pretty high that the reason my answer matches another kanji isn’t because I got the two kanji confused visually, but more out of sheer coincidence. Getting alerted in those cases, amounts to being given a random fact: “Hey, did you know that the reading you typed also happens to match another Kanji that looks nothing like this Kanji?”

That’s potentially interesting information, but it doesn’t exactly relate to the core reason why I love the 丸⬄九 feature of the script, which is helping to identify points of confusion between visually similar Kanji.

I admit, I can also imagine cases where I completely mixed up two kanji that look nothing alike - perhaps I entered こい for 愛. In that case there really was confusion, but I’d argue that’s: (1) probably the minority of cases, and (2) actually a different type of confusion than 丸⬄九.

I’d propose that a minimum level of visual similarity be required for alerting on 丸⬄九 and if it seems worth the effort, a different, optional alert type be added for something akin to 恋⬄愛

But all that said, I still massively appreciate the effort you’ve put into this script, and will happily use it as-is. I can always just configure the setting to minimize the popup by default, and treat it more as a “pull” than a “push” system!

4 Likes

Yes, I originally intended to add more confusion types such as the confusion based on similar meaning that you mention, but when when testing it just with the visual similarity rating, it seemed already sufficient for a useful sorting and I kept it at that.

It never bothered me having to check if the top guess applies to my wrong answer or if it’s a “false positive”, but that’s probably because I never used my script while having to go through hundreds of reviews (I created the script one year after reaching level 60). I can see that it might become annoying to get proposed some highly unlikely guesses during long review sessions.

I will look into adding the setting you proposed. And maybe add a guess type for similar meaning, I’m not sure yet. I just wonder what default color this guess type should get – I’m running out of options from the WK color palette :sweat_smile: Maybe the green from the review forecast? :thinking:

3 Likes

Version 1.13 adds your proposed option “Minimized if all ratings < X” to the dropdown. I also did some experiments with a new guess type for confusions based on similar meaning instead of visual similarity, but then I realized that most kanji on WK only have one meaning defined. To take your example with 恋 and 愛, the first only has the meaning “romance” and the second only the meaning “love”. So I would have to take a detour to the vocabulary and deduce additional kanji meanings from there. For now, I decided that it is not worth the effort.


Version 1.13 patch-notes:

added option “Minimized if all ratings < X”

In the settings dialog under the tab Interface, the dropdown menus for When answer correct and When answer incorrect now contain a new entry Minimized if all ratings < X. When selecting it, a slider appears below that allows you to define a value for X. If all guesses have a rating below X, the guess list will stay minimized. If at least one guess has a rating ≥ X, the whole guess list will be displayed.

The intended purpose of this feature is to only draw the user’s attention if a guess has a high likelihood of showing the reason for confusion.

Because the displayed guess ratings are rounded, it might happen that guesses with a displayed rating of e.g. 0.30 are actually still below a threshold of 0.30.

Please let me know if something doesn’t work.
Link to previous script version for downgrading in case version 1.13 doesn’t work for you

1 Like

Woohoo - thank you so much!

I waited a day or two to make my first mistake that triggered a suggestion before replying: and low-and-behold, the first one I hit was a perfect example of two kanji I get confused by all the time. No more false positives!
image

Again, thanks so much, this is an amazing tool all around. I’m not sure what the forum rules are on the topic, but if you’ve got a Patreon or a “buy me a beer” link and are allowed to share, I’d definitely contribute :smiley:

1 Like

Thanks, I don’t have anything like that, but I’m always happy to hear that my script is helpful to someone out there! :slight_smile:

1 Like

This is a really useful script! :slight_smile: One thing I noticed is that if you answer a question with a really short answer like “a” or “the” the script can add 200+ ms of lag to displaying whether you got the question wrong or right. I was thinking a good way of fixing it for length 1 guesses might be to have rateSimilarity return 0 if one of its arguments has length 1 and the longer string does not have the character in it and then rate the similarity of the strings that do according to how close their length is to 1.

1 Like

I agree that my implementation is certainly not the most efficient, but in this case I think the problem lies somewhere else. Do you by any chance have “Use fuzzy search” disabled? I just checked and found out that with fuzzy search enabled, searchByMeaning("a") returns 0 results, while without fuzzy search, it returns ~800 items. And then the script has to rate the similarity of all the meanings of the 800 items to “a”. Admittedly, with a better optimized rateSimilarity() function this would also not be a problem, but I think it’s even better to prevent the case of having 800 useless guesses in the first place.

I probably shouldn’t just use startsWith() as an alternative to fuzzy search. At least now I know that ~800 meanings on WK start with “a” :sweat_smile:

1 Like

I do have fuzzy search disabled. It was a while ago, but I think the reason I disabled it was that having looser matching criteria for meaning questions made the script more likely to offer suggestions in cases where it hadn’t before, but those additional suggestions often weren’t super relevant which caused me to tune out the suggestions that were relevant.

When I profiled the script, most of the execution time was being spent in the Levenshtein distance function, which is why I was suggesting potentially skipping it for cases where one of the strings is only one letter. (When I benchmarked always skipping it, the runtime of the script became much more reasonable.) Of course, reducing the number of times you call the function by reducing the number of search matches like you are saying is also a perfectly reasonable fix.

1 Like

In version 1.13 I have added the option “Minimized if all ratings < X” so that the guess list does not pop up if the guesses are probably not that relevant. Maybe that would help?

The problem is that I use the rateSimilarity() function for both English vocabulary and Japanese vocabulary, and in the case of Japanese vocabulary, having the longer string not contain the other single character should not immediately result in 0 similarity (because in Japanese, it should also consider the visual similarity of the kanji). It would definitely be more efficient to split up the function into rateEnglishSimilarity() and rateJapaneseSimilarity() and optimize the English version, but I think I would prefer to leave the function as is and instead prevent the 800 guesses from happening.

1 Like

Version 1.14 patch-notes:

Performance improvement for one-letter answers with fuzzy search disabled

With fuzzy search disabled, I just searched for all meanings that start with the given string. This led to a huge number of results when searching for very short strings (e.g. ~800 results for “a”). As a result, you would get hundreds of worthless guesses after entering “a” as an answer to a meaning question. Such a high number of guesses also leads to noticeable lag.

To prevent this, I switch to only considering exact matches if the previous method returns more than 16 results.

Small performance improvements for "Levenshtein distance" function

When writing the ConfusionGuesser script I tried to avoid using for-loops and instead used forEach(), map(), filter(), reduce(), …
Sadly, my “Levenshtein distance” function was extremely slow compared to a version using for-loops (tested on Chromium), so I’m now using for-loops in that function.

An interesting discovery I made in my experiments:

Array initialization
console.time("new Array => fill => map");
for (let i = 0; i < 1000; i++) {
	let d = new Array(1000).fill(null).map((a, j) => j);
}
console.timeEnd("new Array => fill => map");

console.time("Array.from");
for (let i = 0; i < 1000; i++) {
	let d = Array.from({length: 1000}, (a, j) => j);
}
console.timeEnd("Array.from");

console.time("for-loop");
for (let i = 0; i < 1000; i++) {
	let d = [];
	for (let j = 0; j < 1000; j++) d[j] = j;
}
console.timeEnd("for-loop");

Result in Microsoft Edge (Chromium):

new Array => fill => map: 16.60595703125 ms
Array.from: 75.56396484375 ms
for-loop: 7.5732421875 ms

Result in Firefox:

new Array => fill => map: 29ms
Array.from: 16ms
for-loop: 22ms

Chromium is awful at Array.from()! :open_mouth: Until now I had used Array.from() in my “Levenshtein distance” function, but now I’m just using a for-loop.

Please let me know if something doesn’t work.
Link to previous script version for downgrading in case version 1.14 doesn’t work for you

2 Likes