Goodharting Wanikani

Japanese Language
1

Goodhart’s law states that:

When a metric becomes the goal, it stops being a good metric

A classic example would be grades in the school. In theory, the goal of school is “learning”. The way we measure if a student is learning or not is by using a test. The problem is that the test becomes the goal itself, and the student simply focuses on achieving a high score (which does not necessarily mean learning. Many times a high score can be obtained by memorizing without any understanding of the subject).

Here in WaniKani, our real goal (for most of us I guess), is to learn Japanese. One of the unavoidable aspects of learning Japanese is learning Kanji, and WaniKani is an amazing tool for that… I will write another post about how thankful I am to the WaniKani developers, but that’s not the right moment (that will be in level 60). Right now the topic of this posting is reflecting on Goodhart’s law and WaniKani

WK is so well designed that learning Kanji becomes a game, and the goal of that game is completing the 60 levels. Reaching level 60 must feel like killing the final boss, but there is a problem with this. I feel most of us are too obsessed with levelling up, and it is easy to forget that this is not the real goal. I wonder how many of the people who made it to level 60 remembers just a small fraction of all kanjis a few months later after finishing WK. The real goal is reading Japanese, not reaching level 60!

For this reason, I feel there is something missing in WaniKani: A button that allows you to erase a word from the system because you don’t want to learn that word. Why is that? Because there are many words that I will probably never ever use in Japanese, many of them are just leeches that keep coming to me and waste my time. Just an example (but there are many others): I don’t think I will ever need this: 造園. If ever come across this word in the real world, I am happy to look for the meaning at that moment. But having to fight against this word, again and again, feels like a purposeless task. I wonder, is there any third party app doing this specific thing, removing a word from the system?

I know some people might think that this would be cheating: if you want to really FINISH WaniKani you have to do all the words etc. But I have to disagree for the reasons explained above. At this point, I keep doing WaniKani but still need to spend a solid 1h-2h everyday here doing my reviews, and when a lot of really weird words appear, I feel I am a bit wasting my time and I should be instead studying Japanese in a different way.

Thoughts?

32 Likes
2

At least it’s a different word from usual.

Presumably most people are consuming content alongside using WK throughout, which reinforces what they know, and then aren’t going to stop just because they reached level 60. Usually the desire to consume content in Japanese is what drives people to start here, so I don’t think it’s all that common to forget about that aspect of learning. There are book clubs and whatnot here as well to guide people through level-appropriate content if they feel they would benefit from that.

Other tools may be more in line with your preferences. That’s fine.

31 Likes
3

Don’t get me started. I just did the lesson for “sacrifice fly”. If I could kill / erase that word, I would. :smiley:

17 Likes
4

There’s been a lengthy thread or two about this you might be interested in, for example:

If some forum regulars respond with exasperation, that would probably be why. The threads tend to end up long and somewhat combative.

For my part – I agree that clearing Wanikani isn’t the goal, learning Japanese is. But I’d say that’s an argument for slowing down and learning everything it has to offer rather than trying to push forward and assuming some words won’t be useful!
As far as I know there exists no such tool to add the delete feature to Wanikani currently. With that lack for now, if it helps, maybe think of 造園 as just another small way to practice the readings and meanings of 造 and 園 individually, as you may not come across the specific word but certainly will come across new words with those components.

29 Likes
5

Yeah, but don’t forget this word is a gift so you can remember the Kanji for sacrifice and hit. Use that anger to help you remember!

7 Likes
6

I feel like I got some whiplash here. I was with you in the first half of your post, but then after declaring in bold that learning is the goal you stated you wish WaniKani wouldn’t force you to learn things that you personally don’t deem worthy of your time. But if reaching Level 60 quickly isn’t the goal, then why should a few extra seconds on these items matter in the first place?

This was my initial thought when I saw the word as well :joy:

37 Likes
7

But if reaching Level 60 quickly isn’t the goal, then why should a few extra seconds on these items matter in the first place?

Because they keep appearing, that’s all. It would be cleaner if I can put them aside and I suspect that writing a script for that should be easy.

I think that most of the vocabulary is relevant in fact, even things that don’t necessarily appear too often. It is just maybe 5% of the words I think are a waste of time, some of them I don’t know even in my native language. Yes, I could ignore them and keep doing the revisions (I agree with that), but it would be even better it could just simply removing them right away. 5% of 6k words are at least 300 words…

5 Likes
8

IN :clap: DEED :clap:

Nah, he said that reading is the goal. And if you are talking about getting good at reading as fast as possible, learning words that you won’t see very often before words that you will see very often is a bad choice. So he’s right, but I mean thats a problem with wanikani down to the very core and I think everyone who chooses to use it is either aware of that sacrifice or just oblivious.

And to answer your question, learning words you won’t use for a long time should matter because of something called opportunity cost. For example, why would I learn 造園 when I could learn 炬燵.

To reply to OP doe, yes your gripe is perfectly valid. Like I said, its a problem at WKs core doe and I don’t think they will change anything. There are other methods that will have you learn faster, but I think WK does a pretty good trade off of efficiency for stability, ease, and guidance. If you don’t want that sacrifice doe, I would check out refold.la or something.

21 Likes
9

This is what I did recently. (Although my script is really clunky and bothersome to use, as I don’t want to make it easy for me to abuse it.) I have leeches that just keep coming up week after week, for months on end. They were taking up so much time that I eventually started dreading doing my WaniKani reviews, because I knew 98 of 100 reviews would be leeches that I just can’t get.

When I did an analysis of text sources that I am reading, or from the same genre that I want to read, most of these leeches don’t come up even once across over 38,000 unique words (almost a million total words).

By writing a script to auto-pass my leeches when they are in Apprentice, I’ve freed up a lot of time, which I’m currently using to reinforce my recognition of various kanji.

Edit: But if it’s a word I’ll never(?) encounter that doesn’t become a leech, I personally don’t mind the reviews, even if I’ll end up forgetting it after burning. Hopefully the vocabulary will have helped reinforce the kanji for more common words that use it.

2 Likes
10

Well, I’d argue that you’re probably almost always going to see that written in hiragana and not kanji, but that’s beside the point :stuck_out_tongue:

As for your general argument, I don’t really disagree at all. I would just add that I think some people forget WaniKani is a kanji learning platform, and would never be sufficient as a lone study resource or as a primary resource for vocab.

As for how much time is “wasted”, let’s use the OPs numbers of 300 “useless” words of 6,000. Assuming you spend 3 seconds per each review of those items, you’d get roughly 30 seconds total per item from lesson to burn, or 2 hours for all 300 from lesson to burn over the course of a year at max speed. Which would be an average of 20 seconds/day “wasted” in that year. Halved if you take 2 years (I took 2.5 years myself) to finish WaniKani. I guess my point is the items still help you reinforce the readings of those kanji, and the additional time spent is negligible.

Now, leeches are a whole separate entity entirely and WaniKani should absolutely implement something to help deal with this.

2 Likes
11

I think the purpose of Wanikani is teaching kanji so those “useless” vocab help reinforce memorize kanji in different combinations and usages. So if the user goal is to be able to read Japanese and only using wanikani. I think they are expecting wanikani to be something it is not suppose to be.

Wow I sound like a WaniKani fanboy, am I not? :upside_down_face: This is honestly how I see WaniKani though.

13 Likes
12

I am genuinely curious. Is it the reading or the meaning of these kanji/vocab that you can’t get? Or both?

13

I think rephrasing this as “why would I learn how 造 or 園 behave as compounds when I could learn how 炬 or 燵 behave as compounds” would make it match Wanikani’s mindset more, for better or for worse.

With the opportunity cost line of thought… what exactly is the opportunity lost? Is it a pure time “could be doing something else with those accumulated seconds” concern or just that more precisely targeted vocabulary would get people reading more comfortably faster?
There’s so much bulk vocabulary involved in reading it’s hard for me to picture how it could be streamlined without getting caught up in some odd words that happen to come up repeatedly, so wanikani’s vocab choices just felt like preparation for that to me - practice confronting a word base I can’t control.
Like it doesn’t feel a world apart to me from like, a book series where one term happens to come up a lot intermittently over hundreds of pages - “what does that mean again? – ah right.”

7 Likes
14

I mix up readings of some kanji. I mix up meanings of other kanji. I completely blank out on some words.

I just had 反則 come up for review, which nine times out of ten I think is read はんせい (and I have no idea why). Sometimes I do think it’s はんそく, but then I think, “Wait, I always get this one wrong. So if I think it’s はんそく, then that’s wrong. It’s probably…はんせい seems like it might be right.”

I did the lesson for 反則 just a few days over one year ago, and it’s currently Apprentice IV.

From my text analysis of the kind of material I’m likely to read, out of almost one million total words, 反則 comes up five times (once each across five different animes’ subtitle files).

For the curious, I did the lesson for 反省 about 15 months ago, and it’s at Guru.

(But I’m not completely hopeless. I do have 3,100 burns! Not counting a few I unburned because I didn’t recognize them while reading.)

6 Likes
15

If you shared that script that would be a great contribution to the community!

16

The script is so hacked together that I don’t think anyone could get any use out of it without a lot of clean-up. Also, it requires having Ruby and certain Ruby Gems installed, as well as curl.

For the curious...

I have this in one file (json.rb). The text file api_key.txt has an API key (no newline after it) that allows sending review results.

require 'json'

def download_data

	api_key = File.read("/home/chris/Study/Japanese/TextAnalysis/Ruby Tools/WaniKani/api_key.txt")
	
	puts "curl 'https://api.wanikani.com/v2/subjects' -H 'Authorization: Bearer #{api_key}' > curl_1.json"
	puts "curl 'https://api.wanikani.com/v2/subjects?page_after_id=1000' -H 'Authorization: Bearer #{api_key}' > curl_2.json"
	puts "curl 'https://api.wanikani.com/v2/subjects?page_after_id=2000' -H 'Authorization: Bearer #{api_key}' > curl_3.json"
	puts "curl 'https://api.wanikani.com/v2/subjects?page_after_id=3000' -H 'Authorization: Bearer #{api_key}' > curl_4.json"
	puts "curl 'https://api.wanikani.com/v2/subjects?page_after_id=4000' -H 'Authorization: Bearer #{api_key}' > curl_5.json"
	puts "curl 'https://api.wanikani.com/v2/subjects?page_after_id=5000' -H 'Authorization: Bearer #{api_key}' > curl_6.json"
	puts "curl 'https://api.wanikani.com/v2/subjects?page_after_id=6000' -H 'Authorization: Bearer #{api_key}' > curl_7.json"
	puts "curl 'https://api.wanikani.com/v2/subjects?page_after_id=7000' -H 'Authorization: Bearer #{api_key}' > curl_8.json"
	puts "curl 'https://api.wanikani.com/v2/subjects?page_after_id=8000' -H 'Authorization: Bearer #{api_key}' > curl_9.json"
	puts "curl 'https://api.wanikani.com/v2/subjects?page_after_id=9000' -H 'Authorization: Bearer #{api_key}' > curl_10.json"
	
end

def load_data
	
	json_files = Dir["/home/chris/Study/Japanese/TextAnalysis/Ruby Tools/WaniKani/curl_*.json"]
	data = Array.new
	json_files.each do |json_file|
		data.push(*JSON.parse(File.read(json_file))['data'])
	end
	
	radicals = data.select { |d| d['object'] == 'radical' }
	kanji = data.select { |d| d['object'] == 'kanji' }
	vocabulary = data.select { |d| d['object'] == 'vocabulary' }
	
	return radicals, kanji, vocabulary
	
end

That one I use for other scripts as well, which is why it’s in a separate file.

This next one I run once to get text to download a JSON file via curl, then afterwards I run a second time to get the text to pass my leech lessons via curl. The leeches are stored in text files so the script knows what my leeches are. Afterwards, I delete the JSON file.

Note: This is set to only work on Apprentice. I have to give my leeches a fair try at Guru and (hopefully!) above.

require 'json'
require '../../TextAnalysis/Ruby Tools/WaniKani/json.rb'

api_key = File.read("/home/chris/Study/Japanese/TextAnalysis/Ruby Tools/WaniKani/api_key.txt")

radicals, kanji, vocabulary = load_data()

kanji_leeches = lines = File.readlines("Kanji_Leeches.txt", chomp: true)
vocabulary_leeches = lines = File.readlines("Vocabulary_Leeches.txt", chomp: true)

if !File.exist?("assignments.json")
	puts "curl \"https://api.wanikani.com/v2/assignments?immediately_available_for_review\" -H \"Authorization: Bearer #{api_key}\" > assignments.json"
	exit
end

assignments = JSON.parse(File.read("assignments.json"))

kanji_assignments = assignments['data']
	.select { |d| d['data']['subject_type'].eql? "kanji" }
	.select { |d| d['data']['srs_stage'].between?(1,4) }
	.select { |d| kanji_leeches.include?(kanji.select { |w| w['id'] == d['data']['subject_id'] }.map { |w| w['data']['characters'] }.first) }

vocabulary_assignments = assignments['data']
	.select { |d| d['data']['subject_type'].eql? "vocabulary" }
	.select { |d| d['data']['srs_stage'].between?(1,4) }
	.select { |d| vocabulary_leeches.include?(vocabulary.select { |w| w['id'] == d['data']['subject_id'] }.map { |w| w['data']['characters'] }.first) }

kanji_assignments.each do |a|
	puts "curl \"https://api.wanikani.com/v2/reviews\" -X \"POST\" -H \"Wanikani-Revision: 20170710\" -H \"Content-Type: application/json; charset=utf-8\" -H \"Authorization: Bearer #{api_key}\" -d $'{\"review\": {\"assignment_id\": #{a['id']},\"incorrect_meaning_answers\": 0,\"incorrect_reading_answers\": 0}}'"
end
vocabulary_assignments.each do |a|
	puts "curl \"https://api.wanikani.com/v2/reviews\" -X \"POST\" -H \"Wanikani-Revision: 20170710\" -H \"Content-Type: application/json; charset=utf-8\" -H \"Authorization: Bearer #{api_key}\" -d $'{\"review\": {\"assignment_id\": #{a['id']},\"incorrect_meaning_answers\": 0,\"incorrect_reading_answers\": 0}}'"
end
3 Likes
17

I don’t think it follows that if you do this fast you can’t possibly remember what you’ve learned….there are a lot of variables at play there…….

2 Likes
18

Out of curiosity, do you study the leeches separately while they’re out of apprentice and waiting to come back up for review?

The situation you were describing with 反省 and 反則 sounds awfully familiar to me, since it happens a ton with my anki deck nowadays. But there I actually like it when they get stuck at the lowest SRS levels, because that’s when I notice exactly what my confusion is and add a note like “NOTE: this isn’t 反則” which helps things stick a lot better, and then they make it out on their own eventually.

Skipping the apprentice stages for leeches sounds like a recipe for things to bounce around forever. But if you find it helps keep them off your plate during the annoying phase while still helping you distinguish them I can certainly get behind it!

7 Likes
19

@modboy had it above. It’s not how often you’ll see the word 造園, it’s how often you’ll see words that use the kanji 造 and 園 and remember the readings that the word 造園 taught you.

I mean if you want to talk about Goodhart’s law, the goal here isn’t to learn the vocabulary, it’s to learn the kanji. The vocabulary isn’t here for that purpose, it’s here to make the kanji readings stick. You’re supposed to use another source to have a conversational vocabulary. Tofugu does not claim otherwise.

15 Likes
20

All of those “useless” words are opportunities that Wanikani gives you to reinforce the kanji that you’ve learned, that’s what the vocab in Wanikani is for, to teach you new readings and reinforce meanings, is not about the usability of these words or their frequency in real Japanese media.

Like someone else mentioned there are other methods for you to learn Kanji and vocab sorted by frequency in anime, books, news etc, you should try those instead if you feel Wanikani is wasting your time or reaching level 60 is not important, because for me reaching level 60 equals being able to read comfortably anything even if it takes time and I have learn useless words as a sacrifice in order to retain Kanji better.

Also try failing those leeches over and over in one session until they come back to apprentice 1, and start from zero again, reading the mnemonics etc… that’s the only way I can get rid of leeches, having them trapped between apprentice, guru or master, where they go up and down over and over again is something that happens quite a lot, you’ve gotta make them appear more often until they stick, that’s how SRS works.

3 Likes