Mathematical Model of WaniKani

Hi everyone!

Although I’ve mostly been learning kanji and vocabulary over the past nine levels, I’ve also been fascinated with the system behind it. This post is my attempt to create a mathematical analysis to model how (I think) it works.

Note: This is only a model, and it’s a work in progress. If it is not predictive, or it doesn’t fit your experience with WaniKani, fine! That just means it’s not a very good model. This is just intended to be a fun exercise for the mathematically minded among us. And if you enjoy it, feel free to make your own tweaks and contributions

Independent Variable

This one is easy. We are calculating our likelihood to remember as a function of time. Seconds, hours, days - only the scale will change.

Dependent Variable

The next question I had was the specific value we’re modeling. At the most basic, we want to know whether we “recall” an item, or if we have “forgotten” it. We can think of that in terms of a binary success-fail, 1 or 0. So far so good.

That is a discrete value, though, and what we really want is something more continuous. And so, I’m going to let my dependent variable in the analysis be the percent chance of success. That is, how likely is it that a particular item will be remembered?

Function Behavior

Our basic intuition is that as soon as we learn an item, we have a pretty good chance of recalling it. At some point, we enter a period of “forgetting” during which that chance rapidly declines. Then, we experience a long period during which our chance of recall is negligible.

At no point do we really have a 100% chance of recall, nor do we have a 0% chance. Instead, our function should approach those bounds asymptotically, approaching but never quite reaching them.

To model this, I chose the inverse tangent function. I’m sure there are other functions that would work, and arguably better, but I figured this was a good place to start. Here is an example of what that function looks like:

image

So that’s kind of what we want, but it needs to be flipped upside down, and moved around and scaled to fit our model. That’s easy enough! Here is the basic result I came up with:

image

And here is the formula I used:

image
where

image
In this case, p = 99%, which is my threshhold of where I want my percent chance to start, at time y = 0. The variable t is my time scale in hours, where t represents the number of hours to the midpoint of the chart. I used t = 4 for the chart above.

Recap

So what does this get us so far? We have modeled the behavior of our memory from the point at which we learn an item to the point at which we forget it. The theory behind WaniKani is that our reviews should take place during the critical period in between, when we are just about to forget an item.

image

Future Analysis

Over the next week or two, I’m planning on developing out this model. For example:

  • What happens when we get a review correct? Presumably our chance of recall returns to a high level, and the graph sort of stretches out. But I’d like to make that more rigorous.
  • What happens when we get a review wrong? I’m making the assumption that when we get a review wrong, we look at the right answer, which counts as another exposure. WaniKani makes the assumption that the graph contracts, moving our next review closer.

And once all those questions are answered, we can take it out for a spin and see what it has to say about particular topics!

  • So far, we are looking at the “optimal” case. What about leeches? That is, how can we represent leeches in this model, and what can it tell us about a good way to squash them?
  • Wkstats makes it pretty easy to figure out our accuracy percentage. Can we use that value to predict the amount of time it will take us to get a given item to Guru, or Burned? Can we use it to predict the time it will take us to level up?
  • Is there any different arrangement of wait times that would be more efficient? For example, if you consistently miss a given review item, should the timeline be more compressed? Adding an extra Apprentice level and an extra Guru level, for example, but with less time in between?

At the end of the day, I’d like to do a full model of leveling up to level 60, with different starting conditions, to see how long it takes. But I’d say the model needs a lot more work to make that worthwhile.

So what are your thoughts? Any ideas on how to make this better?

8 Likes

Interesting ! :smiley:

On the last question : “Any ideas on how to make this better?”, the inverse tangent really feels arbitrary; given that there is a lot of data potentially available, I feel it’s reasonably doable (at least, worth a try) to train a learning model on the past review data to predict future behaviour – aka predict a custom, user-based function instead of a generic inverse tan. What do you think?

2 Likes

I’d say you should go for it! :smiley:

I just picked a function that fit my criteria and made the math easy. It’s also something I can generalize from, and it’s nice and easy to implement in Excel! My general approach is usually to start with a simple model and only add complexity where the simpler version fails to give good results. For example, there’s no reason to expect that we would see symmetry around that midpoint. Should the period of recall be shorter, and the period of forgetfulness longer? Should the decay be more gradual? These are great questions! But ones I can’t answer unless I pick and approach and explore it.

What would the data source for your proposed model be? I’m not enough of a programmer to mess around with the API, and even so, I don’t know if data is available at the time and time-stamp level. There’s also the problem of overtraining the model to fit an individual’s learning style, instead of modeling WaniKani on a theoretical basis. But I’m sure with adequate data collection, you could overcome that.

Thanks for the feedback. Any other fans of 数学 out there?

Sure, going with a simple model at first and then complexifying it is a good way to go :smiley: I remember reading that with SRS, recall follows a ~log curve (I’ll look back for a source to back this up), so that could be a way to model the recall rate. As far as I know, when you get a review wrong, wanikani just sets you back on the curve (independently of your own memory recall); so, getting a review wrong or correct only changes the future in that it changes where you are on the x-axis of that log-like function.

I’m not sure about the data, but there are userscripts that tell you the next review time (at least at the hour granularity), so, it is probably available somewhere ^^ I’ll try to fumble with the API, I’ll let you know if I get somewhere noteworthy !

As for overfitting, well, that’s what regularization is for :blush:

1 Like

The API v2 returns review data for the next 24 hours - not sure if that would help though. Probably more importantly, the API also has data about when a particular item has been Guru’d.

1 Like

Actually I think it’s a straight line on a log/log graph aka zipf’s law. skip to 3:45 in this vid by vsause https://www.youtube.com/watch?v=fCn8zs912OE
This would explain why each review interval on WK is twice as long as the last.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.