Are reviews actually random?

I always assumed that review ordering is random when you do fewer reviews than the total number of reviews available, but lately, I keep getting the feeling that WK must have tweaked the algorithm so that recently missed items come up a lot more often than chance.

For example in my last two review sessions just now, the proportion of leeches that came up for review was so high that there was only a 1 in 800 chance of that happening by chance, which seems pretty suspicious. And no, I don’t have any scripts installed.

2 Likes

It’s completely random, the only thing not random is that you can only have 10 active items at a time and new items only get put into that list if you complete one.

Probably a case of frequency bias if I could guess

3 Likes

They’re not random. They’re very predictable.

The only thing that’s random is the order of items within a single review session. The items which appear during the session are based on SRS intervals and how frequently you’ve made mistakes in the past.

4 Likes

I think this was the question, though not sure

1 Like

I was going by this quote:

1 Like

My guess stems from the fact that he’s lvl 46, so he probably knows the srs timings and how that works

2 Likes

Yeah, obviously I know about SRS intervals and so on. I’ve been doing WK for years and already gone to level 60 and back. I was hoping that was clear when I said in the original post

I always assumed that review ordering is random when you do fewer reviews than the total number of reviews available

That doesn’t seem to add up. If you had 200 reviews and only did 100. If you happen to get all your leeches in the first 100, then the second 100 can’t have any of them.

The proportion can’t change, only the order.

1 Like

It’s pure chance btw, I even checked the code now, I personally always get my level up items as the last available, so you could have it worse :slight_smile:

1 Like

Ok, here’s some specific numbers if it makes things easier for you:

I had 210 items that I designated “leeches” (note that this set was chosen before I did any reviews today), and 3407 reviews available.

In my first session today, I did 24 reviews and had 5 “leeches” come up. We can use a hypergeometric calculator to calculate the odds. With a population size of 3407, population successes of 210, and sample size of 24, the odds of getting 5+ are 0.013751326, or ~1.4%.

But hey, maybe that was just a fluke, so I did a second review session. This one wasn’t quite as lopsided, but it was still unlikely, as I got 6 out of 52. Since this was just half an hour later, nothing from the first review session would come back up yet (shortest interval is 4 hours), which makes things easy. Population size 3383, population successes 205, sample size 52 gives odds of 6+ at 0.091833235. (9.2%)

Combining the two gives 0.001262828 or about 1 in 792. I hope that helps.

1 Like

Thanks for checking. I guess I must have just gotten really lucky/unlucky today.

The odds calculations are correct, but the thing with statistics is that a non-zero probability means that an event can occur. You can’t know for sure if your scenario is common until you’ve done a lot of experiments. To begin to get reliable data you would need to repeat the experiment at least 30 times.

4 Likes

Just out of curiousity, what if the algorithm had been tweaked to bring leeches toward the beginning of the queue. Would you consider that good or bad?

(I’ve been doing a little bit of leech research in my spare time, so I find this stuff interesting)

3 Likes

Yes, this seems like the more interesting question to me! Thanks for putting it out there. :slight_smile:

They’re actually not random, they follow the pattern of the Illuminati Bible Code. If you can reverse engineer the Code you can use it to pick winning lotto numbers…

3 Likes

And if you reverse engineer it incorrectly, you will lose all your money on lottery tickets. That’s how you know whether you did it correctly.

3 Likes

I think I did that without trying, because all my money is gone

2 Likes

Except that being WaniKani, they use the Crabigator’s Kanji Code instead

1 Like