Official announcement on upcoming API Version 2 breaking changes. Addition of new endpoint /spaced_repetition_systems

Hello all,

This is a follow-up on the announcement a while back about replacing /srs_stages with /spaced_repetition_systems.

We’ve been working to get these changes into our application since the announcement. Now that we have the implementation in place, we can share the details and a timeline with you.

We do want to make clear we have not changed the timings, number of stages, or the passing stage for the two existing spaced repetition systems. All that has changed, at this very moment, is how this data is being stored, read, and delivered.

If you don’t care about the API then skip reading the below.

Limitations of /srs_stages

We created the /srs_stages endpoint during a time when we did not expect WaniKani to have more than a couple spaced repetition systems. In our application, the current SRS information is baked into the code.

Responding to requests, we created the endpoint to share the spaced repetition system details. Due to the nature of how the information exists, we treated it as a “report”. This worked fine for its purpose.

Here is the payload of /srs_stages:

{
  "object": "report",
  "url": "https://api.wanikani.com/v2/srs_stages",
  "data_updated_at": "2018-04-19T21:53:03.000000Z",
  "data": [
    {
      "srs_stage": 0,
      "srs_stage_name": "Initiate",
      "interval": 0,
      "accelerated_interval": 0
    },
    {
      "srs_stage": 1,
      "srs_stage_name": "Apprentice I",
      "interval": 14400,
      "accelerated_interval": 7200
    },
    {
      "srs_stage": 2,
      "srs_stage_name": "Apprentice II",
      "interval": 28800,
      "accelerated_interval": 14400
    },
    {
      "srs_stage": 3,
      "srs_stage_name": "Apprentice III",
      "interval": 82800,
      "accelerated_interval": 28800
    },
    {
      "srs_stage": 4,
      "srs_stage_name": "Apprentice IV",
      "interval": 169200,
      "accelerated_interval": 82800
    },
    {
      "srs_stage": 5,
      "srs_stage_name": "Guru I",
      "interval": 601200,
      "accelerated_interval": 601200
    },
    {
      "srs_stage": 6,
      "srs_stage_name": "Guru II",
      "interval": 1206000,
      "accelerated_interval": 1206000
    },
    {
      "srs_stage": 7,
      "srs_stage_name": "Master",
      "interval": 2588400,
      "accelerated_interval": 2588400
    },
    {
      "srs_stage": 8,
      "srs_stage_name": "Enlightened",
      "interval": 10364400,
      "accelerated_interval": 10364400
    },
    {
      "srs_stage": 9,
      "srs_stage_name": "Burned",
      "interval": 0,
      "accelerated_interval": 0
    }
  ]
}

However, we have plans to add more spaced repetition systems. This kind of report object does not scale for dynamic data like this. As far as the API is concerned, that means moving over to a more RESTful endpoint. For the application itself, we moved the details into a datastore.

To support this migration, we have created a new endpoint called /spaced_repetition_systems.

New Replacement Endpoint /spaced_repetition_systems

This endpoint works exactly like the other collection and resource endpoints.

I’ll briefly go over the details below. But if you would like to go straight to the details then view this pull request on the documentation. The changes in the pull request have been merged into master, which means the changes are now live on production.

A single spaced_repetition_system object returns the following example:

{
  "id": 1,
  "object": "spaced_repetition_system",
  "url": "https://api.wanikani.com/v2/spaced_repetition_systems/1",
  "data_updated_at": "2020-06-09T03:36:51.134752Z",
  "data": {
    "created_at": "2020-05-21T20:46:06.464460Z",
    "name": "Default system for dictionary subjects",
    "description": "The original spaced repetition system",
    "unlocking_stage_position": 0,
    "starting_stage_position": 1,
    "passing_stage_position": 5,
    "burning_stage_position": 9,
    "stages": [
      {
        "interval": null,
        "position": 0,
        "interval_unit": null
      },
      {
        "interval": 14400,
        "position": 1,
        "interval_unit": "seconds"
      },
      {
        "interval": 28800,
        "position": 2,
        "interval_unit": "seconds"
      },
      {
        "interval": 82800,
        "position": 3,
        "interval_unit": "seconds"
      },
      {
        "interval": 169200,
        "position": 4,
        "interval_unit": "seconds"
      },
      {
        "interval": 601200,
        "position": 5,
        "interval_unit": "seconds"
      },
      {
        "interval": 1206000,
        "position": 6,
        "interval_unit": "seconds"
      },
      {
        "interval": 2588400,
        "position": 7,
        "interval_unit": "seconds"
      },
      {
        "interval": 10364400,
        "position": 8,
        "interval_unit": "seconds"
      },
      {
        "interval": null,
        "position": 9,
        "interval_unit": null
      }
    ]
  }
}

We think for the most part the data above is self explanatory, especially if you have worked with /srs_stages before.

The stages is an array of objects. Each object details each stage. position is the stage number. For example, position 1 is SRS stage 1. Interval + interval unit is the length of time we advance from time of review registration adjusted to beginning of the hour.

If you are familiar with how our spaced repetition system works this shouldn’t be new information.

Some things to point out…

  • The past progressives unlocking_stage_position, starting_stage_position, passing_stage_position, and burning_stage_position align with the assignment past tense timestamps of unlocked_at, started_at, passed_at, and burned_at.
  • unlocking_stage_position and burning_stage_position do not have intervals since it does not influence assignment.available_at calculations. Internally, we call anything in-between these two stages the “reviewable” stages.
  • All spaced repetition system share the four key events: unlock, start, pass, and burn.
  • unlocking_stage_position is always 0. starting_stage_position is always 1. burning_stage_position is always the maximum position. The passing_stage_position can be anything but the unlocking_stage_position.

And to remind people what these events actually are…

  • unlock: When the assignment is made available in lessons
  • start: When the assignment exits lessons and go into reviews. In other words when the user has successfully quizzed the assignment in lessons.
  • pass: When the assignment reached a “satisfactory” state. This influences the unlocks of the amalgmations associated to the assignment.subject.
  • burn: When the assignment has reached the end of the spaced repetition system. The assignment has left the review queue.

If we were to map these events to the existing SRS stage names it would be the following:

  • unlock is Initiate
  • start is Apprentice I
  • passed is Guru I
  • burned is Burned

Here is a visual representation of the 10 stage system we are currently using.

Where:

  • U: Unlocking
  • S: Starting
  • P: Passing
  • B: Burning

As mentioned before, for any spaced repetition system, the unlocking, starting, and burning are fixed, while passing could be any stage excluding 0.

Let’s say, for example, we implement a new system consisting of five stages and where the passing stage is 3. Then it would look like the following:

One case we want to point out is the possibility of a two stage system. Not saying we will have this in our application anytime soon (or ever), but the support is there.

This system has the following qualities:

  • No “reviewable” stages. In other words once it exits lessons it is burned.
  • Unlocking stage is 0. Starting, passing, and burning stage is stage 1.
  • Both stages do not have any interval.

Here is a visual of what a two-stage system would look like:

Screen Shot 2020-06-17 at 10.16.45 AM

And no reviewable stages.

Associations to Spaced Repetition Systems

Every subject will belong to one spaced repetition system. In the data structure for subject a new attribute called spaced_repetition_system_id will be available. This is the spaced repetition system used to calculate assignment.srs_stage, assignment.available_at, and the other event timestamps (passed_at, burned_at, et cetera).

In addition review will have a spaced_repetition_system_id. This is the identifier of the subject.spaced_repetition_system_id at the moment the review was created. What this means that an old review’s spaced_repetition_system_id does not necessarily match with the related subject’s current spaced_repetition_system_id. When this is the case this means the spaced repetition system associated with the subject has been updated since the review was created.

Consequences of This Change

Other than this being a breaking change to the API…

We are dropping SRS stage names (burned is here to stay though!). What are SRS stage names? They’re the names we use to identify the stages: Apprentice, Guru, Master, Enlightened, et cetera.

Given we plan on introducing more spaced repetition systems of varying size, keeping the naming system isn’t sane. It hurts since we like naming things and the names have been part of WaniKani since the beginning. But it is time for us to let go…

Let it go Elsa

Dropping the SRS stage names will be a slow transition. For the immediate timeframe, we are going to drop the names from being delivered on the API. These names are delivered in assignment.srs_stage_name, review.starting_srs_stage_name and review.ending_srs_stage_name.

See the Deprecation date section for the API timeline.

On the front facing application, we are going to work towards removing the names and/or replacing components of the sites with features which can support multiple spaced repetition systems. We don’t have any more details to share right now; it’s too early for us to share any concrete details. The changes we’re making are setting the stage for those later features.

Moving forward, we will be calling the stages by either their position (spaced repetition system (SRS) stage 9) or by their event if they have one (burning stage).

One other thing…

Steve Jobs One More Thing

Since we are doing breaking changes to API version 2 there is one more change we are going to sneak in.

@oldbonsai announced earlier in the week about backfilling a small subset of assignments in a legacy state.

Now that these records are following a consistent pattern with the other assignment records of using timestamps for the events there is no longer a purpose for the assignment.passed attribute. We will be dropping this attribute. To check if an assignment is passed or not do a check on a presence of a value for assignment.passed_at.

The deprecation timeline for this change will be on the same timeline as the spaced repetition system.

Deprecation date

We will be dropping the endpoint /srs_stages and the attributes assignment.srs_stage_name, assignment.passed, reviews.starting_srs_stage_name and reviews.ending_srs_stage_name on August 17, 2020. This is a full two months to make the necessary adjustments.

The new spaced repetition system data is now live on the API.

On the API documentation, we have added warning messages about the deprecation in the appropriate sections…

42 Likes

Could you please backfill level_progressions too? :eyes:

5 Likes

Aww, I liked the stage names.

60 Likes

I feel like the stage names will come back, since it’s as simple as someone making an extension to look at whether you’re between stage x and y to know what name should be used.

Of course I could be way off base.

6 Likes

The stage names are too ingrained with the veterans to go away for a long, long time. But I guess it will all depend on how the stages will be presented in practical terms on the site in the future. And we’ll just have to wait to find out what that’s going to look like.

Whatever the new world is going to look like, if the web site removes the stage names in its presentation logic, it won’t take long for a whole generation of new users to come along who are unfamiliar with the old names and won’t have a clue what we mean by “I just Guru’d a few more kanji tonight”.

I kind of expected this would happen. For my app, I have decided to just shoehorn the existing names into the new system for now. For new systems, my app justs adds or removes Apprentice and Guru sub-stages as needed until every stage has a name again. Beyond that I’m just going to await with great interest what this brave new SRS world is going to look like :wink: and adjust as needed.

I genuinely wish the WK team the best of luck in building that brave new world, but I am also going to stock up on popcorn and watch the forum drama that will be inevitable along the way… :popcorn: :popcorn: :popcorn: :smirk:

35 Likes

I’m not sure you grasped the point. There will be new SRS systems, such as the five-stage system in the sample with a passing stage of three.

In that system, there is no way to map all of the existing category names.

3 Likes

But in all seriousness you could just map less of them, or map multiple if I’m being cheeky :stuck_out_tongue_winking_eye: (say hello to guru-master)

1 Like

I’m still not sure where they are going with this. Will we be able to customize the SRS to our liking? Will I be able to say “I know this already” to a new card and it will appear less often/never? Will there be essential cards with more reviews and “side-cards” with less reviews because the vocab is rare?

10 Likes

No. This isn’t for your customization.

I am going to throw out an example (meaning this may or may not happen, not confirmed).

Radicals could be on a shortened spaced repetition system given it is very simple in nature. Like instead of the 10 stages it would take something less than < 10. And possibly even more accelerated.

39 Likes

The way things are set up every subject will have a fixed system that is the same for everybody. That means that everybody will use the same system for, say, 山. If the devs decide to put 山 in a new system with different timings, then that will apply to all users. So you won’t be able to adjust the SRS timings to your own liking, we all get the same timings.

But something like the “side-cards” you describe is definitely possible, although we won’t know WK’s plans in that direction for a while yet. This change is just to put in the foundations that will allow something like those side-cards in the future.

4 Likes

I’m going to miss the category names, I like having something title or alphabet based for this kind of thing rather than numbers. It’s somehow easier and more fun for my brain to wrap around. That being said, it seems like these changes come from a push to improve the system, so I’ll try to give it a fair chance when the dust settles in August. Thanks WK team for always trying to improve!

But I have one question: when it switches to numbered SRS stages, will there still be a color coding system? (I find joy in all the pretty colors in the item count bar)

18 Likes

Thanks @viet! I love this write up.

Yeah, I’ll miss the names, but I like progress more.

However, I have had this thought a lot recently (since I have little use for radicals anymore), so I am a big fan of this potential development!!

7 Likes

@viet you missed a golden opportunity to name the topic “SeRiouS Announcement”

I’ll see myself out now :sweat_smile:

41 Likes

Wait. What?

I thought this was finally a move to address the issues some (or even many?) people have with the current SRS timings, so that users could adjust the timings to suit their needs better. Well… looks like I was wrong :frowning:

7 Likes

This is going to address those issues, but not on an individual basis. They’ll be addressed for everybody. Individual customization was never the goal of this development.

4 Likes

I dunno. Something has to be there so we know how well we’re doing.
Guru means we know it pretty well.

“Stage 7” doesnt have the same flair.

Unless you want to number it like Mario. Stage 1-2. 時間の泳ぎ

14 Likes

The two-levels, instaburn SRS is a perfect way to implement a placement test, by the way. If you can correctly answer the item’s “lesson quizz”, it just gets removed.
(That’s what Kamesame is doing by the way :slight_smile: )

6 Likes

so if I understand this correctly, all this is about is allowing you guys to have more than the current three srs’s (which are basically just different variants of the same srs).

1 Like