LipSurf - WaniKani Voice Dictation. Review while you eat, do chores, etc

In 2017 two things happened: I started getting finger/wrist pain from all the typing I was doing (eg. with WaniKani) and I realized there were many times I wanted to do WaniKani reviews but I didn’t have both hands free…

“No problem!” I thought, and started writing a script to let me use voice dictation for my reviews.

I finished writing the script, wiped my brow and looked at the clock.
It was 2am… near the end of 2018.

And here we are today! I became a bit obsessed with the script and now it’s a very polished, customizeable and fully extensible Chrome extension that let’s you do anything with your voice (WaniKani reviews & lessons, browse reddit, scroll pages, click links, play videos, etc… you betcha buddy)

It has benefits:

  • Practice speaking your reviews (you should be speaking them outloud anyway!)
  • Get more comfortable speaking Japanese and confirming your accent
  • Reviews can be faster once you get the hang of it
  • Less hand/wrist pain and chance for injury
  • Do reviews while you do other crap (eg. eating, chores etc.)

Yeah, so I posted it on here for free, so others could try it out. The feedback consensus was “this is awesome, but can you fix my problem X?” and so I spent lots of time fixing and improving X, Y and Z. Well, as any developer knows too well, a free software tool eventually comes to a crossroad where it will 1. either die because nobody wants to support it anymore or 2. you exchange it for money and thus are incentivized to keep it alive and thriving.

I went with 2. It’s no longer free, and I hope people understand why and will support me. You can still use it for free forever, just not all the features.

→→→YOU’RE LOOKING FOR THE DOWNLOAD LINK←←←

Here’s a video guide:

Just a small part of the new features:

Those ふ, る, and く’s can be really challenging for native English speakers especially, practice and master them.

General Usage

  1. Install the extension
    https://chrome.google.com/webstore/detail/lipsurf/lnnmjmalakahagblkkcnjkoaihlfglon

  2. (optional) install the lightning mode user script so you don’t have to say “next” or “次” all the time.

  3. Do the tutorial for LipSurf (there’s even a special section for WaniKani)

  4. Go to the options and choose your primary English dialect (Australia, UK… etc.)

  5. Make sure LipSurf is activated (click to toggle it on/off from the chrome extension toolbar, or just say the wake-word “grasshopper” if you have Pro and the wakeword activated)

  6. Say “crocodile crab” or “wanikani” WHILE YOU ARE NOT ON A SPECIAL chrome://... page… IMPORTANT: you must be on a normal tab, like google.com or a LipSurf tab (options, tutorial etc.). This is a google-chrome extension API shortcoming.

  7. Speak your reviews in the appropriate language (Japanese for readings, otherwise some dialect of English)

    • Only the correct answer will get filled in… to prevent speech recognizer mis-hearings
    • Say “wrong” or “だめ” when you don’t know the answer (you have to manually mark any wrong answers)
    • (NEW in v1.8) click the live text (transcript) to make a correction or a custom synonym.
    • After 3 wrong answers (3 speech utterances) you will see the answer in the textbox, at this point you can decide whether you were right or wrong and say the appropriate command.
    • Say “correct” or “正しい” to manually mark a work as correct, if the speech recognizer is having trouble
    • (NEW in v0.5) say “えいご” to switch to English if in Japanese/reading mode and then you can spell out kana as you would on the keyboard. (eg. for しょうがつ just say s-h-o-u-g-a-t-s-u) and the answer will be inputted only if it’s correct. Super helpful for the short one-syllable kanji that recognition has trouble with, especially.
    • Say “info” or “じょうほう” to simulate clicking the info button for the current item
    • You can also spell out English words for meaning items
  8. Consider supporting the project! I really need support to keep the thing maintained and improving. I would also super appreciate reviews in the chrome webstore, if you can’t back the project in any other way but you think it’s cool.

Here’s the site that has more details:
https://www.lipsurf.com

Updates:
Most recent update: 4/30/2019 – v1.15.2 Added some Japanese homophones, allowing context window to stay closed as per user requests.
See more details and past updates here: https://www.lipsurf.com/updates/

86 Likes

What are you using for Japanese speech recognition?

The html5 speech recognition API. Works quite well in Google Chrome. Doesn’t work in Firefox, yet!

2 Likes

Looks great so far, but I got stuck when trying the help command. It doesn’t recognize close help any command anymore

edit: works again after turning it off and on again
edit2: I see now. You might want to mention this in the tutorial, if that’s what caused it

image

So what’s going to be included in the “pro” version? I am intrigued.

1 Like

@Kumirei I don’t think the auto-shut-off was your issue, because I posted this less than 20m ago :smile:

Maybe there was some background noise, someone else speaking, or the internet being temporarily down? Either way, if anything is awry turning it off and on again is the best best, although I haven’t had to do that in months myself.

The pro version features are not in stone yet, but I’m thinking: wake word, input for textboxes, and “pro” plugins which would be more advanced plugins for gmail and such that I support directly – as opposed to community plugins which will always be free.

Thanks for being the first person to try it out on WK!

Doing WaniKani so frequently and consistently is definitely exacerbating certain physical problems that I have, and has been causing me some worry. Thank you so much for thinking of those that struggle with physical limitations.

I’ll certainly be giving this a try! The garbage microphone on my computer will likely be a problem, but I’m excited to potentially have an alternative on days where studying conscientiously is actually the wrong choice for me, medically.

Thank you again for sharing your hard work!

13 Likes

This is amazing!!! I’m going to try it out when I get home from work tonight. Thanks for making this.

1 Like

Oh, haha. I wasn’t keeping track. I don’t think any of those were they case, since it worked just fine up until I opened help and nothing changed in between.

Will most likely get pro then. I’ve tried other programs like this but they didn’t work as well as this seems to do. Especially not Window’s native assistants, haha.

1 Like

@neicul :eyes:

3 Likes

@charliec364 @Helix

What great timing! I’ve been thinking recently how much I would like to have something like this! I’m excited to try it out :slight_smile:

2 Likes

Cool concept.

I tried it out. Unfortunately, I don’t think it is for me. I’m from northern england and it really STRUGGLES to understand me. I was saying next for few minutes (previous worked better). It be interesting to see if it would recognise Japanese words though i didn’t have anything to view.

I have a great clear dedicated mic also (Samson). I’ve always had trouble to Speech recognition (even with Google) - though Alexa seems to understand me ok (Amazon did great on that front).

I was just chuckling to myself about something similar.

I’m a non-native English speaker raised on British English. It seems to be quite lost about what I’m saying. :rofl:

It can’t recognize me saying “WaniKani” at all, it seems. The things that rolled across my screen that it thought I said had me laughing out loud quite a bit.

I only had two reviews ready. It took quite a while to get it to recognize me saying 使う, but 高さ went fine.

Edit: with the Japanese recognition; I’m not bashing the recognition system. ^^ I’m not claiming my Japanese accent is good enough for software to recognize everything at once.

1 Like

Voice control while eating could be difficult :smiley:

1 Like

image

It said 0 would mean it didn’t turn off

1 Like

@Omun You guys need to change the language to English (UK). That should work much better for you. It should be on-par with google. The default is English (US).

2 Likes

hmm OK, first bug. I’ll get that one taken care of in the next version. I think leaving some inactivity threshold on there is good though… you know it’s only for when you don’t say any valid command for that period.

1 Like

I can’t get it to recognize single syllable answers at all. Makes kanji reviews practically impossible.
Also can’t access help when it’s in Japanese. nevermind, of course you just have to say it in Japanese

Some of the short utterances are really hard. Exaggerating them sometimes works. I would just say “ただしい” if I knew it. Sometimes I say three quick obviously wrong things first, to see the hint in the placeholder and then decide “正しい” or “だめ” Feel free to email me miko@lipsurf.com we can have a dialogue that way too.