LipSurf - WaniKani Voice Dictation. Review while you eat, do chores, etc


#1

I’ve been working on this beast for over a year. It’s voice control for Google Chrome and I’ve added a WaniKani plugin for it. So now you can dictate your reviews. It’s especially helpful for those with disabilities that affect their typing.

Here’s a video guide:

Just a small part of the new features:

Works decently for testing your accent, too. Those ふ, る, and く’s can be really challenging for native English speakers especially!

General Usage

  1. Install the extension
    https://chrome.google.com/webstore/detail/lipsurf/lnnmjmalakahagblkkcnjkoaihlfglon

  2. (optional) install the lightning mode user script so you don’t have to say “next” or “次” all the time.

  3. Do the tutorial for LipSurf

  4. Go to the options and choose your primary English dialect (Australia, UK… etc.)

  5. Make sure LipSurf is activated (click to toggle it on/off from the chrome extension toolbar)

  6. Say “crocodile crab” or “wanikani” WHILE YOU ARE NOT ON A SPECIAL chrome://... page… IMPORTANT: you must be on a normal tab, like google.com or a LipSurf tab (options, tutorial etc.). This is a google-chrome extension API shortcoming.

  7. Speak your reviews in the appropriate language (Japanese for readings, otherwise some dialect of English)

    • Only the correct answer will get filled in… to prevent speech recognizer mis-hearings
    • Say “wrong” or “だめ” when you don’t know the answer (you have to manually mark any wrong answers)
    • After 3 wrong answers (3 speech utterances) you will see the answer in the textbox, at this point you can decide whether you were right or wrong and say the appropriate command.
    • Say “correct” or “正しい” to manually mark a work as correct, if the speech recognizer is having trouble
    • (NEW in v0.5) say “えいご” to switch to English if in Japanese/reading mode and then you can spell out kana as you would on the keyboard. (eg. for しょうがつ just say s-h-o-u-g-a-t-s-u) and the answer will be inputted only if it’s correct. Super helpful for the short one-syllable kanji that recognition has trouble with, especially.
    • Say “info” or “じょうほう” to simulate clicking the info button for the current item
    • You can also spell out English words for meaning items
  8. Consider supporting the project! I really need support to keep the thing maintained and improving. I would also super appreciate reviews in the chrome webstore, if you can’t back the project in any other way but you think it’s cool.

Here’s the site that will soon have developer documentation:
https://www.lipsurf.com

Updates:
11/18/2018 v1.5.0New English<->Japanese dictionary lookup commands.
…many updates were had in between…
9/26/2018 v0.11.1 – Performance improvements. Fixed bug that would prevent LipSurf from working if too many tabs were open.
9/13/2018 v0.9 – Added “show more info” and “last 10” command. Fixed issue transcribing elongated katakana vowel to hiragana. Added homophones.
9/5/2018 v0.8 – Allow fuzzy matching using Levenshtein distance in English (eg. “subjective” will be accepted for “subjectivity”). Fixed issue where recognizer would stay on in certain scenarios after using WK. Added more number handling (should be all covered now?) Added a developer mode and a way to add custom plugins…(even outside of WK!) docs coming very soon!
9/2/2018 v0.7actually fixed issue with WK overlapping commands… tested better this time.
8/31/2018 v0.6.1 – Fixed issue where when wanikani answer had another command name in it… it would execute the other command. Now if you have the kanji 求人 and you say “help wanted” it won’t open the help screen unless you are a very slow talker :stuck_out_tongue: Also there’s no longer a need to disable the reddit commands if you were having trouble when spelling words ending in “r-u” that won’t open reddit anymore thanks to the same fix. Lastly, answer commands are given priority so if you are doing a kanji that happens to be “正しい” , “次” etc. (a built in WK command) they should still work when you have reviews!
8/14/2018 v0.6 – Fixed issue where live text would show but commands would not be executed. Fixed issue where LipSurf would break after any network issue – sometimes caused by saying many things while LipSurf is activated. Fixed issue with setting auto. off timer to 0 not working. Added English homophones: kay->k, okay->ok, are->r, you->u. Fixed issue where toggling LipSurf on WaniKani would sometimes cause the wrong language to be selected. Fixed issue where LipSurf would stay on if it was shutdown while on a Japanese reading for WaniKani. Fixed issue with recognizing WaniKani vocab/kanji with numbers. Added many default homophones for WaniKani. esp. for Romaji -> Japanese. Fixed issue with apostrophes, hyphens in WaniKani answers.
8/5/2018 v0.5.2 – Fixed issue where spelling out English answers might not have worked. Simple handling of numbers that are in WaniKani – 1-10, 42
8/5/2018 v0.5.1 – Fixed issue with spacing not showing up with multi-word vocab answers. Made special commands show as Kanji or Katakana if it’s a preferred reading instead of hiragana (正しい、次、情報 etc.) Fixed issue where sometimes next would be clicked prematurely. Saying “back” is now safe – it won’t quit your review session.
8/4/2018 v0.5 – Fixed the worst of the bugs that were mentioned by users here. Thanks for your input! Added language switching to make typing kana in English possible (see bottom of 6 above)


The New And Improved List Of API and Third Party Apps
#2

What are you using for Japanese speech recognition?


#3

The html5 speech recognition API. Works quite well in Google Chrome. Doesn’t work in Firefox, yet!


#4

Looks great so far, but I got stuck when trying the help command. It doesn’t recognize close help any command anymore

edit: works again after turning it off and on again
edit2: I see now. You might want to mention this in the tutorial, if that’s what caused it

image


#5

So what’s going to be included in the “pro” version? I am intrigued.


#6

@Kumirei I don’t think the auto-shut-off was your issue, because I posted this less than 20m ago :smile:

Maybe there was some background noise, someone else speaking, or the internet being temporarily down? Either way, if anything is awry turning it off and on again is the best best, although I haven’t had to do that in months myself.

The pro version features are not in stone yet, but I’m thinking: wake word, input for textboxes, and “pro” plugins which would be more advanced plugins for gmail and such that I support directly – as opposed to community plugins which will always be free.

Thanks for being the first person to try it out on WK!


Wanikani + Speech Recognition =?
#7

Doing WaniKani so frequently and consistently is definitely exacerbating certain physical problems that I have, and has been causing me some worry. Thank you so much for thinking of those that struggle with physical limitations.

I’ll certainly be giving this a try! The garbage microphone on my computer will likely be a problem, but I’m excited to potentially have an alternative on days where studying conscientiously is actually the wrong choice for me, medically.

Thank you again for sharing your hard work!


#8

This is amazing!!! I’m going to try it out when I get home from work tonight. Thanks for making this.


#9

Oh, haha. I wasn’t keeping track. I don’t think any of those were they case, since it worked just fine up until I opened help and nothing changed in between.

Will most likely get pro then. I’ve tried other programs like this but they didn’t work as well as this seems to do. Especially not Window’s native assistants, haha.


#10

@neicul :eyes:


#11

@charliec364 @Helix


#12

What great timing! I’ve been thinking recently how much I would like to have something like this! I’m excited to try it out :slight_smile:


#13

Cool concept.

I tried it out. Unfortunately, I don’t think it is for me. I’m from northern england and it really STRUGGLES to understand me. I was saying next for few minutes (previous worked better). It be interesting to see if it would recognise Japanese words though i didn’t have anything to view.

I have a great clear dedicated mic also (Samson). I’ve always had trouble to Speech recognition (even with Google) - though Alexa seems to understand me ok (Amazon did great on that front).


#14

I was just chuckling to myself about something similar.

I’m a non-native English speaker raised on British English. It seems to be quite lost about what I’m saying. :rofl:

It can’t recognize me saying “WaniKani” at all, it seems. The things that rolled across my screen that it thought I said had me laughing out loud quite a bit.

I only had two reviews ready. It took quite a while to get it to recognize me saying 使う, but 高さ went fine.

Edit: with the Japanese recognition; I’m not bashing the recognition system. ^^ I’m not claiming my Japanese accent is good enough for software to recognize everything at once.


#15

Voice control while eating could be difficult :smiley:


#16

image

It said 0 would mean it didn’t turn off


#17

@Omun You guys need to change the language to English (UK). That should work much better for you. It should be on-par with google. The default is English (US).


#18

hmm OK, first bug. I’ll get that one taken care of in the next version. I think leaving some inactivity threshold on there is good though… you know it’s only for when you don’t say any valid command for that period.


#19

I can’t get it to recognize single syllable answers at all. Makes kanji reviews practically impossible.
Also can’t access help when it’s in Japanese. nevermind, of course you just have to say it in Japanese


#20

Some of the short utterances are really hard. Exaggerating them sometimes works. I would just say “ただしい” if I knew it. Sometimes I say three quick obviously wrong things first, to see the hint in the placeholder and then decide “正しい” or “だめ” Feel free to email me miko@lipsurf.com we can have a dialogue that way too.