Kaku - WIP Android Dictionary


#1

Hi all,

I been working on a side project for the past couple of months and just got the major feature working (the dictionary part, obviously). I thought I’d share my progress and see if it’d be useful to anyone else. I built this because I thought it’d be neat to be able to read RAW manga on a mobile device without spending 20+ seconds drawing out the kanji in google translate every time I didn’t know something. OCR Manga Reader kinda did it, but was limited to a self-contained app.

Anyways, here’s a demo: https://www.youtube.com/watch?v=Uq9AwxClY3g

Here’s a test APK of the latest build if anyone wants to play with it: http://fuwafuwa.ca/f/VRD7vniKmR.zip

Note that you require a Android 5.0+ device to run this app - this is a limitation of the Android APIs (and I can’t bring it to earlier Android versions) since MediaProjection was added in API level 21.

Installation instructions: https://github.com/Xyresic/Kaku

Please report any issues you find here: https://github.com/Xyresic/Kaku/issues

The project is at GitHub here: 
https://github.com/Xyresic/Kaku if anyone wants to build it for themselves. I apologize beforehand for no documentation - there’s a couple file dependencies (tesseract’s jpn.traineddata and jmdict.xml) that you need to drop into the app’s data directory on the sdcard (and a block of code that needs to be un-commented then re-commented to build the initial database).


#2

Hey, cool idea! Definitely continue working on it. What other features do you have planned?


#3

You read my mind. I wanted something like this. For a while I was using https://play.google.com/store/apps/details?id=com.niftygnomes.popupjapanesedictionary on my phone but it was such a hassle to copy and paste things. Your version definite is more convenient.


#4
Mempo said... Hey, cool idea! Definitely continue working on it. What other features do you have planned?
For now, my main priority is adding KANJIDIC2 to the database. After that, I'm going to be digging a bit deeper into Tesseract's library to include top OCR candidates to choose from by dragging the kanji around (as well as being able to edit manually in the case that the OCR is horribly off). This function seems to do what I need, so hopefully that won't be too bad:

/**
* Get a reading-order iterator to the results of LayoutAnalysis and/or
* Recognize. The returned iterator must be deleted after use.
*
* @return iterator to the results of LayoutAnalysis and/or Recognize
*/
public ResultIterator getResultIterator();


I also intend to hook into Android's clipboard events (just like popupjapanesedictionary) and provide a interface to search for entries manually. There also seems to be a thing called EPWING, but I honestly have no clue what that dictionary is (or where to even find it), so that's something for the future.

#5

Epwing is a format to save dictionary data. There are different versions of this format with different feature sets. Recent versions also support multi-media. Many Japanese dictionaries are (or have been in the past) available for purchase in this digital format. They can be used by opening them in a reader application that supports Epwing dictionaries.

So it isn’t a dictionary itself. But supporting it as a format would allow people to add their favorite own dictionary that they purchased somewhere or created themselves.


#6

I am not sure if those .idea directories should be in your git.


#7

You know you don’t need to draw in Google Translate right? It has a rather good OCR.


#8
Syphus said... You know you don't need to draw in Google Translate right? It has a rather good OCR.
I started work on this before Google started adding all their fancy contextual search/translation to Android and was definitely disappointed I didn't finish before them. I still think this would be useful, since Google Translate doesn't show kana or alternate meanings, but just translates.


Benkei-dono said... I am not sure if those .idea directories should be in your git.
 Yeah, I'm not sure either. I mostly followed https://intellij-support.jetbrains.com/hc/en-us/articles/206544839-How-to-manage-projects-under-Vers...

#9
Xyresic said...I also intend to hook into Android's clipboard events (just like popupjapanesedictionary) and provide a interface to search for entries manually. There also seems to be a thing called EPWING, but I honestly have no clue what that dictionary is (or where to even find it), so that's something for the future.
jakobd said...

Epwing is a format to save dictionary data. There are different versions of this format with different feature sets. Recent versions also support multi-media. Many Japanese dictionaries are (or have been in the past) available for purchase in this digital format. They can be used by opening them in a reader application that supports Epwing dictionaries.

So it isn't a dictionary itself. But supporting it as a format would allow people to add their favorite own dictionary that they purchased somewhere or created themselves.
EPWING technical information is sort of hard to find, so maybe I can save you some time if you decide to head down that path.  It's technically just an Electronic Book format.

EB Library is a source code package that supports EPWING packages.
It's kind of difficult to find the latest source: http://green.ribbon.to/~ikazuhiro/dic/ebu.html
There's also a Debian Linux-related version (maybe it's identical, I don't know): https://launchpad.net/ubuntu/+source/eb/4.4.3-9

The documentation is in Japanese, but it also has code examples.  (Documentation is in the ./doc directory after you compile)
Many EPWING dictionaries are JIS X 0208 (character coding), so you may need to convert to Unicode for use with Android.

#10
Xyresic said...
Syphus said... You know you don't need to draw in Google Translate right? It has a rather good OCR.
I started work on this before Google started adding all their fancy contextual search/translation to Android and was definitely disappointed I didn't finish before them. I still think this would be useful, since Google Translate doesn't show kana or alternate meanings, but just translates.


Benkei-dono said... I am not sure if those .idea directories should be in your git.
 Yeah, I'm not sure either. I mostly followed https://intellij-support.jetbrains.com/hc/en-us/articles/206544839-How-to-manage-projects-under-Vers...
 I'd recommend adding https://www.gitignore.io/api/android to your .gitignore since it stops all files that are not needed by android from being committed.

#11

Please keep working on it if you can!! :slight_smile: An app like this was exactly what I was looking for!


#12
rfindley said...The documentation is in Japanese...
 This sounds terrifying. I guess I'm going to hope I don't have to cross this bridge.

#13

I was literally searching for the longest time two days ago for something like this, and I happened upon this post on my dashboard when I went to do my review… Man, this is almost too convenient.

Xyresicさん、ありがとうございました!
 
Also, I did steps 1-6, but it was having problems “obtaining the screenshot” or something like that. A full reboot seemed to do the trick for me. 

EDIT: It seems as if that issue occurs when I disconnect the “chromecast” connection that is enabled while Kaku is running. How does the system Chromecast function work in relation to Kaku? Does it have to do with how you can cast your device screen to a Chromecast device?


#14
ichitaka said...EDIT: It seems as if that issue occurs when I disconnect the "chromecast" connection that is enabled while Kaku is running. How does the system Chromecast function work in relation to Kaku? Does it have to do with how you can cast your device screen to a Chromecast device?
This is because Kaku requires getting a screenshot of the screen programmatically, which Android only allows by its MediaProjection APIs added in Android 5. As far as I been able to test, there can only ever be one projection app active at the same time. Therefore any app that requires capturing the screen (i.e. Chromecast), would interfere with Kaku.

#15

Ah, ok. That makes sense.


#16

Just made a couple improvements:

  • Added KanjiDict
  • Show alternative words also deemed likely by the OCR engine by dragging the text
Demo: https://www.youtube.com/watch?v=Uq9AwxClY3g
APK in OP

I think the only major task left is figuring out how to deinflict the verbs now.

#17

That looks really cool and useful, thanks.

I’m also jealous of your programming skill.


#18

This could be really awesome.  I love how you can shape and move the box around anywhere on the screen for the OCR.  In the latest video example you had it over a spot with game dialogue.  This will be great for games and apps that have static text boxes, like RPGs, and could really revolutionize studying with other material.  


#19

Another small update:

  • Add 2 more additional swipe actions: edit and delete (delete isn’t very optimized yet and has some bugs)
  • Add cropped word images to both swap and edit swipe actions


Also, if anyone has downloaded and installed the apk, I would love to know what your thoughts were! I’m sorely missing data on performance, especially how stable it is on some older 5.0 devices.

#20

Good work there. I will try it later when I have some time.