Game2Text - Learn and retain Japanese words from games

Hi everyone, this project is pretty personal to me because I started it as a simple web app for my friend who is learning Japanese by streaming games. :smile_cat:

Game2Text helps you better learn and retain Japanese words from games. It extracts text from games and throws it in a browser so you can look up words with browser extensions like Yomichan/Rikakikun and add them to Anki with AnkiConnect.

It’s not meant to translate the game (although it could be used that way), but as a bridge for quickly looking up kanji and Japanese expressions and add them (with game screenshots!) to Anki. You may already know similar apps that can do some of these things, but this is a unique many-in-one toolbox that works on both Windows and Mac. There 's a web version on the website that basically runs on any platform installed with Chrome.

New

  • Visual Novel Hooker: You can now extract game text from Visual Novels and games like Danganronpa with the built-in Texthooker.

  • Anki Integration: With AnkiConnect addon, you can create cards super quickly with the shortcut Shift-E and create beautiful cards in seconds complete with game screenshot and audio.

  • Game Script Matcher: Import a game script and you can use ocr to assist you to find the correct line of game dialog. More info here

  • Image Filter Profile: If the OCR result is poor, apply image filters by right clicking on the game screen. More info here

Highlights

  • Kanji Finder : It extracts game text with OCR and then you can hover the text for the Yomichan dictionary. You can also enable auto mode so you don’t have to go back and forth between the game and the app.
  • Game Screenshot: This is done through Yomichan’s {screenshot} field that attaches a browser screenshot when you add a new word to Anki. If the game stream in the app is distracting, just press the collapse button above to hide it. When you do need the game stream for the screenshot, press the same button to show your game. Screenshots are great for building context in Anki cards to improve retention.
  • DeepL Translate : DeepL Translate is from my experience the best machine translation engine for Japanese. It isn’t perfect, but it’s good for reference. There is a limit of 140 characters per request.
  • Texthooker Companion : In settings, enable Clipboard Inserter and it will show the contents of your clipboard. Paired with a Visual Novel Texthooker like Textractor, you can take advantage of this app’s translation feature and Yomichan/Anki integration.

Screenshots

Games I've tested

  • Persona 4 Golden - Almost perfect with OCR Space
  • Persona 3 Portable - Acceptable with Tesseract
  • Danganronpa - Windows: use built-in texthooker. Mac: almost perfect with Tesseract
  • Hatoful Boyfriend - Windows: use built-in texthooker. Mac: almost perfect with Tesseract & Light background image filter.

Games tested by community

  • Dragon Quest 11 - almost perfect with Tesseract
  • Unchained Blades 3DS - acceptable with OCR Space
  • Genshin Impact - works, but have to apply light background filter for low contrast environment sometimes
  • Astro’s Playroom - acceptable with Tesseract

Prerequisites

Required: Chrome, Yomichan

Optional: Anki, AnkiConnect

Hotkeys

Refresh: Windows: Ctrl-Q (Cmd-R on Mac)

Demo

See it in action

Download

Getting Started

Questions

Reply to this WK Post

Ask in our Discord!

42 Likes

Finally got around to trying this out! So far it’s great! Like, really great! Fantastic work!
8211f5c16da03ae93a39076f09658d7f

It doesn’t show an user interface for me. It seems like it’s running in the background but nothing happens.

I tried it some time ago but it didn’t recognize the text of the game I played. The text it showed me had nothing to do with the text in the game.

I just gave it ago and tried it over both a PS4 remote play for DQ11 and a steam game but nothing showed up in the translation box… any clues what I could be doing wrong?

Cheers, looks cool!

(Windows)

The OCR accuracy varies from game to game. There are various ways to improve the accuracy:

  • switch the OCR engine
  • apply image filter
  • match game scripts

I described how to do each of them here: Apply Image Filters - Game2Text

1 Like

You have to click the translation button.

Thanks I will try it again once I have some time.

This looks amazing! Well done for making this

Does anyone know if it is easy to get working on Linux?

I just wanted to both share my experience using this program and also bump this thread up because I think the program is quite helpful, but it took a while for me to discover it. Would be great if anyone who might benefit from this would learn about it now.

I’ve used the OCR with a couple games, first a test with Corpse Party and now pretty extensively with the first Ace Attorney game. In both cases, I’ve never managed to get the accuracy perfect, but I’d say it’s usually good enough, if you already have a somewhat intermediate base of knowledge to work off. Before making my Anki cards I frequently have to touch up sentences, removing a few garbage characters or replacing a kanji. Situations matter too – Ace Attorney likes to occasionally throw in red text, and that might as well be invisible to Game2Text.

All of that said, even with a small bit of jank, it’s allowing me to mine sentences for Anki and play this game far more comfortably than I’d be able to otherwise. I’m more than happy to put up with it because the alternative would just be, in all likelihood, either putting the game down for now or at least needing a separate source to do my mining with.

So, while I have the creator’s ear, thank you! You’ve helped me out quite a bit.

1 Like

Hi! I’m trying to figure out how to set Game2Text (and Yomichan) up with Danganronpa (and Little Busters). What’s this “built-in texthooker” - as opposed to Textractor like in Game2Text’s options? In the actual setup, do I have just the game and Game2Text open, and no other external other program?
What’s the Clipboard Inserter-option for, then?

I’m new to this whole texthooker-VN-playing-thing, so any tips would be appreciated.

The built-in texthooker is basically Textractor’s core program (also called command line program). It’s a copy of Textractor but without the user interface and extensions in the standalone Textractor program.

Textractor is bundled in the Windows Release of Game2Text, so you don’t need to install Textractor separately.

Yes, that would work. Obviously, you would need Anki open if you want to use Anki functionality.

It’s a convenient way to get text into the Game2Text program if you’re using an external text extraction app, whether it’s standalone Textractor with additional extensions not present in Game2Text, or other text extraction programs like Capture2Text.

When I first wrote the post, texthooking wasn’t included in Game2Text, so clipboard was the default way for texthooking to work. Now that it’s included, I should probably clarify the uses of the clipboard function.

2 Likes

Got it, thanks.
image
These <CLT>-characters I just have to live with? :smiley: I haven’t done much with Textractor, but I think there’s some way to filter, maybe? Would I need to use Textractor as an app and then the clipboard-functionality for that?

But still, it seems to work fine right now, thank you for creating this! Being able to use Yomichan for this is just awesome.
On another note, could you shed some light on the technical part of this, if you want to? I’ve done a bit of web-programming, but the app-capture-thing is new to me.

This looks great, will try it soon.
Is it compatible with Migaku, so that you can create cards for sentences with only one unknown word?

Good catch.

I haven’t tested enough visual novels with the textractor engine (which is based on another engine…based on another engine) so I haven’t implemented all the filters (html/xml tag filtering in this case) that are present in Textractor.

That being said, if you set “Remove Repeat Sentences” in advanced settings to “quick & brute force”, the repeated phrases like CLT and CLT123 in your given example should be gone. The reason I didn’t set “brute force” to default like Textractor is that there are performance hits, and there are VNs where brute force will remove actual text in the game (which is also a problem with Textractor). Of course, you can also use clipboard if those Textractor filters are needed for your game.

Actually, I wrote a blog post on this, and as more people got interested I decided to move this from a web app to a standalone app which eventually became Game2Text.

The screencapture API is called getDisplayMedia(), which is part of the MediaDevices’ interface.

1 Like

Very cool, thanks for the behind-the-scenes-link! :slight_smile:

In a first test right now, there’s still CLT and CLT4-tags in the output, with Quick + Brute Force. I’ll try using standalone Textractor and html/xml filtering in Textractor and see if that works.

That did work.

As a feature-idea, is it possible to use the captured gamestream directly as a source for the screenshot, instead of the browser-window-screenshot?
Maybe as an option, either the browser-screenshot, the captured gamestream (as an image that’s updated every time the recognition updates?), or that same image cropped with the recognition-border?
I saw that Yomichan also has a clipboard-image-field, maybe this option could use that.

If that’s not possible, I guess I could also just make my own custom screenshot with whatever screenshot-app I’m using, just before I add the card with Yomichan. Since those usually copy to clipboard, that would fill out the clipboard-image-field as well…

I spent some time on it lately and it’s more complicated than it suggests (width/height, resolution, following window coordinates, separate screens, and even occlusion). This will take some time to implement.

1 Like

Very cool, good to know :slight_smile:

I tested this with Shinymas. It can be a bit janky at times but overall I think it’s very nice. I have the original Shinymas scripts to double check when things don’t match up but I think this will help a lot when reading! I did notice when keeping the OCR on auto, it can spaz out a bit though.