# Looking for a dataset of Japanese words by frequency

**URL:** <https://community.wanikani.com/t/looking-for-a-dataset-of-japanese-words-by-frequency/50225>\
**Category:** Resources\
**Created:** [March 20, 2021, 3:45pm UTC](https://community.wanikani.com/t/looking-for-a-dataset-of-japanese-words-by-frequency/50225 "2021-03-20T15:45:36Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![polv](https://sea1.discourse-cdn.com/wanikanicommunity/user_avatar/community.wanikani.com/polv/32/572691_2.png) [@polv](https://community.wanikani.com/u/polv)\
**Post date:** [March 16, 2022, 7:23pm UTC](https://community.wanikani.com/t/looking-for-a-dataset-of-japanese-words-by-frequency/50225/4 "2022-03-16T19:23:34Z")

</div>

About **“vocabularies”** , I feel that I depends on tokenizers, as well as searching methods.

Recently, I tried to search PyPI on Tanaka Corpus (I misremebered for that [InnocentCorpus](https://community.wanikani.com/t/anki-word-frequency-inserter-learn-most-common-words-first/53407)), I found [toiro](https://pypi.org/project/toiro/), which includes downloader for other resources.

Actually, it doesn’t even answered Tanaka at all 🤣

```auto
from toiro import datadownloader, tokenizers

# A list of avaliable corpora in toiro
corpora = datadownloader.available_corpus()
print(corpora)
# => ['livedoor_news_corpus', 'yahoo_movie_reviews', 'amazon_reviews', 'chABSA_dataset']

available_tokenizers = tokenizers.available_tokenizers()
print(available_tokenizers.keys())
# => dict_keys(['nagisa', 'janome', 'mecab-python3', 'sudachipy', 'spacy', 'ginza', 'kytea', 'jumanpp', 'sentencepiece', 'fugashi-ipadic', 'tinysegmenter', 'fugashi-unidic'])

```

About **Innocent Corpus** , apparently, [Yomichan / AnkiConnect creator cached it.](https://foosoft.net/projects/yomichan/)

Furthermore, [ja.wikipedia.org can be downloaded](https://ja.wikipedia.org/wiki/Wikipedia:%E3%83%87%E3%83%BC%E3%82%BF%E3%83%99%E3%83%BC%E3%82%B9%E3%83%80%E3%82%A6%E3%83%B3%E3%83%AD%E3%83%BC%E3%83%89), although I don’t exactly know how you should parse it.

I have always known this another way - [Python’s `wordfreq[cjk]`](https://pypi.org/project/wordfreq/). However, even with this, I don’t feel it is that accurate…

---

_[View the full topic](https://community.wanikani.com/t/looking-for-a-dataset-of-japanese-words-by-frequency/50225)._