From my perspective as a novice, the hardest part about Japanese is just the total package. It’s the dawning realization that nothing you’ve learned in your life as an English speaker will prepare you for Japanese.
When you start to learn Japanese, you start from square one. No vocabulary, no concept of what grammar is or how it works, zero ability to read anything.
I think that’s what makes Japanese fun, though. Everything is new and it feels amazing when things start to click.
I would disagree on the vocabulary, given how many loanwords there are in both directions.
And once you understand Japanese pronunciation, you can usually be at least technically using a Japanese word if you just Japanesify an English word. That has many pitfalls, but it’s often a lifesaver.
I would also disagree on the grammar, considering that the first thing I can think of is that in both languages the adjectives come before the noun 
(Second is no gendered nouns.)
I don’t know if I have anything new to add (but let me bore you anyway!). I have been learning Japanese little by little for 30 years now. I would have said until the last year that kanji was the hardest part, or at least for me the biggest stumbling block. But to echo what @MissMisc said at the beginning, WaniKani has been such a kanji accelerator, that at this point, I feel farther ahead with Kanji than with other aspects of the language. Not that I mind!
Many feel that ideally you should learn everything together so that your listening, speaking, reading, and writing (or to look at it a different way, grammar and vocabulary) all progress at the same level. In practice I don’t believe that often happens. Children learning their native language learn to understand speech before they can speak, speak before they can read, read before they can write. As for vocabulary and grammar, it starts rudimentary and expands with time, though input greatly outpaces output.
Japanese adds the twist that the alphabet itself (and here I’m defining alphabet as the set of symbols that make up the written language) is enormous and takes a very long time time to learn compared to languages with strictly phonetic alphabets. So reading anything beyond writing meant for small children (or language learners) requires a degree of effort that non-character based writing systems don’t require. And thank goodness there is at least the phonetic syllabary to bridge the gap.
Personally, I’m very happy to binge on the kanji taught in WK levels 1-60 before moving on to more extensive practice with reading and listening, and apart from learning to read the kanji taught here, the vocabulary has been essential to my progress in the language. As @jprspereira has said, the number of words one should know to feel reasonably literate feels endless. Before I started WK, I didn’t realize how much I didn’t know. Now I’m starting to get an idea.
To the nitty-gritty of whether Japanese grammar is easy or difficult, for me the most difficult part about the structure of natural Japanese sentences (as opposed to most textbook sentences) is that the lack of pronouns in Japanese disorients me. In English we absolutely rely on pronouns to keep track of who is speaking, who is the subject, and who is the object. In Japanese, there are many cues about who is taking up these roles, but pronouns rarely enter into it. To me that’s the trickiest part of Japanese grammar. So far.
Ehehehe. Yeah, when I learned that adjectives come before nouns in Japanese I almost fell over because there was finally something familiar.
Also, no gendered nouns is very nice. And no plurals, or noun cases. Japanese nouns spoil us with how simple they are.
I reset last year, just been relaxing and reading novels/manga while redoing WK to smash vocab into my head. Taking the slow road this go around. Before I reset I was over level 30 in less than a year and I have too much going on to do that to myself again.
I did the same thing, waiting until level 20ish until expanding into the rest of the language and I would say that it gave me comfortable vocabulary base, and made Mount Kanji much less intimidating to encounter.
My tutor, who generally teaches the children of Japanese expats, tells me she finds it much easier to explain the finer points of the language to someone who has a basic understanding of Kanji, because Kanji illustrates ideas and concepts understanding how it works makes up for Japanese not being related to English/Germanic/Latin.
I wouldn’t call myself an advanced student yet, but that has been my experience.
Have you read Koichi’s article on stroke order over on Tofugu? He teaches some basic rules to help you get the stroke order right most of the time.
It’s helped me get better at writing them, though I still think my Japanese handwriting is a mess. I try to remind myself that I’ve only been learning for a year or two, and my handwriting in English looked much worse one or two years in. 
Yeah, I know exactly what you mean. This is the second restart for me, made it to late 20s once.
Doing it very differently this time.
頑張ろう!
We gonna do it this go round!
edited for picture fix
For me the hardest part of Japanese is speaking it appropriately.
The hierarchy of Japanese society is very foreign to me (even after living and working in Japan for 8 years).
For example, after working all day, I get into a rhythm of speaking politely with 敬語 and 丁寧語 , but once I go hang out with friends switching to a less level of politeness is difficult. They’ve told me its off putting…
humble brag I think this only becomes an issue when people have higher expectations of you, i.e. you’re fluent.
“The rest” won’t necessarily “come easy”, but it will be easier, simply because you’ll be able to understand what the kanji represent, and how to read them. But being able to see kanji and know what they mean doesn’t necessarily help with things like grammar, pronunciation, listening, etc., all of which will bring their own obstacles.
Showoff. But idc, 'cause me and @jprspereira speak 8 languages, combined, if you include the language of love, so, take that…
Assuming at least 1 or 2 more of the languages you speak are Latin-based languages, and speaking 2 of them myself, I can confirm that Japanese grammar is actually easier than the grammar of those languages. But it of course depends on how advanced you’re willing to go. Because of the way Japanese is structured, Japanese sentences can be as simple as one verb – most other languages don’t allow this. But when you get into advanced tenses and conjugations, that’s where things begin to separate. Romance languages are more difficult, in my opinion, because most words have distinguished genders and their articles have quantities, whereas in Japanese, you can simply say 犬は大きいです。 OR 犬は大きいです。 (the exact same sentence) and have it either mean “the dog (1) is big” or “the dogs (multiple) are big”, with the distinction being made based on context. In Italian, for example, the same two sentences would have to be “il cane è grande.” “i cani sono grandi.” If either of the articles don’t reflect the right gender or quantity, the sentence would be grammatically incorrect.
At any rate, every language has its own difficulties. In terms of Japanese, listening is one of the most difficult parts for me. Reading has become easier and easier, though production (formulating my own sentences) is still a struggle.
EDIT: to answer the question in the title of your thread: no. Kanji learning is actually quite easy, when you stick to a system like SRS/WK. It may become difficult once you plow through WK and still need to learn more, because you’ll have to do it on your own, but if you even make it to that point, I think your self-motivation will carry you into the post-WK era quite nicely.
Thanks for your reply. I see most people here struggle with listening, that’s very interesting. I’m still at the beginning, so I don’t understand anything when I listen to Japanese yet (besides some of the words I’ve learnt via WK), so I’m curious to see if this will be the same for me once I’m more advanced.
And sorry if I came across as a showoff, I just mentioned that I’m experienced in learning languages and teaching them because I wanted to clarify that I’m not a stranger to learning new languages and I think that makes a big difference. But yeah, Japanese is not like the other ones I know, so it’s certainly gonna be a challenge.
Cari saluti
I’m going to say time management and knowing what to study and when is the most challenging thing. Not any one skill per se. Focus all your time on one skill and the others will suffer. Ultimately, if speaking doesn’t progress, for example, it will hold other skills from reaching sufficient competency. It will be harder to retain vocab if that vocab isn’t in the part of your brain for rapid recall.
I can’t say for sure this is the case for many people or not, but specifically what makes listening difficult with Japanese is amount of homophones and the number of kanji-compound vocabulary whose meaning are clearer when you actually know the kanji they are derived from in the context used.
Another feature that may make listening difficult is the flatness of intonation and evenness of the syllables. Specifically for English speakers it can be really difficult to follow because long utterances just sound like a blur.
Plus the structure can be a bit too much information to process before one can get to the main verb. Since noun modification always come before the noun, really complex relative phrases can be a bit difficult to process before the listener gets to the main verb. This can change the entire meaning of the sentence if one isn’t careful about following the flow of what’s being said.
There are other things about listening to Japanese that’s challenging, but I think those are some of the main points.
I agree
to learn you need to read, and thats is too boring to me
Oh, no worries, I was just joking. I am a proud showoff, I’ve based the majority of my empire off of it, just ask @CyrusS, my personal assistant. But having your background/experience in language-learning will help immensely, so I’m sure you’ll assimilate seamlessly after the initial hardships of learning the basics. Ma se hai le domande, fammi sapere! Sono contento di aiutarti caro nuovo amico mio
And what an empire it is…
camera pans over to my empire:
Well, I’m still working on it but you get the idea…