Most definitely, but N3 level is already a good starting point!
I think it’s a common issue and many of us went through this. I can honestly say that I started understanding anime only very recently, despite watching it (English subtitles, original audio) for a couple of years, and I think the biggest factor was actually getting started with Tobira which, despite being a textbook, contains more reader-level Japanese.
Based on that and the other things you wrote, I would recommend reading and watching tons in Japanese. It’s going to be a struggle at the very beginning, but if you don’t give up, you will succeed 
. I started reading my first book in Japanese a couple of days ago and it’s an absolute pain, but the fact that I can picture what’s written in the book and slowly work through it, definitely brings me joy. And it’s only going to get easier!
A good source of books seems to be Aozora Bunko. Here’s the main thread where the books have been sorted by WaniKani level. The level is also a bit of an indication of the target audience, because less kanji-dense books are likely for younger audiences:
So thumbs up and don’t give up!