I want to say it’s possible because that’s very close to what I did. It’s not entirely straightforward though – I spent a good year and a half on WaniKani (reaching about level 30) before studying grammar at all, so in terms of kanji/vocab I was well above what you might consider N5, but I couldn’t read even the simplest sentences. Kind of a stupid approach, I know.
Anyway, the biggest thing that got me out of that rut was ploughing through Bunpro as fast as I could stand, and then reading a lot. I got to the end of the N3 material in about three months. It meant a lot of reviews at first but it slowed down eventually.
The most important thing (by my estimation) was that I started reading with a focus on volume, not concerning myself with understanding every little thing. I wrote a post about this in one of the reading club threads, but the gist of it is: you’re going to be studying a lot of new grammar points and the best way to fix them in your brain is to see them in the wild. You want to read as much as you possibly can to increase your chances of that happening; reading at a full sprint, always chasing down the next piece of familiar grammar and skipping anything that might slow you down.
Reading at low comprehension can be rough, so a good way to ease the transition could be to read the manga for an anime you’ve already seen with English subtitles, so you can still follow along based on your memory of the events in the show – I did this with ５等分の花嫁, which would have been far above my level if I was going in blind. Following a book club is great, but in order to be reading at a high enough volume you’ll probably need to read something else on the side as well.
Finally, “reaching N3” depends on a lot of factors, of which reading is only one. Don’t forget to train your listening comprehension as well. The above approach also works for anime, TV etc – in fact, it might even work better because the show goes on no matter how slowly you might read. Keep at it, even if it’s not obvious in the moment that you’re making progress. You will get there if you put the time in.