yeah! After covering N5 and N4 grammar I’d say you could give it a shot! worst that can happen (as happened to me when I first tried to read it quite a while back) is that you decide that it’s too much of a struggle and then you can put it to one side and try again later. And it will feel cool when you do come back later and find that you can read it so that’s still not a bad outcome!
I’d say for me (it might be different for others) what stopped me my first time through was mainly that I had such low reading stamina for reading in Japanese that the first chapter took aaaaaaages. I then switched to reading manga for a while and at least partly due to the last of these challenges, I went from a a chapter of manga feeling like a lot to read, to being able to (depending on the manga ofc) being able to read a volume of manga in a day if I was in the mood to do so. Then when I switched back to trying to read novels I found that it felt a LOT more manageable because I was less tired out just by the experience of reading in Japanese. So what other reading you have done in Japanese might also be a factor in how you find it? good luck with it! I’d say definitely get the audiobook if you can as it’s really good.