The chances of having any degree of fluency without studying grammar is practically zero. Unless you can immerse yourself so completely that you learn like a native Japanese child (i.e. you live in Japan and have to survive with literally no English / other language backup), you don’t really have much choice but to study the grammar rules explicitly. And even if you could do that, it sounds inefficient and painful.
Since Japanese sentence structure is so different from most other languages, I think you’ll find that a little grammar can go a long way. Many people formally study grammar through N5 and N4 (and sometimes N3) before just going 100% to “pick it up from exposure” / looking things up as needed. But you really do need the fundamentals for this to be possible. If you don’t have that and just immediately jump into reading or listening to native content, it’ll just sound like gibberish.
Only you can really decide when you’re ready to jump into the deep end. One way to test your skills would be to join the Absolute Beginner Book Club and see how well you can follow along.