Like… halfway through Genki II, and with planning and the right vocabulary, you can probably navigate Japan with help from strangers with just Genki I. You need to know verbs and a few conjugations, use basic prepositions, use particles, etc.
If I remember correctly, in conjunction with the Genki books, I was getting very basic conversational skills and vocabulary between level ten and twenty on Wanikani. Sort of. I can say that because maybe a quarter to half the Wanikani vocabulary at that stage was also in Genki.
If Genki isn’t your jam, I suggest you try Japanese: the Spoken Language. It’s a little dated, but I got a lot of mileage out of its pronunciation advice in the beginning of the first volume, and it seems to be written more along the lines of how you want to study.
I admire your determination, but Japanese is reaaaaally different from English. It will help a lot if you study the basics enough to understand the basic structure of a sentence and some starting common verb conjugations. I know because I started learning Japanese through anime. I absorbed some common polite phrases at the very beginning, but it took grammar to understand where one word started and ended and what order the words were in and why, and what words had the same root even when the conjugation was different. A little grammar helped me absorb a whole lot way faster than I could without any of that help—it jumpstarted me to being able to benefit from unscripted listening/speaking practice. Otherwise…I could listen and listen and listen, but I wouldn’t have the tools to deduct and decode what’s being said or what part of speech a word belongs in.
Also, remember, Japanese people usually hesitate to correct you. If you use casual form instead of polite, or use the wrong particle, they might not point out the difference if they figured out what you were trying to say. Then again, they may be able to tell you what’s wrong, but they might not be able to explain why, which is the part you actually need to know to remember.