You can certainly learn the language with romaji, especially if you don’t care about reading and writing. It was the standard approach for a very long time, up until perhaps the 1980s, and it had some advocates even after that (eg the authors of Japanese: The Spoken Language) because it lets you get on with mastering pronunciation and grammar without getting tripped up by the writing system.
But I think for most students of Japanese now romaji are considered unnecessary at best, because hiragana really aren’t that difficult to learn to read and write. This is where Japanese has an advantage over Chinese: it has its own sound based system which is a workable fallback when you don’t know the kanji yet and which can be used to annotate the pronunciation. Chinese traditionally didn’t have anything like that, which is why pinyin is useful to fill that gap.