Of course, everyone’s experience may be different, but my own experience disagrees with this point of view. I did bring your argument to my (Japanese) wife who responded, “Our brains just don’t work that way.” In other words (and she clarified thus) that in her opinion it’s far easier for Japanese people to make sense of Japanese written in hiragana than in romaji, just as a general point.
I’ll add to that, since we’re talking specifically about handwriting, that unless your penmanship in roman letters is extremely good, do not be at all surprised if Japanese people have a hard time reading your writing. (Likewise, I often have a hard time reading Japanese people’s hiragana). This is because we grow up reading and writing our own and other people’s handwriting. We know what is generally accepted as a divergence from printed text and what isn’t. Most Japanese people do not have very much experience reading roman handwriting and find it much harder to interpret.
It’s easy to underestimate our own linguistic biases, and thus convince ourselves that romaji is easy for most Japanese people to read. I think you’re overestimating the extent of its use in Japan. A word or two here and there for stylistic purposes, but whole blocks of text meant to actually be read and understood?
I’ll give you a (completely unrelated, but I thought interesting) example of how perception is affected by one’s own language. My wife is quite fluent in English and has lived in Canada for almost 30 years now. But this week was the first time she quite grasped that in English, the word “water” doesn’t necessarily mean “cold water”. Because in Japanese, みず absolutely is cold. Hot water is ゆ, and lukewarm water is ぬるまゆ. So she assumed all these years that the same must hold for English, that unless you add a qualifier, water is cold by definition. So she said something like, “How long do you take to cook hard-boiled eggs if you start from water?”
Anyway, I know that was way off topic, but in conclusion, my point is that writing Japanese in romaji makes some pretty big assumptions, and I wouldn’t be surprised in the slightest if many Japanese people aren’t able to understand it or don’t bother trying to read it in the first place. Japanese: it really is a whole other language.