How would you know which way to pronounce it if you saw it on a sign, for instance?
You wouldn’t. That’s one of the big reasons why people who speak European languages find Japanese so difficult to learn. I’ve seen it called the chicken-and-egg problem.
To wrap your head around the difficulty, you have to understand a bit of the historical context. Kanji are imported from China. When they were brought over, the Japanese kept a rough approximation of how these words were pronounced in Mandarin. That’s the on’yomi reading.
The thing is, the Japanese imported kanji for concepts they already had words for, and they often kept these words with their native (or kun’yomi) readings. That’s why the same kanji will often read differently when used in vocab.
Tofugu has a must-read article on how this Franken-alphabet came to be.
That’s why 牛 is ぎゅう sometimes and うし other times. About the only rule I’ve seen is that when a word has more than one kanji, it’s usually all on’yomi or all kun’yomi – though even this has exceptions.
This is why Japanese school students spend a lot of time simply copying kanji by hand. It’s why furigana exist – to help foreign readers and young children. Outside of kanji, Japanese is actually a pretty approachable language. The grammar is very regular, and word order is relatively predictable. Compared to Russian or English, verbs are a piece of cake in Japanese.
But with kanji, you have the boss fight of languages for anyone who has grown up speaking a European language.