The thing is, all these different readings are used in different words. There’ll be many times when you feel like you’re guessing a reading, but with continued use of the SRS system, the reading will just stick. I know vocab using 上 (and other kanji like it that have a huge number of readings) can seem overwhelming, but it also means you get to see that kanji being used in many more different ways than other kanji. Over time, the sheer amount of exposure makes it a lot easier to figure out what reading to use.
(And then you’ll have kanji like 太 that have one on’yomi reading you ever need to remember and you’ll consistently get it wrong for six months.)