Don’t be frustrated. The more you use the language, the easier it gets to remember these things. There are no rules, and I wouldn’t try to rely on any particular trick because in the end, the goal is for it to just “feel” right because you’ve used/heard it enough.
What you can do is practice KaniWani and Kamesame to reinforce producing those readings, but other than that, keep going and it’ll eventually sink in. You’re early in the process, no need to rush.