Really broadly speaking, there’s two relevant big important sense of the particle に, like @TheCodingFox phrased as “target for an action” and indicating who or what the “action is received from.”
(disclaimer, I don’t practice actual production so these sentences and my explanation might all be completely wrong)
The trick to distinguishing the two (and what, believe it or not at the moment, will start to become unconsciously absorbed over time with practice reading), is getting used to the different forms of verbs.
examples:
1.(私が)ボブにサンドイッチを投げた。= I threw the sandwich to Bob.
2.(私が)ボブにサンドイッチを食べさせられた。 I was forced to eat the sandwich by Bob.
In sentence 1, I, the subject, 投げた’d. 投げる can take に (I think) to mark to who/where I threw, so it slots in just fine. I threw to Bob.
In sentence 2, I, the subject, 食べさせられた’d, so I was made to eat. You might wonder who was the one who did that to me, and it’s marked by に, so Bob made me eat.
In English, especially learning it first as “towards” (or “in”) because that’s simplest, I think starting out this is really confusing, because it feels like “I threw a sandwich AT Bob” and “a sandwich was thrown BY bob” would be two opposite things, so it’s weird that に could be used to convey them both.
And especially because we’re not used to the passive form, when that’s in play it feels like に is marking the one DOING the thing, so why isn’t it the subject?
But the key is a passive verb isn’t about doing, it’s about being affected. The action the subject is doing is BEING affected, so it’s still the one DOING the action, the action is just… to be affected. The に part in the sentence is just an (optional) tipoff about who or what did the affecting.
So in both senses, if you want to try to unify them in your head, に is marking an especially important 3rd thing. It might not be the subject or the object, but it’s important to the sentence nonetheless, either the target informing the direction of the verb, or the thing ultimately causing or supervising the verb, etc. depending on what the verb happens to be.
One other thing to mention is that these two meanings CAN be ambiguous (to some extent). What if both cases happen - After all, maybe Bob would make me throw the sandwich to someone else!!
And while again, I may be wrong and would be happy to be overruled, I think you’ve spotted a case:
先生に ゆっくり話してほしいです
may mean you would like the teacher to speak slowly, but it could also maybe mean you want someone else to speak to the teacher slowly. (I think… in practice one may be truer than the other)
It could be disambiguated with context, (for example if you were asking the teacher this, it wouldn’t be ambiguous because there’s no third person),
or by using different words, for example:
先生に対してゆっくり話してほしいです。
Would, I think, unambiguously mean that you want someone else to talk slowly to the teacher. (because に対して has the “with/at” sort of sense but not the “marking the one you want the ほしい from” sense)
(and probably even just と would work)
(or if you wanted to rephrase to keep the “wanting the teacher to speak slowly” meaning you could use ください if you were talking to him and drop the 先生に, or you could pick a verb that doesn’t take に as well, etc.)
(but what one would or should actually say in different contexts is not something I would know at all so don’t go run off saying any of these sentences)
Mainly I want to convey that you have spotted two meanings that can be ambiguous, but it’s not so unintuitive as it may seem from an English perspective. With time and practice reading (and it’ll be really really satisfying when this happens), you’ll see a verb and not have to sit and piece together who did what anymore, and that’ll make the “wait what is this に doing?” feeling go away at least (…mostly).
