You might be able to think about it like this:
with a non-passive verb, the subject does the action and is marked by が
私が殺す - I kill
With a passive verb, the subject gramatically STILL does the action and is marked by が - but the action is having something done to them
私が殺された。- I was killed.
In that situation, you might want to know who did that something to them. So that’s marked with に:
私がゾンビに殺された。- I was killed by a zombie.
And as usual, parts of this, especially が, are often implied with a topic or context.
In the particular sentence,
今日あった男 に 心持ちほねっぽいねって言われたんだ
今日あった男 に signifies that the “was said” was done by the 今日あった男 / “today encountered man” / “guy I met today”
There’s no は, and no が, and I’m not 100% sure 私が would work with 言われた exactly, but since it’s a person speaking who sounds kind of ticked off, and the thing said by the 男 is a comment about someone’s appearance, we can infer that it was said to/about the speaker.
Speakers do tend to omit themselves as subjects, and the passive form can be used to describe stuff that happened that involved you in an abstract, negative way too.
to sum up:
I’d phrase it more like: “in the passive form, the subject receives the action. The action was done by the one marked with に” and separately “a speaker omitting a subject is often talking about themselves.”
It takes a while to click, but eventually at least it’ll feel a lot more intuitive, in the same way you can read “I was stabbed” without having to stop and go “wait! Who stabbed whom?! ‘I stabbed’… so I did it… but it’s passive?? so it’s opposite and I didn’t do it someone else did?!” just takes time. But maybe that explanation can help a little?
(sorry for the violent examples they just felt simplest in this context)
Incidentally, I don’t think you’re wrong to get confused about the two sense of に, I think in some circumstances it can create ambiguous sentences and other particles are sometimes swapped in for one of the two (the “targeting” sense or the “who the passive action was done by” sense) for clarity.