Welcome!
I’m probably mistaken but I get a sense like
“why am I always the one to have what I say examined/doubted?”
Yup. As usual, after checking in with my daughter, I’m completely wrong. 
俺の方ばかり見て by itself is a clause meaning “<someone> keeps looking in my direction” or “someone keeps looking at me and only me”.
So it’s apparently something like: “the person giving out (announcing?) warnings keeps looking in my direction.”