I think watching with English subtitles is like training wheels… You’re going not going to get very far if you never take them off, and if you keep using them indefinitely they can instill some bad habits. From my experience, I know watching hours and hours of anime with subtitles taught me little beyond common “weeb” words like すごい and かわいい, because most of the sounds were just going in one ear and out the other. However, after I started paying more attention to the audio and forcing myself to listen, I started to pick up words here and there. Not very much learning for the time spent, but it was something. On the plus side, having a strong memory of a scene has helped me remember certain words more effectively than rote memorization. This may not work as well for people who have more visually-oriented memories.
I think my listening comprehension has definitely gotten a lot better since I “graduated” from English subtitles. These days I always watch Japanese shows and anime without any subtitles (unless they have Japanese subs available), and if I’m forced to have them on, I’ve trained myself to look at the center of the screen so I won’t read them. I only understand about 50-70% depending on the material (if I understand less than that I usually just call it quits), but it’s a lot more rewarding and it trains my ear a lot more than watching with English subtitles.
Well, this ended up being a very subjective post but hopefully it contributes something. For N5 level, I would probably suggest starting with a short video, or perhaps textbook audio (like from Genki) and getting to a point where you feel comfortable listening to and processing short sentences without relying on translating into English. (Or at least work on reducing how much you do so)