For me, it depends a bit. I expect to struggle a bit more recalling kanji/vocab I learned recently in a level. But for a kanji that I learned a while ago and have practised repeatedly since; not knowing it swiftly indicates it didn’t stick well enough.
I want to be able to read and recall Japanese, and taking a minute or two to work out one word in a sentence is not a functional level of knowledge for me. If it takes too long I’ll let the SRS do its thing: I’ll take the downgrade to make sure the word pops up in reviews more again.
I don’t want to make mistakes, so having to grudgingly accept a red error screen is a bigger motivation for me to improve that kanji than if I take a long time to pass it. If I know it that poorly already, it’s unlikely I’ll remember after an even longer SRS interval.
If I don’t know a word, I’ll name the radicals in case that brings back the mnemonic, but I really won’t take longer than maybe 10 or 15 seconds before deciding that my recall is too weak. I’m most lenient on new words, since sometimes that act of mentally banging through the mnemonic helps form a recollection (I won’t know a word, but I’ll remember I didn’t know last time, either, and that somehow leads me to remember it? Sure, brain.)
And of course it depends on what your Japanese goals are. If you’re planning to sit tests, you have to practice very swift recall if you want to have enough time to finish. Otherwise it’s important to focus on what you know works for you and how you learn best.