The 50s are all fast levels (3.5 days), so it makes sense to see diminishing benefits.
(edit: probably not, they have just as many Kanji, just less common ones)
(not my graph obviously, it’s jprspereira’s)
There are lots of factors at work here. Many people have a lot of prior knowledge. Some may have good memory.
But i think what’s most important is how you study. I take a long time during lessons to make the item memorable, internalizing mnemonic, meanings and readings, thinking how it could be used in sentences, sometimes creating multiple mnemonics myself. I’ve found a lot of better mnemonics for many items. If you don’t do that, you will spend this time reviewing the item multiple times instead, which isn’t necessarily worse, it saves you creating mnemonics etc. In the end, everyone will have in some way or other spent a certain amount of time to do the learning. How much time that is is not too important to me 
but of course it’s interesting where people encounter ‘walls’, as you say.