It may be confusing and hard to look up in a dictionary because it’s partially in kana, but きみ悪さ is actually 気味悪さ (きみわるさ), the “noun-fied” version of 気味悪い, which means something along the lines of “creepy”. So that last clause is just saying that there are times when you feel some creepiness from Kazuo.
That’s how I see it too.