Have you ever felt bad for not liking a person?

My roommate (random rooming assignments are worse than AIDS, for the record) is an... interesting individual. He is far older than the average college freshman for a variety of reasons that I won't bother to get into here.

Anyway, after yelling at him several times because he didn't understand the concept of how borrowing something works (i.e. you give it back), and how my things are not also his things, he's been trying to be less of an idiot and a better person. This should be good news, yet it makes me even more angry for some reason. Probably because I am trying to find reasons to hate him, and he is giving me less. He always wants to hang out and whatnot. but I always turn him down and never, ever invite him anywhere with me. For those reasons, I feel kind of bad for not liking him, but the fact that I have to live with someone I could never fully trust is not a good feeling.