My new cat won't let me pet him :(?

A co-worker of my boyfriend this adult cat wandering around his neighborhood. He decided he wanted to find the cat a home because he was real sweet and friendly. So my boyfriend and I offered to take him. When we first met him he was such a sweet heart, purring, loving every moment of attention he got. I have two other cats, a fixed male and female. When we brought the new cat home, we were sure to keep them separated. I had read up on how to properly introduce a new cat to your home. He was still being sweet, and very relaxed in the new environment. It was clear he's famliar with indoor living, as I know an all outdoor cat brought indoors makes their familiarity quite obvious. After a couple days we had to go out of town for a couple days, so we let him stay with a friend of mine since we didn't want to leave him alone, locked up for the weekend. My friend was thinking of adopting him but first wanted to see how her current cat reacted. When we got back, her cat was still being quite aggresive towards "Thor" so we took him back. Ever since then, he's completely changed. He doesn't like to be pet any more. He has since gotten to know my cats and gets along with them just fine, but whenever we go to pet him he just slinks or shy's away from our touch. I know my friend never just let him out with her cat, she kept them separate, and I know that she was kind to him, giving him attention and such. It's been 3 weeks since we brought him back to our house and he's not getting any friendlier. We believe he's fixed, hasn't been dislaying any habits of a not fixed male or female. Does anyone have any advice as to what might be wrong? Any recommendations for what we should do?