Chatting with colleagues is very annoying.

Calm down, when you chat with him, you must mostly talk about people and things that are closely related to you. Maybe these people and things make you angry, or maybe your colleague's tone or way of speaking makes you unbearable. In short, don't pay too much attention to his opinion, so you may be less angry.