Update
We went for coffee at work again today. I sent an email in the morning to several teams, his team included. He sent me a message afterwards and asked if I was at the office today. It turns out we were both there so he suggested going for coffee.
He came to my desk and then we went to the 'coffee room'. We talked a little longer than last time, we both laughed, he teased me a little and it was really easy to talk to him. He said his team is going for drinks after work next Thursday, and he said I should join them. I also noticed he touched my arm while he was talking to me. He also told me that because he's training several new hires, he'll be at the office more often now and we should get coffee or lunch again soon.
So I think I'll go for drinks with his team next week, I enjoy his company and I think we get along well. I could feel a little tension between us today. I'm still nervous about the group thing because I don't know people in his team so I hope they'll be nice.