HELP! I Don't Trust My Friend With My Wife

My best friend and my wife work in the same bank. At times, he drops her home on his way back. Lately, they both have been travelling together quite often and that too very late. 

I have offered to pick her up when she gets very late but she asked me not to take any trouble because she could come with him. I am not suspecting anything but somehow it has started making me uncomfortable. 

Is this issue even worth thinking about or should I just forget about it?