Hello!
Provided you observe neither L1/L2 transmission nor queueing errors at the switches, retransmits may indicate a queueing problem within server's OS, which I suppose is Linux. Default Linux is typically not the optimal one for handling such high amount of traffic, so it needs to be tweaked, involving a lot of kernel and drivers performance analysis and tuning (I'm assuming you've already performed some basic tuning, like unloading iptables kernel modules).
Kindly let me know if you want me to do this job for you.