Well, If I remember correctly (and I'm fairly sure that I do.. that box was a quite a nightmare for me at times) the Matrox Triplehead2Go Digital Edition could do no more than
[email protected], was unable to do DX10/11 (@5040x1050), and it was a battle just to get 5040x1050 to work.
This is due to the bandwidth limitations on a single DVI output or something along those lines, and as such, you will never be able to do 5740x1200 with the Triplehead2Go.
What suiken_2mieu is referring to may also be true, but even if you were able to get past 3849x1024, you wouldn't be able to do more than 5040x1050, with the TH2Go Digital Edition.
So, unless you want to downsize your monitors, or play at a lower than native resolution, you need to do either SoftTH, Eyefinity, or Nvidia Surround to achieve such high resolutions.
Hopefully you get that RMA'd card back soon! :doh
Good luck mate..