Heres a few tips:

If using 10.6.8 ATI 577- will not work (maybe only 1 layer playback) - drivers are broken
you have to use an NVIDIA card:
Quadro 4000 / GTX 680

Don't use ProResHQ - use LT or Proxy - much better data rate.
You should also ensure that the content width is exactly divisibly by 16 and the height is an equal number.

The ideal setup would be 3x same hardware revision (4,1 or 5,1) MacPros running 10.8.4 with GTX680 MacEdition. (PC edition also works)
Get rid of the matrox DH2G and simply use the 2x DVI outputs to run the projetors (and if really necessary - one of the other for your desktop) - but you should be able to do this with just 2 outputs and no programming screen.

If you need a control computer - use a laptop or mac-mini to run a cuelist to trigger presets on the others. - Or login to the webserver of tone of the MacPros for control.


S