Particularly with 10.6.8 - the ATI Radeon cards completely sucked. You can still get nVidia GTX285s (and later) on eBay - both 'proper' Mac versions and versions that have been modified to support Mac EFI that work very well and give much better performance.

You don't mention what version of the Active Silicon cards you are using? I am assuming Phoenix cards? Dual channel or Single channel? I never successfully got more than two 1920x1080 inputs working without problems on Mac Pro 5.1s.

Hugh