Nev is correct, this cannot be done.
Video capture cannot be passed across multiple graphics cards.

You need a Matrox TripleHead2Go or a Datapath x4 to achieve more than 2x outputs from a pre 2014 (5,1 Generation or older) Mac Pro, using a single graphics card.

S