I was able to capture a video of the "tearing" as well, here is the frame(s) that it happens in on both the source file and the actual playback on catalyst. It is somewhat subtle at times but on videos with more motion it is more noticeable.
Video size here is 4352 x 720, 30fps,
Apple Pro Res proxy, file size is 58.7 MB and data rate is 29.34 Mbits/ sec
I have files that run on the exact same machine that have double that data rate that run fine, they just don't have any fast motion so I don't see it.
This is with 1 layer running, plenty of system resources showing on the meters and no apparent frame drops. In testing I could play back I think 6 layers of this file with no frame drops.
System is a mac mini with 16 GB RAM, SSD, running 10.8.5 all updates. M364
I have more pics if it's helpful.
link in case the picture doesn't work: https://www.dropbox.com/s/tlpdpue7g7l3x4o/tear.jpg