Probably a weird old driver issue with the dazzle...

Have you tried swapping the video input to the iMac camera and seeing if the same thing happens?

Are you running any other apps that use the camera at the same time as catalyst?
Most (almost all) camera drivers only support one app at a time. In short - ensure that catalyst is the only app running and using the camera/dazzle

Have you saved all your settings and tried it on another Mac with different capture hardware?

Sounds odd?

S