Catalyst is an app to which adding more memory will make no difference.
The graphics card, the nVidia GT120 is a bit on the rubbish side. Other things that make a difference is an SSD drive - run the OS on the system drive and install an SSD drive to play back the media.

If I were you I'd go for a beefier graphics card. You can get PC nVidia GTX285s on ebay modified to work with Mac EFI for a couple of hundred quid. Otherwise it's the more expensive nVidia Quadro card.
Which OS are you using?

Hugh