Mark -

1) Create a camera in 3DS Max and place it at the precise location of the projector that you will be using onsite
2) Make the camera angle the same as your raster image from the projector
3) Render the scene with the appropriate scenic items in place. Lighting should be very flat with no shadows.
4) Open the image in Photoshop and turn the areas within your raster that you wish to mask to black and erase all the areas you wish to have Catalyst images appear. Once onsite, you may need to [Transform] your image in Photoshop due to irregularities of the hang, projection location, etc.
5) Save the image with alpha channels and load into Catalyst

As you already know, it will need to be on Layer 4 in order to mask out other images. You may find that if you are running out of layers for your show, you might want merge your new mask onto your content so that it always is within the content itself.
Give us a call when you start getting into scenery that requires masking but is automated