I've implemented this feature into the next version of Masque, but only for the Icon layer. This should allow skin authors to make better circular skins. I suppose you could make any shape you wanted, but you'd have to create your own spell alerts, etc. The basic circular one I did was a pain in the ass and I really can't be bothered to make more.
I'm not going to bother implementing it for the other layers because those are usually custom textures anyhow and authors should utilize the alpha channel rather than a mask. That is, unless animated masks turn out to be an easily-implemented and interesting prospect.
As for the masks themselves, from what I can tell the engine uses the alpha channel of the texture so the RGB values shouldn't matter. In many cases, skins that use the same "overlay" texture for the flash, pushed, etc, can simply use that texture's alpha as the mask for the icon.