You can use any built in Blizzard textures to mask another texture, SetPortraitTexture works very similar, it masks a round circle on the square race icons.
There is a hexagon texture, masked with the HOTS icon from the game:
Lua Code:
local frame = CreateFrame("Frame", nil, UIParent)
frame:SetPoint("Center", 0, 0)
frame:SetSize(100, 100)
local icon = frame:CreateTexture("Texture", "Background")
icon:SetTexture("Interface\\Icons\\Ability_Ambush")
icon:SetMask("Interface\\ChatFrame\\UI-ChatIcon-HotS")
icon:SetAllPoints(frame)
The problem is that the resolution/edges are highly based on the quailty of the masker AND the original image.
Another instresting masker images:
Lua Code:
icon:SetMask("Interface\\DialogFrame\\DialogAlertIcon")
icon:SetMask("Interface\\DialogFrame\\UI-Dialog-Icon-AlertNew")
icon:SetMask("Interface\\GuildFrame\\GuildLogoMask_R")
icon:SetMask("Interface\\ARCHEOLOGY\\Arch-Keystone-Mask")
icon:SetMask("Interface\\GLUES\\Models\\UI_Dwarf\\UI_Goblin_GodRaysMask")
icon:SetMask("Interface\\GLUES\\Models\\UI_DeathKnight\\IceCrown_CloudsA02_MASK01")
icon:SetMask("Interface\\FriendsFrame\\PlusManz-Horde")
The possibilites are endless.