No matter what I try I can't make proper Round/Hexagonal shaped 3D unit portraits
 

I take "Google is your friend" saying literally, that site saved me so much time and effort. However, in this case I always ended up with dead ends. No one knows the proper way to do it. The closest thing I came across was "attaching texture to the model in weak auras", which I have no idea of. Azerite UI and OWD UI somewhat accomplish this, so to have an understanding I tried having a look at lua files but I got nothing. Example:

https://i1382.photobucket.com/albums...pspryvjmmk.jpg

I do believe if I can find a way to clip textures in weak auras, like an invisible border, I'll finally succeed.

Thanks in advance, hope someone can help me

PS: I'm aware that I can hide edges with KG Panels, but I want something better because I want it to look perfect.

Since i'm pretty sure PlayerModel derived objects can't be masked, i'm assuming you're working with non-animated 3D portraits, which iirc only come from SetPortraitTextureFromCreatureDisplayID, which will discard any set masks and use a standard circle, and also comes with an undefined background tint.
so
would still result in a circled texture, most likely with a non-black background.

It's quite probable that the hexagonal border in that example is just covering the circle edge.

Thanks a ton for the help. I'm still a newbie when it comes to Addons, should I simply paste these to my Addon (bool) or somewhere else?

I'd like to answer my own question in case someone comes across this thread. Apparently with a good frame and overlay you can easily "mask" 3D portrait to appear as circular or hexagonal. I used the unit frame textures from Azerite UI and it looks great. So all you need are a thick unit frame texture and a circular fade texture. Combining these will have a very natural appearance.

But if you want a very thin portrait frame, you are out of luck. It doesn't seem possible, I've kept searching for weeks.

There's a similar function, SetPortraitTexture(). This allows you to set a texture to a given unit's portrait.

PS: What happens if you apply the custom mask after setting the portrait?

Uhh I have no idea how to use the code above. Does this work with 3D portraits btw?

SetPortraitTexture() is what the default UI uses for the unitframe portraits. Just drop it in place for SetPortraitTextureFromCreatureDisplayID() in Elcius' example. This takes a UnitID instead of a number to generate an image from. For example, SetPortraitTexture(texture,"player") will render your own portrait. If the model updates, you have to call the function again to show the changes in your portrait.

Since I'm using Stuf unit frames should I simply edit its lua and paste the code to the correct line? I am very sorry, since I'm just a player who's toying with simple edits even the simplest thing becomes confusing.

I honestly don't use Stuf UnitFrames. I wouldn't suggest tinkering with an existing addon as a first project. Depending on the addon, you can find yourself going way over your head too quickly and it may be discouraging. Instead, figure something you want to do for your own project, ask around if you need help, and use our examples to learn how the system works. Even if you come up with an idea that isn't possible, it's still a learning experience of what addons can and can't do. There are many other resources out there you can look for too.

Well said. Actually I don't have that much use for hexagonal portraits since textures already mask them well enough, but I can still use them in my HOTS UI project. I just found out that there is a way to mask icons with a custom shape so most of my work is done.