Looks like
PlayerTalentFrame inherits
PortraitFrameMixin. Neither
PortraitFrameMixin:SetPortraitTextureRaw() nor
PortraitFrameMixin:SetPortraitTexCoord() call
SetPortraitToTexture() while
PortraitFrameMixin:SetPortraitToAsset() does.
Digging through the art files reveals that the image itself was created with the circular mask, meaning you'll need to swap to a different image instead. You could probably swap the texture with
"Interface\Glues\CharacterCreate\UI-CharacterCreate-Classes" using the same texcoords.