Found a work around.
Code:
local AG = CooldownFrame:CreateAnimationGroup()
local A = AG:CreateAnimation('Alpha')
AG:SetLooping('NONE')
A:SetOrder(1)
A:SetDuration(0)
A:SetFromAlpha(1)
A:SetToAlpha(1)
CooldownFrame.AG = AG
CooldownFrame.A = A
Then just do a CooldownFrame.A:Play() before calling SetCooldown()
And now I can use SetHideCountdownNumbers(true), all works now. Hopefully they fix it cause I hate doing hacks like this.