You nailed it. For me it is a combination of:
- If the buffs/debuffs look good on the unit frame or as their own frame
- Crowded space or they are too small when attached to the unit frame
- Have I skinned my UI
- I prefer to have my debuffs be larger than my buffs
- Personal taste
There is no wrong answer.