I don't think a macro can decipher whether you are behind a target or not. However the ambush/backstab macro is easy:
Code:
/cast [stealth] Ambush; [nostealth] Backstab
If there were a 'behindtarget' filter (I can't find any reference to one, there probably is not) then it would look like the following:
Code:
/cast [stealth,behindtarget] Ambush; [behindtarget] Backstab; Sinister Strike
Note that you can't use fall-through "dumb logic" to perform decision making if a spell fails (not in stealth, not behind target, etc). The following DOES NOT WORK:
Code:
/cast Ambush
/cast Backstab
/cast Sinister Strike
That's probably what you were using before. I'm afraid you'll have to use one button for BS/Ambush and another for SS.