You need to use GetGuildInfo to get the guild name of a unit such as the player.
https://wowpedia.fandom.com/wiki/API_GetGuildInfo
All IsInGuild does is return whether someone is in a guild and doesn't take any parameters.
Unfortunately, needing a unit name to execute means this will have to be someone you are looking at specifically "mouseover", "target" or in a group/raid with "raid2", "party2" etc.
So, the next option to consider is looking at how nameplate addons *scan* the nameplate and extract the information. If one of them is the guild name you can then do your guild check once you have identified which guild the nameplate is showing.
Unfortunately I haven't played with nameplates at all to know what is and isn't possible from that point of access.