UnitCanAttack should be used insteadof UnitIsEnemy and UnitAffectingCombat since Whirlwind will hit a unit whether it is in combat or not. UnitExists is implied if any Unit function returns true so it can be removed. Also, inRange only updates in one place so you should check it's value right after instead of at the end of the loop where it may not have changed.
Code:
function are3EnemiesInRange()
local inRange = 0
for id = 1, 40 do
local unitID = "nameplate" .. id
if UnitCanAttack("player", unitID) and IsItemInRange(63427, unitID) then
inRange = inRange + 1
if inRange >= 3 then return true end
end
end
end