SetAttribute and RegisterStateDriver are not specific to LibActionButton. They are part of the Blizzard secure frame/header system.
SetAttribute does exactly what it sounds like -- sets an attribute's value.
button:SetAttribute("cows", "moo")
print(button:GetAttribute("cows")) ==> "moo"
On their own, attributes don't do anything. It's just a way of storing data on the button, pretty much the same as just using
button.cows = "moo" except it can work within the secure environment. However, certain attribute names are used by other Blizzard code. For example, on an action button, attributes like "type" and "spell" will control what happens when you click the button.
RegisterStateDriver lets you automatically update states (which are really just attributes) based on macro conditionals. As with regular attributes, certain state names and values have built-in behavior. For example, the "visibility" state, when its value evaluates to "show" or "hide", will cause the frame to be shown or hidden.
RegisterStateDriver(frame, "visibility", "[combat] show; hide")
^ This will cause the frame to be shown in combat, and hidden at all other times.
You can read the Blizzard code for RegisterStateDriver and related functions here:
https://www.townlong-yak.com/framexm...tateDriver.lua
Code related to attributes in general can be found in all the FrameXML/Secure*.lua files.