There's a distinction with the terminology you're using. There's no such thing as "clearing" a binding you have set back to what it was. When you set a key to something, the previous setting is lost. Clearing a binding is to remove all actions from it. There is also no inherent default to any key, you either need to remember the default before changing a single binding or use LoadBindings(0) to reset ALL keys.
If your intention is to "reset" 1-4 back to the action buttons, just set them to the action buttons.
SetBinding and
BindingID are what you're looking for.
If I recall correctly, frame:ClearBindings() only removes the frame from any registered bindings, the actual binding is not changed.