Also I guess LoadOnDemand isn't even needed, and removing it lets everything continue to work when you /reload.
I should probably have tried it that way before assuming raid frames weren't loaded until you explicitly joined a party/group.
@Fizzlemizz: I think there is value in keeping the panel visible at all times while it is open
but I guess that might depend on user preference.