Many LoD addons are options frames so they don't need to be loaded all the time once we loaded it to change an option.
For the LoD addons we can use LoadAddon be when the addon is loaded there is no way to "unload" it so a similar function UnloadAddOn(index or "name") would be a good function for the CPU usage.
Actually, unless your configuration options are doing something really obscure in the background or are interacting badly with another addon, there shouldn't be any significant impact, in terms of CPU time, assuming you aren't tampering with the options all the time. Granted, your UI will use more memory perhaps without good reason, but memory usage has become largely irrelevant, assuming your machine is powerful enough to handle Wotlk without many issues and your addons are not "leaking" due to bad coding.
It also depends on the size of your config LoD addon. That being said, a way to unload an addon could be quite useful regardless
It would be very difficult to do, as the LoD addon has free rain to muck about with whatever it feels like. Plus it's already pretty much possible through the garbage collector by basically nilling out all references, removing all SetScripts and performing collectgarbage("collect").I think the only thing which can't be removed are frames.
A well designed config frame will not generate anything more than a single empty frame and a single function on load, both of which take very little time to do. The work is offloaded for when the frame is first displayed. There is no reason to put it in a LoD addon and there's no reason to unload it after it's loaded