Ah, I was only testing it with a single aura. Got the error now I think. Fun fact, most bugs are actually due to me not properly testing something with an aura that isn't #1 in the list
Edit: Mostly fixed it - it doesn't completely fail now, but there still is an error somewhere (which has no effect).
Edit #2: Nailed the other bug. Should also fix error messages when deleting a parent display - issue was just it not validating the displays still existed when it tried to reconfigure them, which was harmless anyway.