Cleaning the Global Namespace (OO)

[Cyrael]

Following is a standard that I've worked on to streamline my own addons. I'm not sure if this implementation exists already elsewhere, and if so then I apologise for replicating it:

Introduction
The XML/Lua implementation in World of Warcraft follows an interesting mechanism. As most of you are aware, every named element in the XML is given a corresponding entry in the Lua global namespace. This relationship is required for Lua to reference XML generated objects. What is important is that this relationship is one way. XML has no need to maintain these specific names within the Lua global namespace, and for this reason they can be modified as the coder sees fit.

While there may not seem to be many useful aspects to deleting XML references, you can still take them out of the global namespace by putting them into your own addon table. A simple example is as follows:

The example XML:

<Frame name="MyMod_Frame1">
    ...
</Frame>
<Frame name="MyMod_Frame2">
    ...
</Frame>
<Frame name="MyMod_Frame3">
    ...
</Frame>

This code will place three entries titled MyMod_Frame1, MyMod_Frame2 and MyMod_Frame3 into the Lua global namespace. If this mod were bigger, there could be several hundred entries in the namespace. This can all be cut down to a single entry without losing any functionality, however, as follows:

The example LUA:

MyMod = {};

MyMod.Frame1 = MyMod_Frame1;
MyMod.Frame2 = MyMod_Frame2;
MyMod.Frame3 = MyMod_Frame3;

MyMod_Frame1 = nil;
MyMod_Frame2 = nil;
MyMod_Frame3 = nil;

What this does is creates a new table, MyMod, within the global namespace. It then copies the reference to each of the XML elements into itself. Once the references are copied, the original references are then deleted. It's important to remember that functions are first class objects in Lua. This means everything is a reference to something else, there are never any actual objects. Lua performs its own garbage collection by watching for objects that are no longer referenced so that it can delete that object from memory. Setting MyMod_Frame1 to nil doesn't delete the function, because MyMod.Frame1 still refers to it. Instead, it deletes the global namespace entry created for the XML object.

There are no negative side effects to this process. Every action that can be performed on the global entry can continue to be performed on the localised entry. Additionally, since the XML never relies on the Lua, it will not interfere with the game engine displaying and operating the components created in the XML.

One Step Further
It may seem like a daunting process to add MyMod.blah = MyMod_blah for every XML element in your addon. Fortunately, Lua provides us with all of the tools needed to perform this operation automatically. The following shows how.

MyMod.gns = getfenv(0);

for _, v in MyMod:GetGNSKeys() do
    if (string.find(v, "MyMod_")) then
        MyMod:RegisterXML(v, string.gsub(string.gsub(v, "MyMod_", "MyMod.XML."), "_", "."));
    end
end

...

function MyMod:RegisterXML(source, dest)
	-- Parse the destination string
	local sfind = strfind; 
	local strsub = strsub;
		
	local sstart = 2;
	local dValue;
	-- Split the variable name at ".", first field is a global name
	local match = sfind(dest, '.', 2, true);
	if ( match ) then
		local currentVar = strsub(dest, 0, match-1);
		dValue = getglobal(currentVar);

		while true do
			if (type(dValue) == "table") then
				sstart = match + 1;
				match = sfind(dest, '.', sstart, true);
		
				if ( match ) then
					dValue = dValue[strsub(dest, sstart, match-1)];
				else
					dValue[strsub(dest, sstart)] = self.gns[source];
					break;
				end
			else
				break;
			end
		end
	else
		self.gns[dest] = source;
	end
	
	self.gns[source] = nil;
end

function MyMod:GetGNSKeys()
    local gnsKeys = {};

    for n in pairs(self.GNS) do
        table.insert(gnsKeys, n);
    end

    table.sort(gnsKeys);
    return gnsKeys;
end

This code assumes that both MyMod and MyMod.XML have already been defined. What it does is iterate through every item in the global namespace (as retrieved by getfenv(0)) looking for every entry that begins with "MyMod_". It then automatically removes the "MyMod_" part from the beginning of the line replacing it with "MyMod.XML." and then proceeds to convert all of the remaining underscores into dots. The result is as follows:

MyMod_Frame1 = MyMod.XML.Frame1
MyMod_Frame2 = MyMod.XML.Frame2
...
MyMod_Frame1_StatusBar = MyMod.XML.Frame1.StatusBar

In effect, this converts the entire XML namespace into your MyMod table. It then indexes into the previously saved reference to the namespace (gns) and deletes the global variable from it. The reason I added the extra step to put the entries into MyMod.XML rather than MyMod itself is to demonstrate that this can be easily done during the string replace step.

The very important part in the process is the GetGNSKeys() function. The global namespace is a table, and we are trying to modify our entries based on its key name. This is not sorted, and if it tries to process MyMod.Frame1 after MyMod.Frame1.StatusBar, the latter will be erased. To fix this, GetGNSKeys() extracts the key information from the global namespace, and very importantly, sorts it alphabetically. Using the above naming scheme where the parent is MyMod_Frame1 and the child object is MyMod_Frame1_Child, this will force the parent to be processed first and will not cause this problem.

Conclusion
The net result of this approach is that your entire addon, not just the Lua side of things, is converted to an object oriented methodology. A quick check for all entries in the global namespace for MyMod_* after performing this will produce a single result only: the MyMod table itself. This cuts down dramatically on the amount of global namespace exposure resulting from your mod, and even further decreases the odds of a name clash.

[Kasheen]

That sounds like quite an interesting method but I wish to put forward a question: Say for instance mondinga did this with his "GypsyMod" (its already OO and its an actionbar mod, this is the only reason im taking it as an example), would this then mean that mods such as "CooldownCount" (overlays a FontString onto action buttons to show their remaining cooldown as a digit) would cease to function as ActionButtonX would no longer exist in the global namespace in order to anchor the FontString?

Therefore doesn't this slightly hamper anyone wishing to write some sort of addon or extensions to your mod?

Kasheen

[Cyrael]

When you convert data across to this methodology, all data is still accessible - Lua has no concept of private data. If you could previously access the global as ActionButtonX, and you change it to be Gypsy.ActionButtonX, as long as all other mods that want to refer to it change their reference from ActionButtonX to Gypsy.ActionButtonX (or Gypsy.XML.ActionButtonX, if that's the layout used), it will still refer to the correct object.

If you want to use something like an anchor, where you have to define the reference point in XML, unfortunately there's no way to do this at design time using this method. A solution does exist, however, by using the SetPoint() API function, which will set the anchor from Lua. Calling that function in your OnLoad event will achieve the same effect as in the XML.

<Frame name="Frame1">
    <Anchors>
        <Anchor point="TOPLEFT" relativeTo="ActionButtonX"/>
    </Anchors>
</Frame>

... is equivalent to ...

<Frame name="Frame1">
    <Scripts>
        <OnLoad>
            Frame1.OnLoad();
        </OnLoad>
    </Scripts>
</Frame>

function Frame1.OnLoad()
    Frame1:SetPoint("TOPLEFT", ActionButtonX);
end

[Gello]

That is interesting thanks for posting that.

I don't want to detract at all from the method or concept of OOP, but unfortunately there are a lot of mod creators under the impression that removing global namespace items will somehow improve memory consumption or performance. This is not so.

Check out: http://www.wowace.com/forums/viewtopic.php?t=802

Over THREE MILLION items in the global namespace made NO DIFFERENCE.

This is not to say OOP is bad or a waste of time or any of that. It's useful but it in no way improves performance or memory consumption.

[Cyrael]

Reducing global namespace clutter will (very slightly) increase memory for the extra table overhead, and will improve performance. This is a fundamental truth of indexing into any table in any programming language, and Lua tables are no different. Your first test contains flawed data:

Variables 1638401-3276800: 3295217 globals, 0 tables, 163834 mem added, 0.100 mem/var, 0.019 sec 100k local calls, 0.021 sec globals, 0.025 sec tables ... Variables 1638401-3276800: 18602 globals, 1 tables, 163833 mem added, 0.100 mem/var, 0.023 sec 100k local calls, 0.022 sec globals, 0.026 sec tables

You cannot use less memory between a global reference and a tabled reference. Your table should be at least 13 bytes bigger than using the global namespace alone. This test appears to be flawed, I'll look further into it when I have time.

In the latter tests, it appears that only the number of GETTABLE calls actually take place. I can assure you that a single GETTABLE call into a 3,000,000 element table will take more time than one GETTABLE call into a 250 element table, and another into a 100 element table.

The object oriented approach provides a facility to avoid searching through data that is determined to be unnecessary. In both cases, your benchmarks either test the wrong information or are flawed, and so your benchmarking tests are ultimately incorrect. I've seen other benchmarks that test the issue accurately, and they show a notable performance improvement across the board when reducing the global namespace clutter. I'll try to find the benchmark code for you to see the difference.

[Gello]

*bangs head against desk*

There is no noticable performance improvement. It's like saying C is faster than assembler.

I look forward to the results of those tests because not a single one has demonstrated a meaningful difference in game. It's just been a lot of regurgitated comp sci teachings, Rowne's emotional FUD tactics and so far not a blip of data to show otherwise.

[Cyrael]

See below for the correct benchmark code.

[Iriel]

There's an inherrent flaw in that example, for the top loop you have created a single nested entity, which is GNSTest.i.j.k (or GNSTest["i"]["j"]["k"] if you'd rather).

In the second loop you're creating 10,000 separate indices and searching for them all.

It's apples and oranges, no matter how you cut it.

[Cyrael]

Following is benchmark code that shows evidence that migrating out of the global namespace and into a tabular format will improve indexing times and ultimately speed up the processing of your mod variables.

local gns = getfenv(0);

local stringIndex = {};
for i = 1, 100000 do
	stringIndex[i] = "o" .. i;
end

-- Run the tabular test first, before the global test clutters the namespace
GNSTest = {};
for i = 1, 100 do
	local iX = stringIndex[i];
	GNSTest[iX] = {};
	for j = 1, 100 do
		local jX = stringIndex[j];
		GNSTest[iX][jX] = {};
		for k = 1, 10 do
			local kX = stringIndex[k];
			GNSTest[iX][jX][kX] = "5";
		end
	end
end

-- Conduct the operation - reference every variable
start_time = GetTime();
result = "";
result2 = "";

for i = 1, 100  do
	for j = 1, 100 do
		for k = 1, 10 do
			-- Superfluous concatenation and index reference to match global test
			local iX = stringIndex[i];
			local jX = stringIndex[j];
			local kX = stringIndex[k];

			-- Superfluous string concatenation to match the global operation
			dummy1 = "ABC";
			dummy2 = dummy1 .. k;
			result = result2 .. GNSTest[iX][jX][kX];
		end
	end
end

end_time = GetTime();

gns_size = 0;

for index in gns do
      gns_size = gns_size + 1;
end 

delay = end_time - start_time;
DEFAULT_CHAT_FRAME:AddMessage("TABLE: GNS=" .. gns_size .. " Time=" .. end_time - start_time .. " Result=" .. result);

-- Run the global test
for i = 1, 100000 do
	setglobal("GNSTest_Global" .. i, "5");
end

gns_size = 0;

for index in gns do
      gns_size = gns_size + 1;
end

-- Conduct the operation - reference every variable
start_time2 = GetTime();
result = "";
result2 = "";

for i = 1, 100000  do
	result = result2 .. gns["GNSTest_Global" .. i];
end

end_time2 = GetTime();

gns_size = 0;

for index in gns do
      gns_size = gns_size + 1;
end 

delay = end_time - start_time;
DEFAULT_CHAT_FRAME:AddMessage("GLOBAL: GNS=" .. gns_size .. " Time=" .. end_time2 - start_time2 .. " Result=" .. result);

Thanks to Iriel for helping me clean this up. The results for this code show a 12% increase in performance when using the tabular method. When the test is reduced to 10K elements instead of 100K, the performance difference increases to 15%.

The result of this test is to prove that reducing the clutter of the global namespace, or indeed any large scale table, will show a noticable improvement in code performance.

[Iriel]

I personally would state the somewhat more general:

Using appropriately structured tables for large data sets, or small tables for extensively referenced data sets, will result in a performance benefit, whether you choose to use OO or not.

It does boil down to a fairly marginal improvement, simply because a table lookup is still in the order of a microsecond, and thus you have to do a whole LOT of them to add up to anything detectable. There are far more effective ways to improve performance of code before jumping on this as the answer.

HOWEVER, it is worth noting that this does effectively show that there isn't a PENALTY for structuring code in tables, and that it can indeed help in a number of situations.

(NOTE: These results do not hold for tables used as arrays, since they exhibit constant time accesses for integer keys within their key range)

[Gello]

hehe finally actual data! Thanks :) I got so disgusted with Rowne's "feelings" and constant babble about how computers are so complex they need to be talked about in analogies.

So part of the conclusion is that the lookup time to indexing a small table is faster than the lookup time of indexing a large table.

How do we reconcile the apparent similarity in performance on that other thread? If it's entirely invalid, what makes it so? The goal was to demonstrate that storing many variables in the global namespace has no performance impact to other mods.

It seems that using the global namespace would create a large indexing for other mods.

I agree the differences are marginal but the dark reality is that perception has more weight than facts in this community, partly thanks to Rowne's blind-leading-the-blind preachings. I think this bit upsets me the most because 1s and 0s are not complicated and we should be able to discuss performance and memory use with actual observation and data like you've provided.

[AnduinLothar]

Ok, I have 'sucessfully' converted the entirety of Wardrobe to use Cyreal's Tables XML methodology. It took aproximately 5 hours to convert a 5000 line addon from non-OO to OO + Tabled XML + Massive headache.

I don't recommend the process. It's painful. Starting out with that idea may be ok, but conversion is dreadful.

As for cavaets there are plenty.

One of the Largest pains in the ass is that you cannont use getglobal(this:GetName().."SomeString") This effectively makes templating dreadful and requires that you define a standard variable OnLoad in the template. While it is possible for your own templates it's not possible when using built in templates that do not account for such oddities. For example, FauxScrollFrame_OnVerticalScroll makes 4 or 5 such getglobal calls to grab the templated children. But if you changed Wardrobe_ScrollFrame to Wardrobe.ScrollFrame then Wardrobe.ScrollFrame:GetName() == "Wardrobe_ScrollFrame" and getglobal("Wardrobe_ScrollFrame".."SomeChild") does not exist, because you nilled it.

Another drag was that you had to put all the conversion in an onload called at the end of the xml file, after all the xml frames are declared. This means that any OnLoads in the previous functions will not understand the Wardrobe.ScrollFrame syntax because it hasn't been defined and must be treated as the default Wardrobe_ScrollFrame which the frame is named at that time.

Also note that it makes the XML 10x harder to read because it says one thing and that is nil. So if it's not your code, you'll be instantly lost. It also means you wont be passing any names of frames to libraries to use.

[Iriel]

Originally Posted by AnduinLothar One of the Largest pains in the ass is that you cannont use getglobal(this:GetName().."SomeString") This effectively makes templating dreadful and requires that you define a standard variable OnLoad in the template. While it is possible for your own templates it's not possible when using built in templates that do not account for such oddities. For example, FauxScrollFrame_OnVerticalScroll makes 4 or 5 such getglobal calls to grab the templated children. But if you changed Wardrobe_ScrollFrame to Wardrobe.ScrollFrame then Wardrobe.ScrollFrame:GetName() == "Wardrobe_ScrollFrame" and getglobal("Wardrobe_ScrollFrame".."SomeChild") does not exist, because you nilled it.

Then dont nil the global, I actually had a continuing conversation with Cyrael about this exact point. The key isn't to clear the global scope, since that'll have negligible impact unless everyone (including blizzard) does it, but to move the references YOU need into your objects. Leave the globals in place and you dont destroy this idiom.


In fact, in light of this post I'll go from my previous "I wouldn't recommend nilling the globals" to "I would recommend you DONT nil the globals, unless you're absolutely sure nobody else would want them".

[Iriel]

Originally Posted by Gello How do we reconcile the apparent similarity in performance on that other thread? If it's entirely invalid, what makes it so? The goal was to demonstrate that storing many variables in the global namespace has no performance impact to other mods.

I haven't looked at the other thread, but everyone does need to keep actual times in perspective, and I'm worried that this doesn't happen enough.

On my linux desktop, even a "slow" global scope lookup plus addition takes approximately 0.52 us (Microseconds). That's dwarfed by pretty much everything else that DOES something:

* Creating strings (Concatenation, etc)
* Calling functions (Usually those also involve a global lookup)
* ANYTHING involvling the blizzard metatable __index
* Creating tables.

Unless you create an artifically simplistic inner loop for your timing test, the actual difference in performance becomes almost unmeasurable because it's dwarfed by the "real work" that goes on afterwards.

For any normal use, this difference isn't enough to make any difference whatsoever, if you save a microsecond for 2 milliseconds of real work, who cares. Developers need to focus on not doing unnecessary work, and using generally efficient practices, not on notions of 'provable optimization' which are really academic at best.

The things every good lua developer SHOULD think about for performance include, in my opinion:

* Avoid creating objects when you dont need them (But dont be afraid of creating them when they help) - e.g. Avoid pattern captures in string.find if you're not actually going to USE them.
* Dont over-optimize code that's only run when the user hits a button or enters a slash command, stick with simplicity and readability to avoid bugs.
* DO optimize function hooks and OnUpdate code so you dont drag down everyone's performance with your code.
* Try and do as little as possible in OnUpdate blocks, and disable them when not in use (But dont scorn OnUpdate altogether if it's the best way to do something)
* Appropriate use of locals, and grabbing local references of globals or subtables to avoid redundant table lookups in large loops, is useful when performance is important.
* Remember that tables-as-arrays are essentially linear time lookups.

[Cyrael]

As Iriel said, it's not a big difference and it's not noticable under 'normal' circumstances. It's not a reason to move over to this methodology because it's faster, but no matter how small the benefit, -strictly- speaking it is faster. If, for whatever reason, your mod is performing thousands of operations in the global namespace, this will make a slight difference. From a CPU perspective, I think the calculation I did last night was that for every 10,000 calls to the global namespace you do, if your data was in a table instead you'd save 4 million instructions in the CPU for an AMD Athlon XP3200+.

The results of the test show that global namespace lookup improves by 12%, not the overall performance of the mod. This is a point of some confusion, and as Iriel said, global namespace lookup doesn't account for much in the grand scheme of things. String concatenations involving variables made a huge impact during the test.

Anyway, these results aren't to get someone to move over for performance reasons, they're purely to contest the assertion that using a tabular approach does not improve performance. The improvement is miniscule, but it is there. That said, putting this methodology into an existing mod probably isn't worth the effort. On a new mod, however, it takes no more time or effort than writing it in any other format, as long as you're familiar with how things work. I posted this more as a novelty point of interest (that it can be done) than anything else, really.