Extract "wow-update-12759.mpq"... meh!
 

Trying to extract the DBC and LUA/XML files but getting weird patch-like content in the files. How does the new patch system work?

I've been trying to figure this out with no luck too. Apparently Tekkub found a way to extract build 12694, which uses the same patch format. I couldn't find any mention of what tools he used though. The actual patch files look like bsdiff binary diffs with a proprietary blizzard header prepended to them. I had trouble applying any of the patches though, even with the extra header trimmed out.
____
Edit: On second look at what Tekkub committed, he didn't extract patch 12694 properly. He has the changes between builds 12644 (pre-streaming client) and 12635 (base streaming client); None of the changes from wow-update-12694.mpq are included though.

Yeah, I noticed that too. I've been opening update exes before (they are mpq too) and this file structure seems familiar, it's only hard to find out how we can now extract the up-to-date FrameXML folder when it appears that each new build will have it's own "patch" MPQ containing diff files and not the whole patched file, hmm...

Perhaps someone at Wowhead may help? They do extract DBC, BLP icons and similar and they are also subjected to the same difference patching system.

I just used MPQ extractor.

You can find it on www.devklog.net

The last time I tried it was a few weeks ago and got a folder worth of files extracted.

Not sure if that's what you're looking for or if even if it extracts it right.

Wait.... I might of tried this on a previous build. I'll try to remember to repeat it when I get home.

Try it yourself Bluespacecow, use the MPQEditor, open "wow-update-12759.MPQ" and extract the "Interface/FrameXML" folder. Open up the files inside and be surprised. ;)

Using OS X so can't use MPQEditor.

I note that at

http://www.zezula.net/en/mpq/download.html

Is a beta version for extracting wow-update-12694.MPQ

Is the problem extracting the files the problem ? Or the format they are in once extracted.

Try the Beta Version at that site above and extract with that if you haven't already. Zezula lists that MPQ (wow-update-12694.MPQ) as a partial MPQ file so wow-update-12759.mpq may be as well

Even with the beta version of that MPQEditor, it still extracts patch files. Its file properties dialog can tell you if a packed file is a patch, but that's it. I even tried using the multi-MPQ mode with build 12694 loaded over top of 12635. Still no luck though.

The problem is that like in this case, I want the latest FrameXML files. I naturally open "wow-update-12759.MPQ" as it's the update file downloaded with the launcher. Inside I see the various localizations and the "base" that is common. I open enUS/Interface/FrameXML/ and see several files. I open for example "ActionButton.lua" and see this (binary data):
Basically the updates contain diff patches and if the file is new then it's normal.

*Edit*

1. Extracted "locale-enUS.MPQ" and it's FrameXML
2. Extracted "wow-update-12694.MPQ" and it's FrameXML
3. Extracted "wow-update-12759.MPQ" and it's FrameXML

I downloaded bsdiff and bspatch (Binary diff/patch utility version 4.3) and when I try to patch a file from #1 by using a file from #2 I get "corrupt file", seems like they also added some custom data to their patching files, hmm...

I think the BSDIFF40 part in the PTCH files is invalid.
For example the bsdiff part of ActionButton.lua from wow-update-12759.mpq says the resulting file would be 146789209322029069 Bytes long (thats 130 Petabyte) :rolleyes:
As far as I understand the format the data in bsdiff files should be compressed with bzip2, but there's clear text in the .lua diffs.

I didn't find any file format beginning with 'PTCH', so maybe that's a Blizzard invention.

Borlox, that's what I also found out.

There is some clear text indeed, the data that is to be inserted/replaced, e.g. is all there. Problem is figuring out where to put, replace, append, delete, e.g. I tought wowhead or mmochamp figured this out as they do read DBC files right, so they need to figure out how to patch their DBC with the updates so they can for example scan Spell.DBC for changes. ;)

Asking Wowhead mods on IRC atm, let's see if there is a way.

I really hope they'll give you a clue...

at least on windows the addonkit from blizzard still works but you need 100% downloaded/patched and need to close the laucher/wow before you start extracting

I think the Addon Kit is skipping the wow-update MPQs too. I can't find any of those changes in its output. As an easy test case, check for <FrameXML/Util.lua>, which should be there since patch 12694.

Addon Kit only extracted from my retail, not beta. Also I think it also only extracts from the locale file and not including the patches.

Found some (if not all) of the Interface files that were patched in this build in the Data\Cache\enUS (or whatever locale you use) directory in the build's patch MPQ. From what I can tell, though, any totally new files aren't in there at all. The new spell flyout files aren't in there, for example, but are in another MPQ in text form rather than as a patch (since they're new).

Looks like it may take a bit more work than before to extract things that can be committed to a revision control repo or diffed before a new kit is released that can cope with the new scheme. Nothing a little scripting can't handle, but it's still a change to adapt to.

Adys from MMO-Champion and Ladik the author for MPQ Editor (not sure of the nick entirely) to integrate this feature into the editor itself most likely, patch the files as they are extracted from the updates -very neat ey?

Adys told me that this would not be available until several weeks later so we just hang in there, someone is working on fixing this issue we have. ;)

I currently work on "Open MPQ in patch mode" feature of MPQ Editor. The desired functionalty is that you open a base MPQ, then add list of patch MPQs (currently, there are wow-update-12694.MPQ and wow-update-12759.MPQ), and it should seamlessly extract patched file already. I hope first experimental version might be available this week.

Great utility, Ladik! I only just discovered MPQ Editor, and I have nothing but compliments for it; Light years better than WinMPQ. If I could request a feature, could you add a command-line option to load multiple MPQs in the given order? Once there are lots of patches, it would make loading them all up much simpler.

Something like this could work well:

Try this: MPQEditor huhu.mpq hehe.mpq hihi.mpq haha.mpq.
However, this opens the multiple MPQs in multi-mode. I guess it's time for big redesign of the entire command line syntax, because it has been there for like 6 years and too many things have changed :-)

Indeed. Yet another reason to redesign command line syntax.

Awesome, did you write it from scratch (the patching func) or did you borrow some ideas from the bsdiff sources? Just curious. I guess you wrote from scratch as you are probably super good at programming. :P

He is. Trust me ;) (endarion here, Ladik^^)

I took ideas from bsdiff source, plus help from people from the MPQ community, as they already figured out how to interpret Blizzard version of BSDIFF. You will see it soon in the next BETA of StormLib anyway.

That's great news, thank you Ladik!

What is this MPQ community you're talking about?

First rule of the MPQ community, ...

That they are very secretive? Any forum you can redirect me to? Or are you its only member?

Bunch of people who ever e-mailed me that they need something, or that there is a new MPQ format that I can't handle :-). There is no forum or an official website, most probably the guys don't even know each other. They all know my MPQ tools so they write me frm time to time.

Any ETA on the tool?

Hopefully tomorrow (19.08.2010). I'll post here when at least experimental version is ready.

We should throw a release party :banana:

Yes :banana:

Ladik, you're the man. ;)

What's your reason?

I want to feed my talent planner with current data.

Link it;) Btw not a word from Ladik....makes me sad :(

No point in worrying nor stressing him, he will release his tool when he feels it is stable enough for a public alpha/beta release. ;)

Ladik can your tool be used to extract the latest .blp files from the mpq?
or is it only lua & xml?

It will support any kind of files in the MPQ, don't worry.

Vladinator thanx for the heads-up m8

Ok, seeing how eager you all are :p, I made experimental version of MPQ Editor that supports opening multiple MPQs in patch mode. You can download it here:
removed
This link will only be valid for a few days, I'll remove it as soon as I get something more tested. Do not use it for anything else that extracting a file from patched MPQ.

Good news: Each step of the patching process is verified by MD5, both before patching and after patching. So once you manage to extract something without an error message, you can be sure that it's correct.
Bad news: I hadn't much time to test it yet. This build is by no means finished.

Usage: Best by command line, like this:
MPQEditor.exe /patch "locale-enUS.MPQ" "wow-update-12694.MPQ" "wow-update-12759.MPQ"

In case you are unable to extract a patched file, please, send me an e-mail to zezula-at-volny-dot-cz. Please, don't forget to include:
- The name of the primary MPQ
- List of patches you entered
- Full name of the extracted file

More tomorrow, it's late here and I'll go something more relaxing for rest of the (already ending) day.

I ran this And tried to extract the DBFilesClient folder, but it crashes on WorldMapContinent.dbc

For those interested, now that Ladik has released MPQEdit with patch support. Here is a Python library to read PatchFiles (PTCH).

http://www.pastethat.com/9pGwm

Released under BSD.

Sorry, guys, but I had to remove the link. You know our rules! :)

For those who don't know, see rule #8 on this post.

Adys, I think Jerome is a cool name. ;)

Could you post a really short sample?

patchFile = open("wow-12345-patch/testfile.lua") # patch file
oldFile = open("old/testfile.lua")
f = PatchFile(patchFile)

print f.apply(oldFile.read())

oldFile.close()

I really don't see how it violates it

MPQ editor is an external program , not an addon.

If you try Adys' python script, append this at the end to actually use his library as a application:
It includes guidelines, basically takes in 3 arguments, the file, the patch and the output where to write the final file.

I think it's correct, tough I tried to patch ActionButton.lua from local-enUS.mpq with the wow-update-12694.mpq patch and getting MD5 errors.

I'm looking into it. If it's just an error, you can pass validate=False to apply()

Ah, my apologies then. I thought that MPQ Editor doesn't fall into category of "addons" :)

For all others who e-mailed me: Thanks for bug reports, I'll fix them problems today, and hopefully a more stable version should be ready this evening.

Who is Jerome?

Edit: Found the error, the right base files are in enUS\patch-enUS.mpq, not in locale-enUS.mpq :rolleyes:

I extracted the files and tried the python script with validate=False, but that gives me another error message:
I couldn't get any file in FrameXML patched...

"MpqEditor.exe enUs\locale-enUS.mpq wow-update-12694.mpq" opens the file without any error message, but it's empty.
The same happens when both update mpqs are used.

Thanks for all the work you've done! I'm really looking forward to a working version :)

That would be me. Hi!

Thanks for the reports guys. Below is an updated PTCH parser:
http://dpaste.com/231608/

Also here is my blizpatch.py in case anyone wants it. Mind you you'll need to change imports to make it work.

http://dpaste.com/231609/

A new BETA build of MPQ Editor, available on my website, should be now able to handle all patches properly. I fixed several bugs there. Please, let me know if you find a file that you are unable to extract, or it doesn't look well after extracting.

This version should be far more stable that the one I published yesterday late in the night (note for myself: Never ever release anything when it's close to midnight).

As usually, there is new BETA version of StormLib available, which also contains algorithms for uncompressing and applying Blizzard patches.

Note: The original download link for the experimental version is longer valid.

Works fine for me.

Thanks a lot, Ladik and Adys, your work is much appreciated.

Even more reason to remove the link.

Extracted the files.
But for some odd reason there are alot 1kb .blp files
and some larger that can't be converted to .png

Don't know if this is a bug, or just the patched data

Is there a way to open a patch archive for multiple mpqs?
I'm trying to hack support for the wow-update-*.mpqs into the WoWModelViewer. My first idea was to open all the patch archives for each mpq, but that doesn't work, as they may be opened just one time.

Does anybody have a better idea, or is anyone already working on this?

Name some of the files you're referencing.

No need, all models are viewable at least for me.