WoWInterface

WoWInterface (https://www.wowinterface.com/forums/index.php)
-   Cataclysm Beta (https://www.wowinterface.com/forums/forumdisplay.php?f=82)
-   -   Extract "wow-update-12759.mpq"... meh! (https://www.wowinterface.com/forums/showthread.php?t=34469)

Vlad 08-14-10 08:50 AM

Extract "wow-update-12759.mpq"... meh!
 
Trying to extract the DBC and LUA/XML files but getting weird patch-like content in the files. How does the new patch system work?

Saiket 08-14-10 09:27 PM

I've been trying to figure this out with no luck too. Apparently Tekkub found a way to extract build 12694, which uses the same patch format. I couldn't find any mention of what tools he used though. The actual patch files look like bsdiff binary diffs with a proprietary blizzard header prepended to them. I had trouble applying any of the patches though, even with the extra header trimmed out.
____
Edit: On second look at what Tekkub committed, he didn't extract patch 12694 properly. He has the changes between builds 12644 (pre-streaming client) and 12635 (base streaming client); None of the changes from wow-update-12694.mpq are included though.

Vlad 08-15-10 10:22 AM

Yeah, I noticed that too. I've been opening update exes before (they are mpq too) and this file structure seems familiar, it's only hard to find out how we can now extract the up-to-date FrameXML folder when it appears that each new build will have it's own "patch" MPQ containing diff files and not the whole patched file, hmm...

Perhaps someone at Wowhead may help? They do extract DBC, BLP icons and similar and they are also subjected to the same difference patching system.

Bluspacecow 08-15-10 11:05 AM

I just used MPQ extractor.

You can find it on www.devklog.net

The last time I tried it was a few weeks ago and got a folder worth of files extracted.

Not sure if that's what you're looking for or if even if it extracts it right.

Wait.... I might of tried this on a previous build. I'll try to remember to repeat it when I get home.

Vlad 08-15-10 11:08 AM

Try it yourself Bluespacecow, use the MPQEditor, open "wow-update-12759.MPQ" and extract the "Interface/FrameXML" folder. Open up the files inside and be surprised. ;)

Bluspacecow 08-15-10 12:21 PM

Quote:

Originally Posted by Vladinator (Post 202610)
Try it yourself Bluspacecow, use the MPQEditor, open "wow-update-12759.MPQ" and extract the "Interface/FrameXML" folder. Open up the files inside and be surprised. ;)

Using OS X so can't use MPQEditor.

I note that at

http://www.zezula.net/en/mpq/download.html

Is a beta version for extracting wow-update-12694.MPQ

Is the problem extracting the files the problem ? Or the format they are in once extracted.

Try the Beta Version at that site above and extract with that if you haven't already. Zezula lists that MPQ (wow-update-12694.MPQ) as a partial MPQ file so wow-update-12759.mpq may be as well

Saiket 08-15-10 12:30 PM

Even with the beta version of that MPQEditor, it still extracts patch files. Its file properties dialog can tell you if a packed file is a patch, but that's it. I even tried using the multi-MPQ mode with build 12694 loaded over top of 12635. Still no luck though.

Vlad 08-15-10 01:02 PM

The problem is that like in this case, I want the latest FrameXML files. I naturally open "wow-update-12759.MPQ" as it's the update file downloaded with the launcher. Inside I see the various localizations and the "base" that is common. I open enUS/Interface/FrameXML/ and see several files. I open for example "ActionButton.lua" and see this (binary data):
Code:

PTCHB`@AMD5_({†œv"*aV*Ͷ0*0XFRMBSD0˜BˆBSDIFF40AAA„e
€€        „#€
„š€…X„€€        „4%€
„€…3„E€€„€€„€K„y*€…”`ƒ;€@ET_STABLEPET_STABLEET_STABLEPET_STABLEET_STABLE_UPDATPET_STABLE_SHOWHas to update everything for now, but this event should happŽen infrequently

Basically the updates contain diff patches and if the file is new then it's normal.

*Edit*

1. Extracted "locale-enUS.MPQ" and it's FrameXML
2. Extracted "wow-update-12694.MPQ" and it's FrameXML
3. Extracted "wow-update-12759.MPQ" and it's FrameXML

I downloaded bsdiff and bspatch (Binary diff/patch utility version 4.3) and when I try to patch a file from #1 by using a file from #2 I get "corrupt file", seems like they also added some custom data to their patching files, hmm...

Borlox 08-16-10 08:33 AM

I think the BSDIFF40 part in the PTCH files is invalid.
For example the bsdiff part of ActionButton.lua from wow-update-12759.mpq says the resulting file would be 146789209322029069 Bytes long (thats 130 Petabyte) :rolleyes:
As far as I understand the format the data in bsdiff files should be compressed with bzip2, but there's clear text in the .lua diffs.

I didn't find any file format beginning with 'PTCH', so maybe that's a Blizzard invention.

Vlad 08-16-10 09:33 AM

Borlox, that's what I also found out.

There is some clear text indeed, the data that is to be inserted/replaced, e.g. is all there. Problem is figuring out where to put, replace, append, delete, e.g. I tought wowhead or mmochamp figured this out as they do read DBC files right, so they need to figure out how to patch their DBC with the updates so they can for example scan Spell.DBC for changes. ;)

Asking Wowhead mods on IRC atm, let's see if there is a way.

Borlox 08-16-10 10:28 AM

Quote:

Originally Posted by Vladinator (Post 202716)
Asking Wowhead mods on IRC atm, let's see if there is a way.

I really hope they'll give you a clue...

yssaril 08-16-10 10:50 AM

at least on windows the addonkit from blizzard still works but you need 100% downloaded/patched and need to close the laucher/wow before you start extracting

Saiket 08-16-10 11:58 AM

I think the Addon Kit is skipping the wow-update MPQs too. I can't find any of those changes in its output. As an easy test case, check for <FrameXML/Util.lua>, which should be there since patch 12694.

Vlad 08-16-10 12:31 PM

Addon Kit only extracted from my retail, not beta. Also I think it also only extracts from the locale file and not including the patches.

Verissi 08-16-10 01:59 PM

Found some (if not all) of the Interface files that were patched in this build in the Data\Cache\enUS (or whatever locale you use) directory in the build's patch MPQ. From what I can tell, though, any totally new files aren't in there at all. The new spell flyout files aren't in there, for example, but are in another MPQ in text form rather than as a patch (since they're new).

Looks like it may take a bit more work than before to extract things that can be committed to a revision control repo or diffed before a new kit is released that can cope with the new scheme. Nothing a little scripting can't handle, but it's still a change to adapt to.

Vlad 08-17-10 02:01 AM

Adys from MMO-Champion and Ladik the author for MPQ Editor (not sure of the nick entirely) to integrate this feature into the editor itself most likely, patch the files as they are extracted from the updates -very neat ey?

Adys told me that this would not be available until several weeks later so we just hang in there, someone is working on fixing this issue we have. ;)

Ladik 08-18-10 01:04 AM

I currently work on "Open MPQ in patch mode" feature of MPQ Editor. The desired functionalty is that you open a base MPQ, then add list of patch MPQs (currently, there are wow-update-12694.MPQ and wow-update-12759.MPQ), and it should seamlessly extract patched file already. I hope first experimental version might be available this week.

Saiket 08-18-10 03:09 AM

Quote:

Originally Posted by Ladik (Post 202856)
I currently work on "Open MPQ in patch mode" feature of MPQ Editor. The desired functionalty is that you open a base MPQ, then add list of patch MPQs (currently, there are wow-update-12694.MPQ and wow-update-12759.MPQ), and it should seamlessly extract patched file already. I hope first experimental version might be available this week.

Great utility, Ladik! I only just discovered MPQ Editor, and I have nothing but compliments for it; Light years better than WinMPQ. If I could request a feature, could you add a command-line option to load multiple MPQs in the given order? Once there are lots of patches, it would make loading them all up much simpler.

Something like this could work well:
Code:

MPQEditor.exe /multiple "enUS\locale-enUS.MPQ" "wow-update-12694.MPQ" "wow-update-12759.MPQ"

Ladik 08-18-10 03:15 AM

Quote:

Originally Posted by Saiket (Post 202861)
could you add a command-line option to load multiple MPQs in the given order?

Try this: MPQEditor huhu.mpq hehe.mpq hihi.mpq haha.mpq.
However, this opens the multiple MPQs in multi-mode. I guess it's time for big redesign of the entire command line syntax, because it has been there for like 6 years and too many things have changed :-)

Quote:

Originally Posted by Saiket (Post 202861)
Once there are lots of patches, it would make loading them all up much simpler.

Indeed. Yet another reason to redesign command line syntax.

Vlad 08-18-10 04:16 AM

Awesome, did you write it from scratch (the patching func) or did you borrow some ideas from the bsdiff sources? Just curious. I guess you wrote from scratch as you are probably super good at programming. :P

silkfire 08-18-10 04:20 AM

Quote:

Originally Posted by Vladinator (Post 202868)
Awesome, did you write it from scratch (the patching func) or did you borrow some ideas from the bsdiff sources? Just curious. I guess you wrote from scratch as you are probably super good at programming. :P

He is. Trust me ;) (endarion here, Ladik^^)

Ladik 08-18-10 04:21 AM

Quote:

Originally Posted by Vladinator (Post 202868)
Awesome, did you write it from scratch (the patching func) or did you borrow some ideas from the bsdiff sources?

I took ideas from bsdiff source, plus help from people from the MPQ community, as they already figured out how to interpret Blizzard version of BSDIFF. You will see it soon in the next BETA of StormLib anyway.

Borlox 08-18-10 04:44 AM

That's great news, thank you Ladik!

silkfire 08-18-10 04:47 AM

What is this MPQ community you're talking about?

Adys 08-18-10 04:48 AM

Quote:

Originally Posted by silkfire (Post 202872)
What is this MPQ community you're talking about?

First rule of the MPQ community, ...

silkfire 08-18-10 06:13 AM

That they are very secretive? Any forum you can redirect me to? Or are you its only member?

Ladik 08-18-10 06:56 AM

Quote:

Originally Posted by silkfire (Post 202872)
What is this MPQ community you're talking about?

Bunch of people who ever e-mailed me that they need something, or that there is a new MPQ format that I can't handle :-). There is no forum or an official website, most probably the guys don't even know each other. They all know my MPQ tools so they write me frm time to time.

silkfire 08-18-10 08:09 AM

Quote:

Originally Posted by Ladik (Post 202877)
Bunch of people who ever e-mailed me that they need something, or that there is a new MPQ format that I can't handle :-). There is no forum or an official website, most probably the guys don't even know each other. They all know my MPQ tools so they write me frm time to time.

Any ETA on the tool?

Ladik 08-18-10 08:11 AM

Quote:

Originally Posted by silkfire (Post 202880)
Any ETA on the tool?

Hopefully tomorrow (19.08.2010). I'll post here when at least experimental version is ready.

silkfire 08-18-10 08:44 AM

We should throw a release party :banana:

Vlad 08-18-10 10:03 AM

Quote:

Originally Posted by silkfire (Post 202884)
We should throw a release party :banana:

Yes :banana:

Crowley 08-18-10 02:39 PM

Ladik, you're the man. ;)

silkfire 08-18-10 04:37 PM

Quote:

Originally Posted by Crowley (Post 202901)
Ladik, you're the man. ;)

What's your reason?

Crowley 08-19-10 08:02 AM

Quote:

Originally Posted by silkfire (Post 202914)
What's your reason?

I want to feed my talent planner with current data.

silkfire 08-19-10 09:01 AM

Quote:

Originally Posted by Crowley (Post 202961)
I want to feed my talent planner with current data.

Link it;) Btw not a word from Ladik....makes me sad :(

Vlad 08-19-10 09:36 AM

No point in worrying nor stressing him, he will release his tool when he feels it is stable enough for a public alpha/beta release. ;)

Quokka 08-19-10 09:41 AM

Ladik can your tool be used to extract the latest .blp files from the mpq?
or is it only lua & xml?

Vlad 08-19-10 09:47 AM

It will support any kind of files in the MPQ, don't worry.

Quokka 08-19-10 10:39 AM

Vladinator thanx for the heads-up m8

Ladik 08-19-10 02:49 PM

Ok, seeing how eager you all are :p, I made experimental version of MPQ Editor that supports opening multiple MPQs in patch mode. You can download it here:
removed
This link will only be valid for a few days, I'll remove it as soon as I get something more tested. Do not use it for anything else that extracting a file from patched MPQ.

Good news: Each step of the patching process is verified by MD5, both before patching and after patching. So once you manage to extract something without an error message, you can be sure that it's correct.
Bad news: I hadn't much time to test it yet. This build is by no means finished.

Usage: Best by command line, like this:
MPQEditor.exe /patch "locale-enUS.MPQ" "wow-update-12694.MPQ" "wow-update-12759.MPQ"

In case you are unable to extract a patched file, please, send me an e-mail to zezula-at-volny-dot-cz. Please, don't forget to include:
- The name of the primary MPQ
- List of patches you entered
- Full name of the extracted file

More tomorrow, it's late here and I'll go something more relaxing for rest of the (already ending) day.

dubcat 08-19-10 06:50 PM

Quote:

Originally Posted by Ladik (Post 202998)
In case you are unable to extract a patched file, please, send me an e-mail to zezula-at-volny-dot-cz. Please, don't forget to include:
- The name of the primary MPQ
- List of patches you entered
- Full name of the extracted file

I ran this
Code:

MPQEditor.exe /patch "locale-enGB.MPQ" "wow-update-12694.MPQ" "wow-update-12759.MPQ"
And tried to extract the DBFilesClient folder, but it crashes on WorldMapContinent.dbc

Adys 08-19-10 07:41 PM

For those interested, now that Ladik has released MPQEdit with patch support. Here is a Python library to read PatchFiles (PTCH).

http://www.pastethat.com/9pGwm

Released under BSD.

Seerah 08-19-10 10:17 PM

Sorry, guys, but I had to remove the link. You know our rules! :)

DesertDwarf 08-19-10 11:17 PM

Quote:

Originally Posted by Seerah (Post 203046)
Sorry, guys, but I had to remove the link. You know our rules!

For those who don't know, see rule #8 on this post.

Vlad 08-19-10 11:32 PM

Adys, I think Jerome is a cool name. ;)

Could you post a really short sample?

Adys 08-20-10 12:07 AM

Quote:

Originally Posted by Vladinator (Post 203058)
Adys, I think Jerome is a cool name. ;)

Could you post a really short sample?

patchFile = open("wow-12345-patch/testfile.lua") # patch file
oldFile = open("old/testfile.lua")
f = PatchFile(patchFile)

print f.apply(oldFile.read())

oldFile.close()

Bluspacecow 08-20-10 12:52 AM

Quote:

Originally Posted by DesertDwarf (Post 203056)
For those who don't know, see rule #8 on this post.

I really don't see how it violates it

MPQ editor is an external program , not an addon.

Vlad 08-20-10 01:45 AM

If you try Adys' python script, append this at the end to actually use his library as a application:
Code:

if len(sys.argv) < 3:
  print "Invalid arguments, run like this (filepaths can be relative):"
  print "mpq.py \"originalfile\" \"patchfile\" \"savetofile\""
else:
  if os.path.exists(sys.argv[1]) and os.path.exists(sys.argv[2]):
    old = open(sys.argv[1], 'r');
    new = open(sys.argv[2], 'r');
    m = PatchFile(new)
    r = m.apply(old.read())
    #print r
    w = open(sys.argv[3], 'w')
    w.append(r)
    w.close();
  else:
    print "Invalid arguments, run like this (filepaths can be relative):"
    print "mpq.py \"originalfile\" \"patchfile\" \"savetofile\""
    print "The files originalfile and patchfile must exist."
    print "Remember to use quotes to encapsulate paths with spaces!"

It includes guidelines, basically takes in 3 arguments, the file, the patch and the output where to write the final file.

I think it's correct, tough I tried to patch ActionButton.lua from local-enUS.mpq with the wow-update-12694.mpq patch and getting MD5 errors.

Adys 08-20-10 01:52 AM

Quote:

Originally Posted by Vladinator (Post 203062)
I think it's correct, tough I tried to patch ActionButton.lua from local-enUS.mpq with the wow-update-12694.mpq patch and getting MD5 errors.

I'm looking into it. If it's just an error, you can pass validate=False to apply()

Ladik 08-20-10 02:14 AM

Quote:

Originally Posted by Seerah (Post 203046)
Sorry, guys, but I had to remove the link. You know our rules! :)

Ah, my apologies then. I thought that MPQ Editor doesn't fall into category of "addons" :)

For all others who e-mailed me: Thanks for bug reports, I'll fix them problems today, and hopefully a more stable version should be ready this evening.

silkfire 08-20-10 02:45 AM

Who is Jerome?

Borlox 08-20-10 05:14 AM

Edit: Found the error, the right base files are in enUS\patch-enUS.mpq, not in locale-enUS.mpq :rolleyes:

Quote:

Originally Posted by Adys (Post 203063)
I'm looking into it. If it's just an error, you can pass validate=False to apply()

I extracted the files and tried the python script with validate=False, but that gives me another error message:
Code:

>python patch.py data\locale-enUS\Interface\FrameXML\ActionButton.lua Data\update-12694\Interface\FrameXML\ActionButton.lua Data\patched-12694\Interface\FrameXML\ActionButton.lua

Traceback (most recent call last):
  File "patch.py", line 186, in <module>
    r = m.apply(old.read(), validate=False)
  File "patch.py", line 153, in apply
    nb, ob = ord(new[cursor + i]), ord(old[oldCursor + i])
IndexError: string index out of range

I couldn't get any file in FrameXML patched...

"MpqEditor.exe enUs\locale-enUS.mpq wow-update-12694.mpq" opens the file without any error message, but it's empty.
The same happens when both update mpqs are used.

Thanks for all the work you've done! I'm really looking forward to a working version :)

Adys 08-20-10 06:18 AM

Quote:

Originally Posted by silkfire (Post 203066)
Who is Jerome?

That would be me. Hi!

Thanks for the reports guys. Below is an updated PTCH parser:
http://dpaste.com/231608/

Also here is my blizpatch.py in case anyone wants it. Mind you you'll need to change imports to make it work.

http://dpaste.com/231609/

Ladik 08-20-10 06:28 AM

A new BETA build of MPQ Editor, available on my website, should be now able to handle all patches properly. I fixed several bugs there. Please, let me know if you find a file that you are unable to extract, or it doesn't look well after extracting.

This version should be far more stable that the one I published yesterday late in the night (note for myself: Never ever release anything when it's close to midnight).

As usually, there is new BETA version of StormLib available, which also contains algorithms for uncompressing and applying Blizzard patches.

Note: The original download link for the experimental version is longer valid.

Crowley 08-20-10 07:32 AM

Quote:

Originally Posted by Ladik (Post 203077)
A new BETA build of MPQ Editor, available on my website, should be now able to handle all patches properly. I fixed several bugs there. Please, let me know if you find a file that you are unable to extract, or it doesn't look well after extracting.

Works fine for me.

Thanks a lot, Ladik and Adys, your work is much appreciated.

Seerah 08-20-10 08:56 AM

Quote:

Originally Posted by Bluspacecow (Post 203061)
I really don't see how it violates it

MPQ editor is an external program , not an addon.

Even more reason to remove the link.

Quokka 08-20-10 09:00 AM

Extracted the files.
But for some odd reason there are alot 1kb .blp files
and some larger that can't be converted to .png

Don't know if this is a bug, or just the patched data

Borlox 08-20-10 10:41 AM

Is there a way to open a patch archive for multiple mpqs?
I'm trying to hack support for the wow-update-*.mpqs into the WoWModelViewer. My first idea was to open all the patch archives for each mpq, but that doesn't work, as they may be opened just one time.

Does anybody have a better idea, or is anyone already working on this?

silkfire 08-20-10 10:54 AM

Quote:

Originally Posted by Quokka (Post 203091)
Extracted the files.
But for some odd reason there are alot 1kb .blp files
and some larger that can't be converted to .png

Don't know if this is a bug, or just the patched data

Name some of the files you're referencing.

silkfire 08-20-10 10:54 AM

Quote:

Originally Posted by Borlox (Post 203109)
Is there a way to open a patch archive for multiple mpqs?
I'm trying to hack support for the wow-update-*.mpqs into the WoWModelViewer. My first idea was to open all the patch archives for each mpq, but that doesn't work, as they may be opened just one time.

Does anybody have a better idea, or is anyone already working on this?

No need, all models are viewable at least for me.


All times are GMT -6. The time now is 04:39 PM.

vBulletin © 2020, Jelsoft Enterprises Ltd
© 2004 - 2020 MMOUI