Background on REWise

Published: 05-09-2024

The REWise project started while packaging older proprietary game assets (for private use) that have a free-software implementation of the game engine. For some of the games the assets could just be copied from the install medium, others where done with innoextract. But then I got stuck with packaging assets for the use with Nuclide and iortcw.

After viewing some installer files with a hex-editor it was noticed that they where all created with something called 'Wise Installer', the string Initializing Wise Installation Wizard kept popping up.

Searching online for a tool to extract files form Wise installers had some results, most of them where however instructions on how to extract the files by executing the installer from cmd on that proprietary OS. But I found two projects that looked prommising:

Both projects are unusable for me but they revealed that extracting files is most about deflated data at the overlay-offset of the PE installer file, so that is how it started.

The inner workings of DEFLATE where new to me so I studied how WiseUnpacker was doing it while trying to make sense of it all with the .ZIP Specification and DEFLATE Specification:

And even dived into what Huffman coding coding is and how it works, it's amazing! (https://en.wikipedia.org/wiki/Huffman_coding)

Confident enough, coding has started in the Python language because it's easy prototyping and it has a library called pefile what makes it very easy to get the overlay-offset. The first attempts failed and now a confession has to be made, wanting some feedback on what was going wrong, so wrote a small simple main program for WiseUnpacker (it's a library). Yeah.. I actually tried to compiled a C# program. Many demons had been fought but eventually it compiled and different Wise installers where tested. On all installers the inflation worked but the renaming/moving of the inflated files did not work on all installers, we get back to that later and ignore it for now since the first step is a working inflation process.

After putting some debug prints in WiseUnpacker and comparing them with debug prints of the Python project it was found that some data types where not behaving in Python as I thought, after some tweaking, some more debugging and more tweaking it finally was outputting valid files. No filenames however, just EXTRACTED_%09d.dat. Great the first results!

So the first result was booked with the Python project, but it was very slow! Then I did a rewrite in C, it was much much faster, so cool!

Next was to make sense of the extracted files, this was another journey. After studying WiseUnpacker it was clear that it is doing a lot of guess work and was looking for a inflated file containing some stuff (filename and file-datetime), so that is interesting but the guess work apparently does not work on all the installers I tested. More research needed!

After comparing the first X (20? IDK) inflated files from different installers with a hex-editor it was noticed that the first inflated file is always a DIB file and on all the tested installers it was even the same (md5). The second inflated file seemed way more interesting as it contained a lot of strings that are clearly filenames. Also note that the UNIX file command is very useful to get most file types of unknown files (should have used it in the first place).

The first try of renaming/moving the inflated files was to just go over all filenames from the second inflated file and see what happens, but nope that did not work out very well..

Then the time did come to do the initial reverse engineering of the second inflated file, by comparing multiple ones. With the use of ImHex and it's PatternLanguage to write and test different structs on different files, it was not long before realizing there is a header and after the header a OP code + data for that OP code, then repeat OP code + data utill EOF. It was just a matter to get the data structure for each OP code right, also knowing what OP codes are used, this did take some time because when the data structure was not right, then a random byte would be assumed as a OP code.. and there where messy results.

After some time of puzzling all shareware installers known to me at that time and all the installers of retail CD's I own where extracting as expected with their file-path/-name and even CRCs where valid. At this time the code was cleaned up and checked multiple times, then published the code to notabug and left a message at the FragNet #lobby IRC channel and got fast and positive reactions, some of which resulted to some changes in the source to support different platforms like *BSD and MSYS2, so thanks to them for the feedback!

The first release of REWise was born, version 0.1.0, this was somewhere in May 2023. I don't know exactly when this all started but approx end Jan / begin Feb 2023. So it took something of 4 months to come to the first release. This all was done as a hobby in spare time.

There has been progress afterwards, like replacing the custom inflate and CRC32 functions with zlib, it inflates much faster now.

Not that long ago I came across some installers that are currently not working, after some inspection those turned out to be either Multi-Disc installers, installers created with the zip option and/or Multi-Language installers. So time to do some bit hunting again, for recent development changes see the dev branch.