Python based tag manipulation library?

This is about a year in the making and I'm about 99% done with the entirety of it. I have a few weeks left before I feel confident it is ready for extensive testing.


This isn't exclusively a Halo library either. It's an all purpose pure python binary structure library with no dependencies.

This is going to be released for free when I'm done with it, but I'm going to be ready for testing in a week or so.

So, any takers? I'll need proof of your experience in Python so I don't just hand it out to anyone. Proof will be done on a case by case basis.


After I'm done with the library I'll be using it to write that "Guerilla 2"(it wont be called that) that I've been talking about for a while. This will allow you to edit any binary structure you define in the same way you edit tags.
Edited by MosesofEgypt on Jul 16, 2015 at 01:32 AM

I just saw this and hugged the screen. Plze help him so we can get guerrila the sequel already!

I'd love to help, but even though I know a bit of Python, I've never actually done anything worthwhile with that knowledge.

One way that someone could really help is to help write the descriptors for the rest of the tag formats. A descriptor is basically a nested dictionary with keys describing the structures. I've already got the descriptors made for:
bitm, hmt, metr, pphy, schi, scex, senv, soso, str#, ustr, and tagc

which are:

bitmap, hud_message_text, meter, point_physics, shader_transparent_chicago, shader_transparent_chicago_extended, shaver_envrionment, shader_model, string_list, unicode_string_list, and tag_collection.

It's not a lot, but these were necessary ones with just the right amount of diversity for me to implement the right features in the library. That's 11 tag types down and 62 to go. Implementing Open Sauce support is as easy as just adding the proper structure data. Shader_Model already supports open sauce and I used CMT's cyborg shader models to help me test it.

Here's a test to show you what the descriptor for a point_physics tag looks like, and what it looks like when I load it and print it out in the command prompt.


I would have shown a more complicated tag, but a lot of the code is cut away into shared blocks and is only referenced in many spots, meaning you wouldn't SEE what all's actually in it just by looking at one file. I actually had to paste the header back into the point physics descriptor there to show it. Normally that field looks like this.

0:Combine( {0:{ DEFAULT:"pphy" } }, Tag_Header),

The library is still being worked on so speed isn't my primary concern right now. Versatility, ease of use, as low a memory footprint as possible for python, and type safeness are.

Here's another example, this time it's part of the Gauntlet Dark Legacy xbe. More specifically, it's the section in the executable where the secret character data and cheats are stored. It also shows off the Struct_Array, which is a data type that allows you to create an array of structures.

Edited by MosesofEgypt on Jun 14, 2015 at 03:52 PM
Edited by MosesofEgypt on Jun 14, 2015 at 04:59 PM

I am actually going to be relying on the tag structures you provided me. They will be what I write the rest of the descriptors from. If you're directly offering help writing the tag descriptors then I'd gladly accept it. Also, I don't really consider us to be competing. I'm writing something that happens to be able to do what your program will be able to. Mine will be much more versatile, but yours will be faster in every way and tailored specifically for Halo. We really aren't aiming for the same thing, just running in parallel in some areas.

Also, as of right now I'm specifically aiming for just the HEK versions of the tags. I can derive the meta versions from those later once I've got the V1.0 stuff working. Also, map structures would be nice, but again, not until the HEK formats are done.

I've written this library to be able to handle very arbitrary structure configurations as well as a very useful meta structure parser. It's also extendable so you can write your own data types, readers, writers, encoders, decoders, etc. I've built it to be more like the 010 editor(just found about it yesterday) than Guerilla. I only call the complete structures "Tags" to separate the whole object that has a filename attached to it(and an extension, link to the constructor and handler, etc) from the data type called Struct. In my library, a tag is simple an object that stores a potentially nested series of structures with data.

You can specify the size of something(like strings or integers) in different ways. You can specify it as a constant that is defined in the descriptor, it can be fixed and defined by the data type(UInt16 has a fixed size of 2 for example), you can specify it as a string that points to some other variable in the structure(such as halo's reference strings), OR you can specify a function which will be given information the current block and it will use that to determine the appropriate size(such as TGA files that determine their sizes as width*height*BPP).

Here's the tga descriptor to show you what I mean when I say that a function can be provided as well as showing the Bit_Struct. When within a bit struct, offsets and sizes are in bits instead of bytes. Keep in mind that I had to move brackets and shrink the font size so the descriptor and one of the 'Get_Size' functions would fit on screen.

I don't yet support RLE compressed TGA textures because I'd have to write a custom reader for it which, while it isn't hard to do, I'm just not wanting to find a good place to put the routine yet. I don't want to make a collection of miscellaneous structures and special readers for certain parts of them until the rest of the library is done. Also yes, I realize that part of that function is redundant. Fixing it now lol.

I have also added support for structures on a bit level which can be adapted for various data size encodings. I'll be using this for the compressed normals in models and bsp's so you can edit them as 3 separate 11/11/10bit signed scalar ints rather than 1 large unsigned int.

Also, these descriptors don't need all the info that you're seeing. If you leave out the offset it'll calculate them for you based on the offset and size of the previous data(though I don't currently have support for specifying byte alignment), if you leave out the field's NAME and just provide a GUI_NAME(which is what will be shown in the gui based editor), it will try to convert the GUI_NAME into a suitable identifier and use that. The only things that really NEED to be specified, are TYPE, NAME(or GUI_NAME), and SIZE(if the type has a variable size like raw data or a string).


And lastly, here's an example of my loading a tga file in the command prompt and modifying it. Notice how the Image_ID_Length automatically updates to what the new size of the array is when I place a new bytearray in it.

If the writing routines were finished I could save this just by calling test_tga.Write(). You can also load and save individual structure pieces, so like, I could save a function block from a weapon and load it into another weapon. or I could save the whole array of them and replace the entire "functions" block of one weapon with the entire "functions" block of another.
Edited by MosesofEgypt on Jun 14, 2015 at 06:04 PM

Well, I'll let you know when the library is done enough to start building descriptors for it. I have absolutely no documentation on it other than the comments strewn around the functions, and not everything has enough of a description.

Once I get a few more things tweaked I'll be able to start compiling documentation on how the library works. There's a few levels of abstraction, depending on how you want to work with it.

you should ask WaeV over at opencarnage. He's been working on a slick little program called Quickbeam for a while now, and I know it uses a lot of python in its construction.

http://opencarnage.net/index.php?/topic/981-quickbeam/

He'd probably be able to test this stuff out thoroughly, and it might even fill some gaps and such that he may have. Could be a win/win for both of you
Edited by NeX on Jun 17, 2015 at 11:05 AM

I will certainly talk to him about it. For right now though, I'm gonna focus on the write routines. Should have them finished in a couple days. After that I'll be doing a couple reorganization things in the debug log creation and the thing should be ready to test.


Edit: I can now successfully save structures. I just need to implement 3 more routines and figure out a good way to handle setting meta pointers and the write routines will be done.
Edited by MosesofEgypt on Jun 17, 2015 at 08:28 PM