I've done a lot of typing today and REALLY don't want to type up the whole feature set of this thing, so I'm just going to paste the readme into here.

Download is here:
http://chief-01.deviantart.com/art/Halo-Bitmap-Optimizer-and-Converter-471995477

I have some examples to show what my method of DXT compression can do. The only reticule it actually messed up was the warthog chaingun. This could be avoided simply by splitting it off onto its own bitmap, but everything else ends up only taking up 1/4 the filesize it did originally. Granted, this isn't a whole lot smaller when you consider how small these reticules are, but it WOULD make a difference of maybe a meg or so for custom hud overlays the size of the entire screen or other such crap.

If you didn't need an alpha channel for transparency though, it would actually be 1/8 the size. Btw, the alpha channel is perfect. ALWAYS use DXT5 instead of DXT3 if you need an alpha channel.




The purposes of this program are to convert any and all bitmap tags to and from PC and Xbox format for use in Arsenic and the Halo Editing Kit as well as helping users optimize said bitmaps. Make sure to read all of the below stuff before you start using it, there are some VERY important things you should be aware of first. If I see anyone complaining about something that I explained below, but they didn't bother to look, I will not even acknowledge their complaints. This program requires Python 3.3 to run.

The majority of you who will end up finding this useful are those messing with Arsenic. Regular Custom Edition users may find it useful for it's ability to optimize bitmaps, convert between different formats, make DXT textures using a different method than Tool which may end up looking better, it's general ability to help you get an idea of where you can optimize your maps the most as far as bitmaps go, and the ability to prune useless data from tags, thus freeing up space in your tags folder.



Steps:

1: click "Browse..." and select the folder containing bitmaps that you want to operate on. This does not have to be a root tags folder, just a folder containing bitmap tags.

2: Hit "Load" and wait for the program to say it is finished indexing and loading all the tags.

3: Choose a tag or multiple tags in the "Tag List" window and, in the main window, specify what format you want them converted to, how many times to cut the resolution in half, and any other conversion settings.

4: Hit "Convert"

5: Go make a sandwich cause this may take a while..... Make me one too while you're at it.

6: Once the conversion is finished, a debug log will be created in the folder where the bitmap converter is located and the tag list will be cleared. The log's name will be the timestamp of when it was created.



DESCRIPTION OF SETTINGS:


===GLOBAL PARAMETERS===

---Don't reprocess tags---
Tells the program to ignore tags that have already been processed and to ignore tags that have no conversion settings different than the default ones. This box is checked by default and unchecking it should only be done if you want all the tags to be processed.
Unchecking the "Don't reprocess tags" box can be useful if you wish to prune the uncompressed original TIFF data from the tag to reduce its size. This data is pruned by tool when the tag is compiled into a map, but if you wish to reduce the size of your tags folder or reduce the size of tags you upload to Halomaps, then this may come of use.

---Backup old tags---
Tells the program to rename the tag being modified with a ".backup" extension after it has completely written the new, modified tag. Only the oldest backup will be kept; reprocessing a tag will not edit the .backup file.

---Read only mode---
Prevents the program from making edits to tags. Instead, a detailed log will be created containing a list of all the bitmaps located in the folder that was specified. The bitmaps will be sorted by type(2d, 3d, cubemap), then format(r5g6b5, dxt1, a8r8g8b8, etc), then the number of bytes the pixel data takes up.

---Write debug log---
Tells the program to write a log of any successes and errors encountered while preforming the conversion. If a tag is skipped it will be reported as an error.



===GENERAL CONVERSION PARAMETERS===

---Save as Xbox/PC tag---
Xbox and PC bitmaps are slightly different in the way they are saved. Xbox has the pixel data for each bitmap padded to a certain multiple of bytes and cubemaps have the order of their mipmaps and faces changed. A few other differences exist, but these all make a big difference. Save to the correct format.

---Save as swizzled/un-swizzled---
Texture swizzling is not supported on PC bitmaps, but is required for good preformance in non-DXT Xbox bitmaps. Swizzling swaps pixels around in a texture and makes them unviewable to humans. For PC save as unswizzled, for Xbox save as swizzled. DXT textures can not be swizzled so don't worry about that.

---Number of times to halve resolution---
I tried to think of a shorter way to phrase it, I really did. This is pretty obvious, but what isn't so obvious is that if a bitmap has mipmaps the way the program will halve resolution is by removing however many of the biggest mipmaps you tell it to.
If no mipmaps exist (HUD elements for example) the program will use a slower method of downresing, using a simple bilinear filter to merge pixels. The good side to this though, is that it calculates for gamma which means it maintains the brightness of the original image so you don't end up with faded bitmaps.



===MULTIPURPOSE SWAP===

PC multipurpose bitmaps channel usage:
Alpha: Color Change
Red: Detail
Green: Self Illumination
Blue: Specular\Reflection

Xbox multipurpose bitmaps channel usage:
Alpha: Detail
Red: Specular\Reflection
Green: Self Illumination
Blue: Color change

This program can swap the channels from PC order to Xbox order or vice versa. If you want to swap them though, make sure you are converting to a format that supports all the channels that you want to keep. For example, swapping an Xbox texture's channels to PC will require an alpha channel in the new texture if you want to keep the color change channel.



===FORMAT SPECIFIC PARAMETERS===

---Alpha cutoff bias---
Some formats (DXT1 and A1R5G5B5) are able to have an alpha channel, but it's limited to one bit. This means the only possible values are solid white or solid black. "Alpha cutoff bias" is used as the divider where an alpha value above it is considered solid white and a value below it is considered solid black. The default value is 127.

---P-8 Bump Conversion Mode---
P8-bump only has a palette of 250 colors to choose from and when you compress a 32bit or 16bit texture to it you are likely to lose some detail. This palette does not at all cover the full range of normals that you may see in a normal map, and in fact actually misses a lot of the top left, top right, bottom left, and bottom right tangent vectors that you may see. The two modes I have created each use the palette differently to achieve different results.
I could go into the specifics of this problem and how/why these two conversion methods exist, but here's the short simple answer: Auto-bias is good when you want to preserve the depth of the normal map and Average-bias is good when you want to preserve the smoothness of the normal map. Auto-bias sacrifices smoothness to allow the normal maps to stay vibrant and strong while Average-bias sacrifices the depth and strength of the normal map to allow the color gradient to stay more or less smooth.
The best mode to use is usually Auto-bias as the drop in smoothness is usually unnoticible and that is why it is the default mode.

---Monochrome channel to keep---
In A8 format only the alpha data is stored and the intensity channel(RGB merged) is assumed to be solid black.
In Y8 only the intensity channel is stored and the alpha is assumed to be solid white.
In AY8 only the pixel data of 1 channel is stored (just like in A8 and Y8), but this pixel data is used for both the alpha and intensity channels. That means the same exact image is shared between the alpha and intensity channels no matter what. This is useful for reticles for example.

This setting serves two purposes; to specify whether you want to convert to A8 or Y8 when you select "A8/Y8*", and to specify which one of these two channels to keep when you convert to AY8. Since only either the alpha or intensity pixel data is saved when converting to AY8 you need to specify which to use. The default setting is intensity.

---Swap A8Y8 channels---
On PC, HUD textures used in meters(like health and ammo) have to be 32bit color. The RGB channels are used for the image that is displayed and the alpha is used for the gradient mask that erases parts of the meter if they are below a certain value.
On XBOX, HUD textures used in meters(like health and ammo) have to be in a monochrome format. The alpha channel is used for the image that is displayed and the intensity channel is used for the gradient mask that erases parts of the meter if they are below a certain value.
HUD meters converted from PC to Xbox, or vice versa, need to have their intensity and alpha channels swapped. This setting will swap them when you convert to or from an A8Y8 bitmap.

---DXT1 Transparency---
You may know the DXT formats by Guerilla's names: "Compressed with color-key transparency"(DXT1), "Compressed with explicit alpha"(DXT3), and "Compressed with interpolated alpha"(DXT5). DXT1 bitmaps are actually capable of having an alpha channel, though it has some strict limitations. First off the alpha channel is 1bit, meaning either solid white or solid black. The other, BIGGER, limitation is that if a pixel's alpha is set to full black then the red, green, and blue for that pixel are also full black.
This type of alpha channel is perfect for things where it renders as transparency, like on the holes for the warthog's chaingun belt, but should NEVER be used for things where the alpha channel does not function as transparency, like in a multipurpose map or the base map in an environment shader.
The "Alpha cutoff bias" affects what is determined to be white and what is determined to be black.



===FORMAT CONVERSION===

The "Format to convert to" settings are more or less straight forward, but there are a few miscellaneous things you should be aware of before you convert to a format.

* This program is capable of converting to the DXT formats, though it uses a slightly different method for compression than Tool uses. This different method actually creates better UI textures compressed as DXT5 than Tool, having little to no artifacts in most cases. My compression method isn't perfect though, and is absolute poopy crap when compressing normal maps to DXT. If a texture doesn't look good as DXT when tool creates it try having tool compress it as 32 bit color and have this program turn it into DXT. The results may shock you.

*Not every format can be converted to. I've made it so only format conversions that make sense can be done. For example, there is no sense in converting a DXT1 texture to A8R8G8B8 since it is a larger file size and there is no improvement to the texture quality.

* Not all the formats this program can convert to are supported by Custom Edition. P8-bump, A8Y8, AY8, Y8, and A8 are Xbox only formats.

* Converting to 32bit color was an afterthought and as such I did not make a button specifically for it. You CAN convert the Xbox only formats(P8, A8Y8, AY8, Y8, A8) to 32 bit color though, as this would be the only way to make a usable Custom Edition texture from them. When one of these formats is selected, the "P8*/32Bit" button's function will be converting the bitmaps to 32 bit color. If a 32bit or 16bit color image is selected though, the button's function will be converting the selected tags to P-8 bump. If a mixture of these formats is selected the appropriate conversion will be used.

* Bitmaps that are not a power of 2 dimensions will be skipped entirely. So much of this program revolves around the bitmaps being in power of 2 dimensions that I did not want to try and rework all of it just to get those very rare bitmap types incorporated. The CMD window will notify you of any bitmaps that are not power of 2 dimensions and also of corrupt bitmaps.





MISCELLANEOUS WINDOW INFORMATION:

* If the program encounters an error it will be displayed on the Python CLI screen(the black empty CMD screen).

*If you wish to move the windows independent of each other click "Un-dock Windows" on the menu bar.

* The "Tag List" window can sort the tags 4 different ways. If the same sorting method is clicked again it will reverse the order the tags are displayed.

* If you want to only show certain types of tags you can enable and disable which ones show up in the Tag List window. Look under the "Enable/Disable Types" and "Enable/Disable Formats" and uncheck the types/formats you don't want to show up.

* I was originally planning a preview thumbnail, but because it would slow down browsing through tags and would be more annoying to implement than I care to deal with, I decided not to. Just open the tags in guerilla to see what they look like.

* During the tag load/conversion process the text box at the bottom of the main window will give information on which tag is being processed

* A Tag being highlighted in green signifies that, based on the tags current conversion settings, it will be processed in some way when "Convert" is clicked. If a tag is white it will be ignored when "Convert" is clicked.

* The "Selected Tag Information" window will display information about the selected tag, but ONLY if JUST one tag is selected. If more than one tag is selected the info displayed will not update. Selecting a different bitmap index on the same window will change which bitmap the window is displaying information about.

* If the program seems to be frozen then check the Python CLI screen(the black empty CMD screen). If it shows an error then the program may indeed have frozen or crashed. If not then just give it time. Depending on how you are converting it and the bitmaps dimensions, a conversion may take from a tenth of a second to 3 minutes. BUT AT LEAST IT'S AUTOMATED RIGHT?!?!?!
Edited by MosesofEgypt on Aug 1, 2014 at 06:53 PM
Edited by MosesofEgypt on Aug 2, 2014 at 05:07 PM
Edited by MosesofEgypt on Aug 2, 2014 at 05:09 PM

http://chief-01.deviantart.com/art/Halo-Bitmap-Optimizer-and-Converter-471995477 hmm will it make 2048x2048 down bye 5mb in 32bit or not?

What are you asking? Depending on the results it may be best to convert it to DXT1 or DXT5. Be warned, converting such a huge texture to DXT will take a while, probably around 5 minutes. But to be fair Photoshop takes about 1/3 the time to make a DXT texture that my program does. I've got a way to speed it up a little bit, but I won't get much of a speed improvement out of Python.

32 bit for the 2048x2048 texture big as it is it would be 30mb ..I don't use dxt1 for the most part I use 16 bit for just about all the textures I have.... I just would like to no if this thing can make 32bit 2048x2048 texture as 15mb not 30mbs
Edited by DOOM899 on Jul 31, 2014 at 07:51 AM

Each of the formats will end up with a predetermined size. This isn't like JPEG where you can specify the quality and be able to save an image of a certain resolution to the same file format and change its final size. Here's how large a 2048x2048 would be and how to calculate it.

2048x2048 = 4,194,304 pixels

32bit = 4 bytes per pixel
16bit = 2 bytes per pixel
A8Y8 = 2 bytes per pixel
Monochrome = 1 byte per pixel

DXT1 = 8 bytes per texel
DXT3/5 = 16 bytes per texel
1 texel = 16pixels


32bit = 4,194,304 * 4 = 16,777,216 bytes(16MB)
16bit/A8Y8 = 4,194,304 * 2 = 8,388,608 bytes(8MB)

Monochrome = 4,194,304 bytes(4MB)

DXT1 = (4,194,304 / 16) * 8 = 2,097,152 bytes(2MB)
DXT3/5 = (4,194,304 / 16) * 16 = 4,194,304 bytes(4MB)



This program can't make a 32 bit texture smaller than whatever it already is. It is simply not possible. Also, deciding not to use any DXT formats is entirely wasteful. There are times when you won't even notice the difference between the compressed and uncompressed textures and you'll end up with a file that's 1/8 or 1/4 the size as I described above.

Heck, I used this to optimize certain textures for CMT's B30 Evolved and managed to save around 100MB cause I took a few minutes to think about if the increased resolution and fidelity was really adding much. My DXT compression method actually works well enough on HUD elements that about half of them I compressed as DXT5 for B30 Evolved and everyone was fine with it.

Try having my program compress it as DXT. If you don't like it you can just delete the tag and rename the backup that it creates of the original. It's not perfect and like I said, it creates crap quality normal maps in DXT.



I'll post some guerilla screencaps to illustrate the lack of significant difference once I get permission from TehLag and Ifaf since they probably would count it as a leak.

EDIT: No go on the screencap.
Edited by MosesofEgypt on Jul 31, 2014 at 05:12 PM

mhm does this run wwith python 3.4 ? I didn't succed running it yet... I suck with these things

I programmed it in python 3.3.2. It doesn't use any modules other than the standard, though I did take a python 2.7 module called mTkinter which makes tkinter threadsafe and reworked it to run on python 3.3.2. THAT module may not be compatible with python 3.4.

here's a link to python 3.3.2

https://www.python.org/download/releases/3.3.2/

Also, this release is partially a wild beta test. Some features I implemented a long time ago and I haven't tested them since then, but I don't remember doing anything to break them. If anything breaks I want you guys to tell me. I'm working on improving the speed of anything involving dxt and 16 bit. I may be able to cut a quarter of the time out of some things on average.
Edited by MosesofEgypt on Jul 31, 2014 at 03:09 PM

Yep I can confirm it only works with this version of python.

Oh if you like coding and have time to waste it would be cool to have an app that sorts which tags are the heaviest on a map file, to optimize and track useless too high quality stuff or badly optmized.

Edit: tool crashed on reading the tagset

Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Python33\lib\threading.py", line 637, in _bootstrap_inner
    self.run()
  File "C:\Python33\lib\threading.py", line 594, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Thibault\Desktop\Halo\Tools\Halo Bitmap Optimizer & Converter.py", line 156, in __Bitmap_Conversion_Main
    if (Load_Tags(self)):
  File "e:\Applications\Games\Halo CE\Arsenic 2.0.0.9\CE-XBOX Bitmap & Shader Converter\ReclaimerLib\Tag_Functions.py", line 46, in Load_Tags
  File "e:\Applications\Games\Halo CE\Arsenic 2.0.0.9\Halo Bitmap Optimizer & Converter\Halo Bitmap Optimizer & Converter - Encrypted\ReclaimerLib\Tag_Constructs.py", line 72, in Load_New_Tag
  File "e:\Applications\Games\Halo CE\Arsenic 2.0.0.9\Halo Bitmap Optimizer & Converter\Halo Bitmap Optimizer & Converter - Encrypted\ReclaimerLib\Tag_Constructs.py", line 17, in Construct_New_Tag
NameError: global name 'Read_Tag_Header' is not defined

Edited by bourrin33 on Jul 31, 2014 at 05:56 PM

That's definitely weird. It's unable to find a function in the common block structures module which reads the generic 64 bytes at the beginning of a tag. The weird thing is that the function it's looking for is loaded by one of the modules it's already loading. I guess I'm gonna have to explicitly import it when I distribute this as bytecode(bytecode is the easiest way to remove all the comments).

I'll have a fixed one uploaded in a minute. I've also improved the speed on the DXT conversion by about 30% or so.

EDIT: On second thought I might as well just release it as the original .py files >_>. This will also make it so it can run on any version of python after 3.3.2 since it won't be the bytecode you'll be running.

New download is up.
Edited by MosesofEgypt on Jul 31, 2014 at 06:31 PM

I succeeding scanning the tagset, however :

Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Python33\lib\threading.py", line 637, in _bootstrap_inner
    self.run()
  File "C:\Python33\lib\threading.py", line 594, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Thibault\Desktop\Halo\Tools\Halo Bitmap Optimizer & Converter\Halo Bitmap Optimizer & Converter.py", line 252, in __Bitmap_Conversion_Main
    if (self.Get_Will_be_Processed(Tag_Index) and self.Tag_Collection[0][Tag_Index][1][2][3][30][2] != 0):
  File "C:\Users\Thibault\Desktop\Halo\Tools\Halo Bitmap Optimizer & Converter\Halo Bitmap Optimizer & Converter.py", line 454, in Get_Will_be_Processed
    not(Is_Xbox_Bitmap(Tag)) ) and self.Default_Conversion_Flags[-4]) or self.Default_Conversion_Flags[-2]:
  File "C:\Users\Thibault\Desktop\Halo\Tools\Halo Bitmap Optimizer & Converter\ReclaimerLib\Tag_Operations\Bitmap_Operations.py", line 942, in Is_Xbox_Bitmap
    return((Convert_16bit_Flags(Tag[1][6][2][9][2]) and (Tag[1][6][2][16][2] == 1073751810)))
IndexError: list index out of range

Happened when trying to convert bitmaps. The tag where it failed is not one of thsoe I wanted to edit.

Btw I understand nothing to what you say about coding...

EDIT: Okay so I'm an idiot. I fixed the issue and admittedly some other bugs related to the tag sorting in the window. You should be able to use everything perfectly fine now, even if you have corrupt or empty bitmaps(which it will now tell you are empty).
Edited by MosesofEgypt on Aug 1, 2014 at 01:05 PM

Now it highlights all the bitmaps in green as if they were going to be converted...

Yeah, logic can be confusing sometimes when you have to compare a bunch of different things and decide which path to take based on what's true and what isn't.

I've figured out there's a hole in the logic that, subconsciously I left in because all tags need to be processed at least once if they are Xbox tags and it transferred over to PC tags as well. Because I've processed all my tags before I didn't even notice this hole in the logic since they were all flagged as having been processed.

It'll be entirely safe to process all your tags at least once, especially since it makes backups if you tell it to, but I'm sure you don't want to do that since it'll be SSSSLLLOOOOOOOOWW. Also, do you have an E-mail, AIM, or Skype? Forum messaging is slow and annoying and I have a feeling I'll have overlooked SOMETHING else....


EDIT: It's fixed. Here's what the current logic looks like.

    #compares all the conversion settings of the tag with the tag's data to tell if it will be processed
    def Get_Will_be_Processed(self, Tag_Index):

        Tag = self.Tag_Collection[0][Tag_Index]

        #only run if the bitmap contains bitmaps and we are NOT in read-only mode
        if (Tag[1][2][3][30][2] == 0) or self.Default_Conversion_Flags[-2]:
            return(False)
        
        #if "Don't reprocess tags" is unchecked we'll reprocess all of them
        if not(self.Default_Conversion_Flags[-4]):
            return(True)

        #if the bitmap has already been processed, or is a PC bitmap OR if we are just creating a debug log we start to skip it
        if Is_Processed_by_Reclaimer(Tag) or not(Is_Xbox_Bitmap(Tag)) or self.Default_Conversion_Flags[-2]:
            Flags = self.Conversion_Flags[Tag_Index]
            Format = Tag[1][6][2][2]

            #if all these conversions end up being default we skip processing the bitmap by returning False
            if ( Flags[2]=='0' and Flags[3] == 0 and Flags[9] == 0 and Is_Xbox_Bitmap(Tag) == Flags[0] and (Flags[7]== False or Format!= 3)
                 and(Is_Swizzled(Tag) == Flags[1] or (Format == 14 or Format == 15 or Format == 16)) ) or self.Default_Conversion_Flags[-2]:
                return(False)
        return(True)

Edited by MosesofEgypt on Aug 2, 2014 at 12:10 PM
Edited by MosesofEgypt on Aug 2, 2014 at 12:17 PM