I'm sure that anyone who ever come across these tool compile errors -

the meter definition ui\hud\cyborg shield does not specify a stencil bitmap group.
the meter definition ui\hud\cyborg body does not specify a stencil bitmap group.

has been at least a little curious as to what's in them and why they are there.

The error is because guerilla doesnt know about the stencil data im about to talk about,
so if you resave a meter tag after opening it, guerilla erases the stencil data and
then tool complains that it isnt there. This doesnt matter for custom edition maps,
but xbox maps wont load without a valid stencil.

Well I got bored yesterday and decided to give it a crack. Through knowledge of
binary structures and hunches I was able to determine that the actual stencil data
is stored in a slightly compressed format, where the image is divided up into
horizontal lines that each have a 6 byte little endian header composed of unsigned
x and y offsets that the line starts at and the number of pixels wide the line is.
The header is ordered like this:

0:{ TYPE:UInt16, NAME:"X_Pos" }
1:{ TYPE:UInt16, NAME:"Y_Pos" }
2:{ TYPE:UInt16, NAME:"Width" }

There is nothing specifying how many lines exist, so you just have to try to read
another line and if there isnt enough data then you've reached the last line.

The body of the tag's data specifies the dimensions that the meter image will be
when uncompressed. This is in a field that is hidden from guerilla, but is easy
enough to locate in a hex editor. Meter_Width is at offset 212 and Meter_Height
is at offset 214. Both are little endian unsigned Int16s

Using all of this knowledge, I was able to write an extractor that assembled the lines
of ui/cyborg_body.meter and ui/cyborg_shield.meter into a single tga image.
The red channel is completely blank for some reason, the green channel LOOKS like
its 90% noise, the blue channel looks like its probably the image opacity, and the
alpha seems to pretty clearly be the meter gradient mask.

ARGB:

Alpha:

Red:

Green:

Blue:



I know that you guys have to be completely enthralled with this super useful discovery.
Seriously though, I was hoping to find some kind of hidden halo asset that none of us had
ever seen before and it'd be kinda cool to look at, like something in a museum.
Thought some of you might find it interesting.


Here's a link to a folder containing all of the image data separated into channels and with the
original raw image as well. The combined.tga is the raw data that has been untouched by
photoshop's grimy tweaking hands(photoshop likes to rescale values before saving), so if
anyone wants to try and figure out how the image color data is represented, that's your best bet.

https://www.dropbox.com/sh/4kv854tdwf3rwqt/AABBiL2mJMnj4aAVba844IDGa?lst


EDIT:
Actually, if you look at ui/hud/bitmaps/hud_health_stencil.bitmap you'll see that
the bitmap 0 merged with bitmap 2 make up the blue channel above and bitmap 1 merged
with bitmap 3(masked off by the blue channel) make up the alpha.

This is starting to lead me to believe they really did just end up corrupting the
green channel somehow. Either that or they only wanted the opacity and meter mask.
They could do this by using the blue channel as the transparency for a solid color
that they determined by mixing the empty_color and full_color(defined in the tag)
based on how full the units health of shield is. Idk, need to get back to homework tho.
Edited by MosesofEgypt on Jan 18, 2016 at 02:40 PM

Can I just say, I have spent weeks in the past trying to figure this weird yet harmless error out using what little intellect I possess.

Then one day you arrive with your superior intelligence and and technical know how, and you figure out the damn thing within a second!

Regardless begrudgingly I'm forced to congratulate you on your endevours.

Btw how are the 3d printed helmets coming along?

https://www.dropbox.com/sh/4kv854tdwf3rwqt/AABBiL2mJMnj4aAVba844IDGa?lst

Edited by Super Flanker on Jan 18, 2016 at 05:35 PM

Truth be told, I spent a while trying to figure it out the first time I wrote a definition for the meter tag for my library. I spent a day or so trying to tear apart the image data without any luck and just said "eff it, I've got more important things to do."

Btw, about my python library(remember that thing? it's coming along nicely). I've been working on it a ton and it's finally almost feature complete(talking about the non-Halo specific stuff). I took a small break this week to add some halo related stuff to it, a few of which were a couple things to test it out.

One of those tests was what I call HEK_Tag_Scanner. It loads up all available tag definitions(I currently have only written 32 out of the 83 possible tag types), scans each of them and logs if/where each one can reference another tag, and builds a list out of that. Then it scans the tags directory you have specified and loads up all tags that can reference each other, checks if any of the references are invalid, and makes a log of all of this. Here's the console printout:

This program will scan the tags directory and locate
tags that reference other tags. A log will be created
in the tags directory and any tag references that
cannot be found in the tags directory will be logged.

This program will periodically print the path of the
tag it is currently indexing/loading/scanning relative
to the tags directory as a sort of progress update.

Press Enter to begin scanning in:
    e:\Applications\My Repos\Reclaimer\tags\



Indexing...

Loading 22 tags...

Scanning 22 tags...
    Scanning 'Soul' tags...
    Scanning 'devi' tags...
    Scanning 'flag' tags...
    Scanning 'fog ' tags...
    Scanning 'foot' tags...
    Scanning 'hud#' tags...
    Scanning 'item' tags...
    Scanning 'itmc' tags...
    Scanning 'metr' tags...
    Scanning 'mply' tags...
    Scanning 'ngpr' tags...
    Scanning 'scex' tags...
    Scanning 'schi' tags...
    Scanning 'senv' tags...
    Scanning 'sgla' tags...
    Scanning 'smet' tags...
    Scanning 'soso' tags...
    Scanning 'spla' tags...
    Scanning 'swat' tags...
    Scanning 'tagc' tags...

Writing logfile...
--------------------------------------------------------------------------------
Finished scanning tags directory.
Check the tags directory for the log.
--------------------------------------------------------------------------------

Press enter to exit.

and here's the output log

--------------------------------------------------------------------------------
2016-01-18  17:55:41
Debug log for HEK Tag Scanner


halo\ui\test.ui_widget_collection
    UI_Widget_Definition
        ui\shell\solo_game\pause_game\pause_game.ui_widget_definition
        ui\shell\solo_game\pause_game\pause_game_split_screen.ui_widget_definition
        ui\shell\error\error_modal_fullscreen.ui_widget_definition
        ui\shell\error\error_modal_halfscreen.ui_widget_definition
        ui\shell\error\error_nonmodal_fullscreen.ui_widget_definition
        ui\shell\error\error_nonmodal_halfscreen.ui_widget_definition
        ui\shell\solo_game\player_help\player_help_screen_a10.ui_widget_definition
        ui\shell\solo_game\player_help\player_help_screen_a30.ui_widget_definition
        ui\shell\solo_game\player_help\player_help_screen_a50.ui_widget_definition
        ui\shell\solo_game\player_help\player_help_screen_b30.ui_widget_definition
        ui\shell\solo_game\player_help\player_help_screen_b40.ui_widget_definition
        ui\shell\solo_game\player_help\player_help_screen_c10.ui_widget_definition
        ui\shell\solo_game\player_help\player_help_screen_c20.ui_widget_definition
        ui\shell\solo_game\player_help\player_help_screen_c40.ui_widget_definition
        ui\shell\solo_game\player_help\player_help_screen_d20.ui_widget_definition
        ui\shell\solo_game\player_help\player_help_screen_d40.ui_widget_definition
        ui\shell\error\error_abort_to_dashboard.ui_widget_definition
        ui\shell\error\error_abort_to_dashboard_you_have_no_choice.ui_widget_definition
        ui\shell\main_menu\loading_screen.ui_widget_definition


halo\bases\test.device
    Repowered
        repowered.sound
    Delay_Effect
        delay effect.sound
    Open
        open.effect
    Close
        close.sound
    Opened
        opened.effect
    Closed
        closed.sound
    Depowered
        depowered.effect


halo\systems\test.flag
    Physics
        flag physics.point_physics
    Blue_Flag_Shader
        blue flag.shader
    Red_Flag_Shader
        red flag.shader_transparent_generic


halo\level\test.fog
    Fog_Map
        fog map.bitmap
    Background_Sound
        background sound.sound_looping
    Sound_Environment
        sound env.sound_environment


halo\systems\test.material_effects
    Effect
        some effect.effect
    Sound
        some sound.sound
    Effect
        some effect.effect
    Sound
        some sound.sound
    Effect
        some effect.effect
    Sound
        some sound.sound
    Effect
        some effect.effect
    Sound
        some sound.sound
    Effect
        some effect.effect
    Sound
        some sound.sound
    Effect
        some effect.effect
    Sound
        some sound.sound


halo\ui\test.hud_number
    Digits_Bitmap
        digits bitmap.bitmap


halo\bases\test.item
    Material_Effects
        material effects.material_effects
    Collision_Sound
        sound.sound
    Detonating_Effect
        effect1.effect
    Detonation_Effect
        effect2.effect


halo\systems\test.item_collection
    Item
        some item.item


halo\systems\test.multiplayer_scenario_description
    Descriptive_Bitmap
        some bitmap.bitmap
    Displayed_Map_Name
        im guessing a string.unicode_string_list
    Descriptive_Bitmap
        some bitmap.bitmap
    Displayed_Map_Name
        im guessing a string.unicode_string_list
    Descriptive_Bitmap
        some bitmap.bitmap
    Displayed_Map_Name
        im guessing a string.unicode_string_list


halo\systems\test.preferences_network_game
    Pattern
        some pattern.bitmap
    Decal
        some decal.bitmap


halo\shaders\test.shader_environment
    Lens_Flare
        lens flare test.lens_flare
    Base_Map
        base.bitmap
    Primary_Detail_Map
        pri test.bitmap


halo\shaders\test.shader_transparent_glass
    Diffuse_Map
        diff map.bitmap
    Diffuse_Detail_Map
        diff detail.bitmap
    Specular_Map
        spec.bitmap
    Specular_Detail_Map
        spec detail.bitmap
    Background_Tint_Map
        back tint map.bitmap
    Reflection_Map
        refl.bitmap
    Bump_Map
        not a height map.bitmap


halo\shaders\test.shader_transparent_meter
    Meter_Map
        meter map.bitmap


halo\shaders\OS test.shader_model
    Specular_Color_Map
        SPECULAR.bitmap
    Detail_Normal_2_Map
        DETAIL_NORMAL_2.bitmap
    Base_Normal_Map
        NORMAL.bitmap
    Detail_Normal_1_Map
        DETAIL_NORMAL.bitmap
    Reflection_Cube_Map
        cubemap.bitmap
    Diffuse_Map
        base map.bitmap
    Multipurpose_Map
        multipurpose map.bitmap
    Detail_Map
        detail map.bitmap


halo\shaders\test.shader_model
    Reflection_Cube_Map
        cubemap.bitmap
    Detail_Map
        detail map.bitmap


halo\shaders\test.shader_transparent_plasma
    Primary_Noise_Map
        pri noise.bitmap
    Secondary_Noise_Map
        sec noise.bitmap


halo\shaders\test.shader_transparent_water
    Reflection_Map
        reflmap.bitmap
    Base_Map
        base.bitmap
    Ripple_Maps
        ripple.bitmap


halo\misc\test.tag_collection
    Tag
        ui\multiplayer_game_text.unicode_string_list
        ui\shell\bitmaps\white.bitmap
        ui\shell\main_menu\settings_select\player_setup\player_profile_edit\controls_setup\controls_keyboard_button_names.unicode_string_list
        ui\shell\main_menu\settings_select\player_setup\player_profile_edit\controls_setup\controls_mouse_button_names.unicode_string_list
        ui\shell\main_menu\settings_select\player_setup\player_profile_edit\controls_setup\controls_axis_direction_names.unicode_string_list
        ui\shell\main_menu\settings_select\player_setup\player_profile_edit\controls_setup\controls_device_labels.unicode_string_list
        ui\ui_input_device_defaults.tag_collection
        ui\shell\main_menu\settings_select\player_setup\player_profile_edit\controls_setup\controls_gamepad_names.unicode_string_list
        ui\shell\bitmaps\cursor.bitmap
        ui\saved_game_file_strings.unicode_string_list
        ui\ui_default_profiles.tag_collection

As you can see, all the references are missing because these are just some test tags that I threw reference paths into for debug purposes. This will come with my library, along with other things. For example, I intend to make a tag dependency list displayer and a program to copy a tag with all its required dependencies to a separate folder.


Also, about the helmets, I'd like to know the same thing. College and my library have been taking most of my time and I've been too busy to even call my friend, but if things havent changed since a few months ago then I know he's having a super tough time selling them. He couldnt find enough preorder buyers and had to lower the price and make the first few at a loss. I might ask him soon enough.

Even though I personally have little to no problems finding at fixing tags with broken directory references.

Your dependancy viewer should significantly stream line the process of locating the specific tags and ammending their respective fields and directory paths.

Thankyou for investing your crucial time and energy into something which I undoubtedly believe will assist all corners of the map modding community!
Edited by Super Flanker on Jan 18, 2016 at 06:08 PM

Well more than that, it can help you locate tags that have problems, but that you aren't actively using. Like if you were creating assets and considered something done, then went and made other stuff, renamed something that your asset needed, and now something is broken that you dont feel the need to check again.

And dont worry about me using my time on this, I've been a part of the halo modding community since 2004 when I was messing with Halo Trial and HMT. I love making things for people to play with that help them. This also helps me test my library since it exercises every part of it.
Edited by MosesofEgypt on Jan 18, 2016 at 06:22 PM

This has actually given me an idea for a feature.

I am no programmer but would it be possible to include a function which will remove all tags harbouring broken sub tag file paths so a map utilizing broken tags can still compile successfully?

Let me elaborate:

Say I have a map utilizing a custom weapon tag.

However the wep tag references files which are missing or perhaps accidentally deleted

Would it be feesible to set a command to either:

A) Automatically delete all errorness tags referenced in the scenario.

Or

B) Have tool somehow build anyway whilst ignoring the broken tags.

Personally I feel method a to be more destructive whilst method b to be more complicated to achieve.
Edited by Super Flanker on Jan 18, 2016 at 06:32 PM

I could literally just add:

Block.Filepath = ""

to line 136 in the program and then this to line 137

Tag.Write(Temp=False, Int_Test=False, Backup=False)

and it would accomplish method A. It would still log all the missing filepaths though, which could be used later to replace the missing references with their old names.

Alternatively, line 137 could be

Tag.Write(Temp=False, Int_Test=False, Backup=True)

and what that would do is it'd write the tag, and keep a backup of the old. The backup would be the same filename, but with .backup appended to the end.

Method B isn't feesible since I'm not controlling tool. This library is its own standalone thing.


EDIT: I just realized you meant only things required by a scenario. It'd require some more work since I'd have to implement the other 51 tag types and then write a new loading routine that loads all the tags tool looks for(like globals, hud_globals, the scenario, etc) and then load all dependencies of each of those tags until all required tags have been loaded. From that point on the code would be the same as what I've currently got, so no changes.

I wont be releasing any of this stuff until I have an actually complete 83 tag definition set, otherwise it'd be almost pointless.

EDIT EDIT:
Btw, in my pursuit to make definitions for all tag types, I've even worked on the ones that guerilla crashes trying to make, the base classes(shader, device, unit, object, and item). I tricked guerilla into making a few for me to pick apart, and I dont have time to do anything else with them other than write the definitions. Guerilla still loads and saves these just fine. As long as you have them you can load and edit them.

https://www.dropbox.com/sh/mw48xl8nk25uy5f/AAAG1SibGvQJ90wozJdcyYVoa?dl=0

The reason I'm sharing these is because I'm curious as to how halo will actually handle these and if they will actually work in a map. Let me know what happens. They're basically pieces that larger tags are composed of, like a biped and a vehicle are both a unit and object, so they have those tag structures combined into them.
Edited by MosesofEgypt on Jan 18, 2016 at 07:36 PM

I'll download, have a little play then report back if I incur any problems.

No problem, and yes, I'm doing this in python. I don't have the c structs(you only handed me mode and i think mod2), but they would be helpful. I don't have any OS defs other than the soso, and that one I've had for a while. I've been really just making the tag in guerilla, entering in unique information per field, saving it, loading it in a hex editor, removing the header, and recording the offsets the values are at. I intend to keep doing it this way, and then filling in any missing values with data from external sources, like xml plugins, OS defs, and your c structs. I'll especially need to consult external sources for information about like, is this a uint8, 16, or 32, since my tag defs can also be directly imported into the .map definition and reused as meta defs.

I'll be able to write a tag ripper when I've got all 83 defs written, and I intend to use a hash:tagpath mapping to determine the filepath of protected tags based on their data(nulling out the pointers first to make sure they dont screw the hash). If they dont match a hash it'll do a heuristic check to determine what the tag is used by, what it is, and try to organize it by that. Like, I could have it check the weapons string list for the name of a weapon, sort it into a folder under that name in weapons/ and put all its dependencies that only it uses into that folder with it. I intend to also make it so you can create a set of hashes from already known tags(like say, your cmt folder) so it is expandable. I don't believe in people protecting their content when there is nothing at stake for them, especially since it stifles learning and additional content creation.
Edited by MosesofEgypt on Jan 19, 2016 at 01:06 PM