Yeah why do you want to make a new one??? Seems like an odd request.

While I'm not aware of any potential limitations on the size/tagcount of sounds.map, bitmaps.map, or loc.map, the files could be easily expanded upon. The format is pretty simple and I intend to do tests later on to see if I can, but here's what I've found out so far.

The file starts out with a 16 byte header formatted like so.

0:{ TYPE:Enum32,    NAME:"map_resource_type",
  0:{ NAME:'Bitmaps', VALUE:1 },
  1:{ NAME:'Sounds',  VALUE:2 },
  2:{ NAME:'Strings', VALUE:3 },
  },
1:{ TYPE:Pointer32, NAME:"tag_paths_pointer" },
2:{ TYPE:Pointer32, NAME:"tag_headers_pointer" },
3:{ TYPE:UInt32,    NAME:"tag_count" },

which is followed by 2 parallel arrays. The first is an array of null terminated filepath strings, and the second is an array of 12 byte tag headers. Parallel arrays means both arrays have the same number of entries and each entry in one directly corresponds to the same index entry in the other array.

Example: entry 21 in the filepath strings in bitmaps.map is scenery\halo\bitmaps\halo outer ring and entry 21 in the tag data array is header of that tag.

The tag headers are each formatted like so

{ TYPE:Struct, NAME:"tag_header",
  0:{ TYPE:UInt32, NAME:"id" },
  1:{ TYPE:UInt32, NAME:"size" },
  2:{ TYPE:UInt32, NAME:"offset" }
  }

where offset points to the actual meta data of the tag in the file, size is how many bytes large it is, and id is something I don't yet fully understand, but I know it's an identifier number related to how halo catalogs and accesses each tag in memory.

Though, for some reason sounds are split into an entry for the raw sound data and an entry for the tag meta that describes it. bitmaps are split the same way, an entry for the lump of pixel data and an entry for the tag meta that describes it. Here's what the strings array looks like for a few tags:

weapons\assault rifle\fp\bitmaps\assault rifle_fp__pixels
weapons\assault rifle\fp\bitmaps\assault rifle_fp
weapons\assault rifle\fp\bitmaps\assault rifle multipurpose__pixels
weapons\assault rifle\fp\bitmaps\assault rifle multipurpose
weapons\assault rifle\fp\bitmaps\display_fp__pixels
weapons\assault rifle\fp\bitmaps\display_fp
weapons\assault rifle\fp\bitmaps\display multipurpose__pixels
weapons\assault rifle\fp\bitmaps\display multipurpose

The meta data that is pointed to is literally the same as you'd find in a map, but there is no indication as to what each tag type is since the resource maps dont store a 4 character id(like bitm for bitmap, mod2 for gbxmodel, snd! for sound, etc)so you kinda already have to know what format the tag is before you try to access it. That's not a problem for bitmaps and sounds.map, but loc.map can store hud_message_text, font, unicode_string_list, and string_list tags.


tl;dr
I don't know of any tools out there to do this, but I have the knowledge and framework to start trying it out when I have the time.

To answer matt, you could create a larger resource map that holds shared bitmaps that are frequently used so as to make smaller maps. I imagine you could also do stuff like fix the borked multipurpose maps for the default tagset and upload them so people only need the fixed bitmaps.map.
Edited by MosesofEgypt on Jan 30, 2016 at 01:22 PM

Yes, that would be pretty fantastic - having a "stock+" bitmaps.map file that contains everything required for stock maps, but also contains say, a series of bitmaps that recur in a series of new maps.

OS has some kind of "mod" utility that repacks shared data into new resource cache files, but AFAIK they don't add to the existing one, so you would have to swap them out as needed.

Being able to add a tag to it would be fantastic - what really matters though is how the game knows not to add it to the map cache file itself, but rather just provide the reference to the resource file's entry instead. I would assume that tool does a scan of the tags during compilation and checks them against those in the resource files, and then adds either the data or the reference appropriately - but that's just a guess. It may have some internal reference mode that has a static copy of what's referenced in the resource files, so that adding to the bitmaps.map would do nothing, as tool would have no idea if it doesn't scan through the existing resource files each time.


Personally, I would start by determining what properties or conditions make tool refrain from putting the data in the map cache itself in favor of using the resource file. Is it just the name? Matching tag names/header info? Would recompiling a new bitmap with the same name as one in the bitmaps.map file result in the new bitmap tag being added to the map cache, or would the map simply reference the name to use the data in the bitmaps.map?




This has been done IIRC. Programs like HMT and Eschaton actually directly edit the bitmaps.map file when doing bitmap injection. Look at early iterations of the FTW mod, or other PC-based mod projects, and many of them will come with a caveat of "make sure you download this bitmaps.map". Unfortunately, the images must keep the same parameters, so you can't replace a color-key-transparency 512x512 bitmap in the bitmaps.map with an interpolated-alpha 512x512, nor can you change the size. If you try to, either the image will lose quality as it is downgraded (dunno), or (more commonly and what I have seen personally) it will brute force SHOVE the new data in, shoehorning it in between the next references, which ends up corrupting all data from that point on in the bitmaps.map file, resulting in some crazy results. You may end up seeing some horridly corrupted version of a blood splatter sprite instead of wall plating textures - or have a solid block instead of a reticle, etc.

I'm pretty sure it just checks if the tag path is already located in the appropriate resource map and then maybe some secondary data comparison checks. I'm pretty sure I'd just have to increase the size of the arrays for the new entries, add the new tags, fix all the pointers, and resave it and it'd probably just work as is with tool. Tool would scan it to see if the tag its trying to compile already exists in the appropriate map, sees it does, and doesnt compile it into the map.
Edited by MosesofEgypt on Jan 30, 2016 at 01:51 PM

Bump.

I'm gonna tear down what I said a little bit though. If one were to make an expanded shared resources map, then both the end user and the map maker would have to be using it for it to have any effect.

If the map maker has it, but the end user doesnt, the end user has to download it to get the map to work. If the end user does, but the map maker doesnt, then there is no benefit to having it since the map will still be compiled with the 'shared' tags.

Another problem is that the ordering of the tags in the resource maps is very important. when a map is compiled it specifies each tag in the maps tag array with a header. That header says if the tag is indexed in a shared map, and if so, which index it's in. That means if you have a map that uses a modified bitmaps.map(lets call it A.map), but you have a different one(B.map) which contains the right tags, but they are in a different order, it wont be able to find the proper tag.

Modified resource maps are much MUCH more a logistics problem than an implementation problem. We have these default resource maps that everyone uses because they came with the HEK, but if people start making modified resource maps that replace the default ones then we'll quickly run into a problem of "who is the authority on the matter?" and people who make their maps to work with a shared resource map will conflict with others who use their own resource map.

Modders will also cause themselves headaches if they ever consider a resource outdated and remove it from their next version of a resource map, because this will ruin backwards compatibility with their previous maps.

The ONLY actual way for this to work would be for everyone to agree on what a new stock+ set of resource maps would contain and for a single programmer to make them and upload them. That would probably have to be me or someone else with the gumption to do it. Tools couldn't be released because of the chance of causing all the problems stated above. Any updates to the maps would have to preserve backwards compatibility, so all updates would be strictly additions.
Edited by MosesofEgypt on Jan 30, 2016 at 02:32 PM

Another bump.

I completely agree that it'd be a logistical issues, but I doubt the scope would be that big an issue. It would only really be worth doing for projects of large scope in terms of map numbers with shared assets. A single map wouldn't benefit nearly enough to be worth the trouble.

You can't even take care of my facility.

You can make new ones with OS. We use them for SPV3 since we have about a gigabyte of shared bitmap data, and our sound files continue to grow as we add the custom soundtrack.

Initially perhaps, but having 35MB of shared bitmaps per map in a 15 map collection just makes you download it 15 times. The idea is to eliminate the current need for redundancy.

Providing they have the time which nobody does.

Which OS already has available; you can compile 'mod sets', which are additional sets of bitmaps.map, sounds.map, loc.map. You can either run a compile that both generates and uses them for the 'mod set', or just references it. So, you start by compiling some garbage map that uses your whole tag set to produce a mod set, with all of your bitmaps and sounds, and then you compile whatever maps you want to distribute that reference that set.

Its part of the build-cache-file-ex, the starting string is the name, first boolean is whether to generate a new set, second boolean is whether to use a set, and third is the yelo, followed by typical scenario path.

Which IIRC requires using .yelo, which is completely useless for multiplayer.

How is it useless for multiplayer? Sure, some script commands dont work in multiplayer, but it still adds multiple functions. You're going to need a modification of the Halo application to include multiple resource maps regardless, because without one, it will only ever try to reference the default resource maps, and even if you replaced the default ones, unless everyone had exact tag indexes, and cooperated on creating some mega shared resource (which you would need to redownload everytime someone changed a resource of theirs, or added a new one), you'd then have to close halo, and replace resource maps between every map unless it was made by the same group.

In order to play yelo maps as I am very sure you know already you require OS which a majority of the guys who play online don't have installed and probably don't need to install due to the overall lack of yelo mp maps out there.

If OS was available via a convenient hac2 auto download, yelo map usage might bump up a nudge but I don't really see how its possible to make hac2 auto download & install OS if it has detected that you don't have it or the yelo map already installed & downloaded.

If someone can come up with a list of tags that are used enough to be considered common then when I have the time I can write a program to make expanded bitmap and sound maps. Any other common types of tags might be able to be put in the localization map, but I'm not sure.

I will only consider practical stuff: no 2048x2048 a8r8g8b8 bitmaps or guns with 8000 triangles. I know that its hard enough to find common stuff, let alone stuff that can't be the hugely unoptimized crap that's currently out there, but idk how halo works with the resource maps and if it has a limit on their size. Stuff like the wasp, bulldog, commonly used scenery and textures, etc.

I can't promise I'll do this, but if I see a nice list of stuff then I'll be more inclined.

Sorry, I should have said "worthless" rather than "useless", since obviously OS provides a ton of new features and abilities that are applicable to multiplayer, and that sehe has made SAPP compatible with the OS dedicated server.

I have no real issues with OS myself, and would like if it were more widely accepted.

The issue is that it doesn't provide the same useful features as hac2, the dominant multiplayer mod currently, and in fact goes so far as to be incompatible - which means your average person playing will choose hac2 in favor of OS.

It can be as useful as you want, if no one uses it or is willing to sacrifice features in order to do so, then it is worthless.

.yelo support only adds to the problem, because it isn't backwards compatible like an OS .map file.

OS .map files can be used by regular CE installs, but once you start implementing features only workable with .yelo files, you've forced users and developers to make a choice, and it's a choice that has already been made since OS's inception in the realm of CE multiplayer.

The whole point of the discussion was about how extending the resource files would be beneficial to end users and developers alike - and the answer to that is absolutely NOT OPEN SAUCE, at least in its current form.

Get lazy devs to make it compatible with Hac2 and its existing superiority in the features it provides, and maybe OS will stop being relegated to tinkering and single player projects that exist more as a curiosity than a reasonable alternative/addition.