Advanced chunk'ing

[Dominus]

I'm about to add this to the Exult Studio documentation along with initgame.dat hints and tips but who knows how long that will take.

So, for those that struggled with the chunks and messed them up etc...

The U7 world is made up of preformed chunks saved in the u7chunks file. The order of the chunk files used in the map, is saved in the u7map file. In Exult Studio when you change the order of the chunks, by adding or deleting a chunk, Exult Studio will automatically save the new order in the u7map file.
But what if you really messed up existing chunks by accident and you want to reset the chunks?
Unfortunately we do not have an option in Exult Studio for doing that. You will have to hex edit the u7chunks file.

NOTE: PLEASE MAKE BACKUPS OF YOUR u7chunks AND u7map BEFORE PROCEEDING WITH HEX EDITING

Our u7chunks file has two different formats.
The stock chunk format:
If you use #255).
In the second byte, the top hex digit is entirely for the frame; the bottom hex digit is half for frame (the top half) and half for shape (the bottom half).
You divide the second byte by 4 to get the frame number, or get the remainder of division by 4 for shape.
For example frame #11 of shape #1016 (0x03F8), which would make the first byte be 0xF8 and the second 0x2F (hex of 4 x 11 + 3).
(The original u7chunks format is similar, the second byte needs you to add 128 to the summ, though. So the above example would be first byte 0xF8, second byte 0xAF (hex of 4 x 11 + 3 + 128))


The extended chunk format:
If you use more than 1024 shapes u7chunks will have a ten bytes long header (0xff, 0xff, 0xff, 0xff, 'exlt', 0, 0 (FFFFFFFF65786C740000)) and each chunk will take up 768 bytes. Each shape using two bytes for the shape number and 1 byte for the frame number.
For example frame #11 (0x0B) of shape #1016 (0x03F8), which would make the first byte be 0xF8, the second 0x03 and the third 0x0B.


You can then save an unchanged map with Exult Studio and compare the resulting pristine u7chunks file with your edited one to find little problems.

To get a pristine map, make a backup of your u7chunks and u7map files and delete (or rename to something else (u7chunksnew u7mapnew) them from your mod or patch folder, start a game with with your mod or patch, enter map edit mode, in Exult Studio select U7CHUNKS, select the first chunk and move it once down and then up again. Then hit "Save map" (file -> save ... -> map).


NOTE: DO NOT DELETE A WHOLE CHUNK via hex editing from the u7chunk file - This would mess up your map very badly as the order of the chunks in u7map will then no longer match the actual order of the chunks.

[Knight Captain]

May I ask a serious question?

Would it make sense to replace these 25 year old formats with something that is modern and human-readable? Like JSON?

It's great that much of the formats from Origin are understood, but continuing to be wedded to them holds onto their problems as well. And as familiar as I am with a lot of Usecode, even I'm intimidated by map editing.

I'm quite impressed with what you've been doing with this, so may my question not take away from that.

[agentorangeguy]

I made the mistake once of adding and deleting chunks.... oooh it made a big mess. Now, if I need a custom chunk I just edit one of the existing ones that don't appear anywhere on the map, like the flashy ones or the ones they started to work on but never finished

[Knight Captain]

Could ES also create .orig or .stock versions prior to any changes?

[Dominus]

@AgentOrange: the safe way is to just duplicate a chunk that is looking close to what you need and edit the clone. But you have to have the map loaded in Exult, so Exult and Studio do their magic of ordering the chunks in u7map.
For keeping everything in order I *now* copy the 512/768 bytes of the chunk I want to clone and append these to the end of the u7chunks file with an hex editor. This is otoh best done when neither Exult or Studio is open.
For OS X I found a good hex editor that makes all 512/678 bytes blocks visible and numbers them so I can quickly find which chunk corresponds to which bytes block. "Synalyze it Pro".
Of course only after stumbling over badly wrangled chunks did I discover the joys of hex editing the u7chunks file
I logged a feature request with Marzo that it might be nice to be able to duplicate a chunk to the end of the chunkslist instead of adding it next to the duplicated one (and that Studio does the same ordering to chunk groups that it does for u7map).

@KnightCaptain: you only need the original once so doing it right away for a new project is best instead of waiting for this reqiest to be fullfilled at some point in the future.
Changing the way we store this information is probably not in the cards now anymore. Might have been good back then.

[Dominus]

Another thing I learnt from making sense of the above:
Currently the "small, stock u7chunks format allows only 64 frames (#0 -#63) of a shape to be used in a chunk. If you have a shape with more frames you cannot use them.
Unless you have more than 1024 shapes in shapes.vga, then the big u7chunks format is used which allows more than 64 frames. The limit is probably 255 frames (#0 - #254).
The limit of shapes in shapes.vga is 2048, meaning the highest number of a frame is #2047 (since numbering starts with 0).

[Knight Captain]

Would it still be useful to have them sorted in ES? I know it would be a manual task to list them all.

[Dominus]

You mean the chunks sorted in groups? It would be useful, yes, but it would need to be done with a pristine U7 map and the caveat is that the groups currently are not corrected when you addor remove chunks, so it can quickly go bad ;(

[Knight Captain]

New chunks are a lesser use case.

Is it possible to add new ones beyond the 3072 limit?