PlaneShift

  • Status Closed
  • Percent Complete
    100%
  • Task Type Feature Request
  • Category Engine → Tools → Updater/PSLaunch
  • Assigned To No-one
  • Operating System
  • Severity Medium
  • Priority
  • Reported Version
  • Due in Version Undecided
  • Due Date Undecided
  • Votes 3
  • Private
Attached to Project: PlaneShift
Opened by Seytra - 25.01.2010
Last edited by Talad - 03.10.2022

FS#3971 - Split materials.zip into zips per race / add override folder

Since I (and I expect several others as well) am using a custom skin for one character, I need to re-modify materials.zip after each update. Due to the use of binary diffs in the updater, it is not possible for it to simply exchange the changed files in the archive. So I need to copy back the old archive each time before updating.
With the newest update, however, this for some reason did not work: the update failed and now it’s downloading the whole 160 MB. Again. After already having downloaded an update file of 170MB for essentially nothing. This is the fifth time it did that, although the first time it happened despite of me having copied back the original file FAICT (the other times I had forgotten to make a backup or to copy it back, etc., pp.).

Also, the “backup” it does did not work: instead of copying the archive before messing with it, it created a file of zero length, so I can’t even interrupt the update and just use the old file, and instead will have to sit through another wasted hour of leisurely downloading a file that it has just deleted (shame on me for moving the old file back instead of copying). Also, again.

It would save a lot of download time (and bandwidth, at least if I’m not the only one with such issues) if the files that are likely to be customised were in separate archives, or at least in one file like the previous models.zip.
Of course, the ideal way to handle this would be to have an override folder or at least an override.zip which gets mounted over the vfs, like it was done in NWN, likely among others.

I’m going to make a script to auto-copy them, but still maintain the FR to have an override folder.

Closed by  Talad
03.10.2022 21:10
Reason for closing:  Will not fix
Franta commented on 26.01.2010 14:04

I think many thing can be specified directly in skin definition file, though I am not sure whether it's possible for this. Anyway, I vote for splitting the files to say maximally 30MB parts so it will speed up the updates and lower bandwidth. That is for both meshes.zip and materials.zip

Seytra commented on 26.01.2010 14:43

Indeed the solution of bdiff is a nice and elegant one, but it comes at a price: it's causing the update file to almost alwayws include, in effect, the entire materials.zip. The reason is that the zip structure changes the entire file whenever even the tiniest of changes has been made to the archive.

That means that even if you only change a file of 10 KB to an archive, the binary diff is unable to find a set of small changes and instead is forced to exchange the entire file. This is a major inefficiency in using zip (or, AFAICT, most somewhat advanced compressors) together with bdiff in this role. Since the material files are pretty likely to change in any given update, this leads to every single player having to download essentially the whole materials.zip with each update, likely draining significant bandwidth from the server as-is (160MB*100 players*10 updates/month=160GB/month), plus requiring the server to store all this theoretically redundant information because the updater must be able to bootstrap from any version to the current one (which is a really nice feature IMO).

I assume that because of the compression, there's no space-efficient archive format that keeps most of the file constant, so it might be necessary to use eg. tar alongside zip and bdiff. The resulting process would be:
- unzip the file, yielding a tar file
- apply binary diff to tar file
- zip the tar file

That way, the updater would still be (reasonably) independant on the compressor used (in that it would just be a program called and not intrinsic to the updater as it used to be), while reducing the required download size significantly.
The drawbacks of course would be
- extra code to handle the generic decompression
- the associated case-logic to support uncompressed files, and / or only do this for files specified in an extra "format" file
- the inability for players to use the simple (and ATM ubiquitous) zip format only
- the increased memory requirement for the temporary uncompressed file

The latter might be mostly compensated by decompressing to disk, since ATM the downloaded file also takes up lots of space on the drive.

Thus, even without caring about user-modifications (for which the aforementioned override folder would be the better anyway), the update process would benefit from smaller files.

CXDamian commented on 16.02.2010 05:19

Setting New

Neeno Pelero commented on 01.08.2011 08:50

This patch loads all zip-files from the path "/planeshift/art/mods/", so it's not necessary to modify the "materials.zip" or other files from PS.
These files are loaded in alphabetic order. If there are e.g. double dds-files, the dds from the first zip-packet gets loaded.
The zip-packets must contain a "mod.xml" with the following content:
<xml>

  <mod_information>
      <name>NAME</name>
      <author>AUTHOR</author>
      <description>DESCRIPTION</description>
  </mod_information>
  <mount_path>MOUNT PATH</mount_path>

</xml>

Patch: http://pastebin.com/w0sVANKM

- There are no more corrupted files → saves bandwidth
- it's easy to handle with the modular zip-file strukture - > possible to deactivate mods fast

bonifarz commented on 08.08.2011 21:14

I think above patch deserves a separate task to clarify votes for each feature request.
Even though closely related, splitting the materials.zip is not exactly the same as providing better support for other mods than art/skins/

weltall commented on 14.08.2011 06:18

i'm not really planning to implement a feature to mod graphics. it goes under mesh mods to find items fast in a place

Franta commented on 15.08.2011 13:11

I am adding some missing icons into these huge files, like delta blades and so, but on every update it's "repaired" and I need to do it again. It could be good to be able to insert separate mod that would fix this until it will be fixed in the main zip file. Even I don't know why it's not already fixed while these icons was included just few versions ago.
About repair - I suggest to make pslaunch to download zip files into *.zip.download file first (i.e. meshes.zip.download) and after the file is fully downloaded, rename it into *.zip (meshes.zip in this example), it could save time and bandwidth by stopping the update and using the old file in these cases, when it's not needed.
When it's possible, where should I put the missing icons without adding them into the huge zip file?

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing