Search Members Help

» Welcome Guest
[ Log In :: Register ]

 

[ Track This Topic :: Email This Topic :: Print this topic ]

 reply to topic new topic new poll
Topic: ISZ format revision 2, LZMA/dict. size 64kb-64mb/XTS, not CBC< Next Oldest | Next Newest >
 Post Number: 1
Iwo Search for posts by this member.

Avatar



Group: Members
Posts: 5
Joined: Apr. 2008
PostIcon Posted on: Apr. 15 2008,07:05  Skip to the next post in this topic. Ignore posts   QUOTE
Greetings,

"Chunk data may be compressed by ZLIB or BZIP2 method.". ZLIB has 32 or 64kb dictionary size, BZIP2 has 100,200,300,400,500,600,700,800,900kb dictionary sizes. Deflate is quite fast and well known, but it is a weak algorithm in general.
BZIP (bzw) is better, but it has one great disadvantage - decompression speed is equal to compression speed - SLOW.

So, how about releasing ISZ2 format with LZMA:
http://www.7-zip.org/sdk.html
and larger dictionary size. WINRAR and WINACE use LZMA with 4mb dictionary size and special multimedia algorithms. Also 7-ZIP uses special converters to compress EXE files better.
Dictionary size problems - larger size - better compression ratio, slower compression, more memory needed for compression, smaller size - a contrario.

Dictionary size - mem needed for compr. / mem needed decompr:
64kb - 38mb/3mb (megabytes)
2mb - 60mb/4mb
4mb - 83mb/6mb
8mb - 129mb/10mb
16mb - 221mb/18mb
32mb - 405mb/34mb
64mb - 709mb/66mb

"Memory (RAM) usage for LZMA compressing is about 11 times more than dictionary size. Memory usage for LZMA decompressing is close to value of dictionary size."

Also about CBC (for encryption) - it is old algorithm. Better is LRW and currently the best one is XTS:
http://www.truecrypt.org/docs/modes-of-operation.php

"XTS mode is in fact XEX mode [12], which was designed by Phillip Rogaway in 2003, with a minor modification (XEX mode uses a single key for two different purposes, whereas XTS mode uses two independent keys). XTS mode was approved as the IEEE 1619 standard for cryptographic protection of data on block-oriented storage devices in December 2007."

Also optional recovery blocks will be nice (like in WinRAR). Look there:
http://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction
And there: http://www.ice-graphics.com/ICEECC/IndexE.html

So, ISZ could be much better. Have fun for coders ;)

Peter

Edited by Iwo on Apr. 15 2008,07:06
Offline
Top of Page Profile Contact Info 
 Post Number: 2
xoben Search for posts by this member.

Avatar



Group: Super Administrators
Posts: 2200
Joined: Nov. 2004
PostIcon Posted on: Apr. 15 2008,09:45 Skip to the previous post in this topic. Skip to the next post in this topic. Ignore posts   QUOTE
Thank you Iwo for your nice suggestions.
Offline
Top of Page Profile Contact Info 
 Post Number: 3
Iwo Search for posts by this member.

Avatar



Group: Members
Posts: 5
Joined: Apr. 2008
PostIcon Posted on: Apr. 15 2008,13:40 Skip to the previous post in this topic. Skip to the next post in this topic. Ignore posts   QUOTE
I hope, that my suggestions will be useful. It is very seldom to see any thanks from authors or forum crew (I post many suggestions for many products). I also have something to add:

1. LZMA decompression speed is very good - ~36-38mb/sec on Core Duo 2GHz (8mb-2mb dictionary size).
2. About features - providing better compression spares more bits for error recovery, so these suggestions are connected in some way. So, EZB could add a nice option of "Target image size" like in ICE ECC target ECC file size. Also option to calculate needed % of recovery data for MORE >1 images to fit ALL these images on given media will be VERY nice. Currently I make ISZ files and then add ECC file and then burn files on DVD. So, Ultraiso could optimize and compress image(s), then add recovery data to fit on for example 700mb CD or single or double layer DVD. It will be an ideal solution for making backups - backup will be:
a. More useful than backup made by compressing ISO file with winrar.
b. More fault-tolerant than normal CD/DVD or normal image file.
c. More protected (in case of security) than backups made in other formats (see also below).

And also, please read my suggestions for different product - (some of) these suggestions will apply also for UltraISO:
http://forums.truecrypt.org/viewtopic.php?t=10588
Offline
Top of Page Profile Contact Info 
 Post Number: 4
Iwo Search for posts by this member.

Avatar



Group: Members
Posts: 5
Joined: Apr. 2008
PostIcon Posted on: Apr. 15 2008,14:03 Skip to the previous post in this topic. Skip to the next post in this topic. Ignore posts   QUOTE
Please be advised, that "word size" parameter gives best results for Deflate is it's at its maximum size (258), so you could optimize your current compression algorithm. BUT! For LZMA best results are achieved for WordSize/FastBytes = 160 or 192, not 273 (LZMAs maximum)!

Examples (PSI IM, 280 files, 34.3mb folder):
psi.zip for FB=128 (Deflate32) - 14167148 bytes
psi.zip for FB=258 (Deflate32) - 14165981 bytes
psi.7z for FB=160 (LZMA/8mb) - 10021377 bytes
psi.7z for FB=273 (LZMA/8mb) - 10026389 bytes
psi.7z for FB=160 (LZMA/64kb) - 11065683 bytes
psi.7z for FB=160 (LZMA/48mb) - 9873343 bytes

So LZMA easily outperforms Deflate! Also, I think that compression speed was comparable (engine of 7-zip is now very optimized).
Offline
Top of Page Profile Contact Info 
 Post Number: 5
xoben Search for posts by this member.

Avatar



Group: Super Administrators
Posts: 2200
Joined: Nov. 2004
PostIcon Posted on: Apr. 16 2008,00:50 Skip to the previous post in this topic. Skip to the next post in this topic. Ignore posts   QUOTE
Thank you Iwo. Please understand that not all suggestions will be realized in future versions.
Offline
Top of Page Profile Contact Info 
 Post Number: 6
Iwo Search for posts by this member.

Avatar



Group: Members
Posts: 5
Joined: Apr. 2008
PostIcon Posted on: Apr. 16 2008,09:48 Skip to the previous post in this topic. Skip to the next post in this topic. Ignore posts   QUOTE

(xoben @ Apr. 16 2008,05:50)
QUOTE
Thank you Iwo. Please understand that not all suggestions will be realized in future versions.

I understand ;) It is quite common, that authors have different vision of their program than users.

Not all suggestions are equally important:

XTS supports is theoretically better than CBC, but I don't know any attack on CBC.

Certainly the most important suggestion is about LZMA and dictionary size. I also think that BZIP support should be abandoned (It's SLOW and weaker than LZMA).
Offline
Top of Page Profile Contact Info 
 Post Number: 7
Iwo Search for posts by this member.

Avatar



Group: Members
Posts: 5
Joined: Apr. 2008
PostIcon Posted on: Jun. 13 2009,17:46 Skip to the previous post in this topic. Skip to the next post in this topic. Ignore posts   QUOTE
Since LZMA is more popular - I saw Linux and sourceforge packages in ".lzma" - do you consider extending ISZ by additional algorithm ? I tested many images and NTFS compression beats ISZ in some cases.

"LZMA SDK is placed in the public domain." So you can do anything with it.
Offline
Top of Page Profile Contact Info 
 Post Number: 8
dma_k Search for posts by this member.

Avatar



Group: Members
Posts: 8
Joined: Apr. 2010
PostIcon Posted on: Apr. 06 2010,18:19 Skip to the previous post in this topic.  Ignore posts   QUOTE
I vote for this feature as well.
Offline
Top of Page Profile Contact Info WEB 
7 replies since Apr. 15 2008,07:05 < Next Oldest | Next Newest >

[ Track This Topic :: Email This Topic :: Print this topic ]


  		reply to topic new topic new poll

» Quick Reply ISZ format revision 2
iB Code Buttons
You are posting as:

Do you wish to enable your signature for this post?
Do you wish to enable emoticons for this post?
Track this topic
View All Emoticons
View iB Code