Do you use PAR files when downloading binaries from Usenet?

Hack your OS, customize your GUI, announce great software, and configure, configure, configure...

Postby BKoT » 01/01/02, 8:23 am

Then FSRaid is for you.

This applications handles all of your existing PAR file needs plus, it is poised to take the PAR file concept to the next level.

Check it out here:
<A HREF="http://www.fluidstudios.com/fsraid.html">Fluid Studios</A>

--
-BKoT-
(Big Kahuna of Tuna)
BKoT
Member ***
 
Posts: 19
Joined: 01/01/02, 12:00 am
Location: Detroit

Postby bob » 01/01/02, 12:06 pm

I'm not sure I understand the advantage over split files.....
bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
 
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis

Postby BKoT » 01/01/02, 2:21 pm

The usage of PAR files isn't an advantage _over_ split files. They're meant to work along with them.

Have you ever downloaded a set of rar files from Usenet, only to find that you're missing three of them because they weren't "complete" on your news server? If a set of PAR files was also posted, you wouldn't have to go begging for reposts of those missing files.

PAR files are nothing but sets of checksums, created using the same method used when hard drives are set up in a RAID. Based on the information stored in the PAR files, you could recreate those missing files, or even repair a file if it were corrupted in some way.

This whole concept for PAR files was meant to cut down on the number of repost requests in the newsgroups, thus saving on hard drive space as binary posts get larger and larger all the time.

But it has other uses as well. I myself use it as an extra safeguard for my backups. I use rar to create the archive sets and then I par those files, saving the pars in another location. If something were to ever happen to portions of my backup archives, I could always recreate the parts that had been lost.



_________________

-BKoT-
(Big Kahuna of Tuna)

<font class=editedby>[ This Message was edited by: BKoT on 2002-01-01 22:22 ]</font>
BKoT
Member ***
 
Posts: 19
Joined: 01/01/02, 12:00 am
Location: Detroit

Postby bob » 01/01/02, 2:53 pm

But what's the advantage then over creating a second backup? Don't you have to have the complete set of PAR files too? Are the PAR files smaller than the rar's? I don't get it. It looks like duplicated effort.... It's one thing to post checksums... but quite another to redundantly post pars... I'm not saying you're wrong. I just don't get it. Tell me more...
bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
 
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis

Postby BKoT » 01/01/02, 5:08 pm

For my use, it really isn't a second backup per se, as none of the actual data is stored. Only checksum information. Yes it takes a little extra storage space, but I just use it as piece of mind, since I've been burned before. "Agh.. My drive crashed, no matter, just grab the backups. What?? The network drive where I store my backups crashed over the weekend??"

As for size, no. However many par files you have, each one will always be as large as the largest file you're making a parity set for. But, 99% of the time, the number of par files created doesn't come anywhere near the total number of files you're storing that checksum data for.

(what?)
I'll explain.
I've got a folder with a rar file that's been split into 22 separate files, and each file is 1 meg in length.
When I create the parity set (checksum data), the application creates 3 new files, each 1 meg in length. This is based on a checksum to data ratio of 10%. The higher the percentage ratio, the more checksum information is stored.

So now, I've got my 22 archive files and three parity files. If for any reason I lose up to three of my files. (Say, .r05, r.06 and .r20), I can use the parity set to recreate those files and I'm back in business.

If the percentage ratio during the par creation is set to a higher value, then the number of possible files you can recreate if any are missing or are corrupted is greater, but payment for that comes in more files created in that parity set.


Some info that describes the technique... This is taken directly from the help file in the application.

"..let’s consider four files, each of which only contain a single byte. These bytes are: 1, 88, 32, 99. We’ll calculate a parity byte as (1 + 88 + 32 + 99 = 220). With this information, we can restore any of the original bytes. All we have to do is subtract the remaining bytes (the ones we still have) from the parity value. The result will be the original byte. If we lose the “88” we can calculate it by: 220 – 99 – 32 – 1 = 88. It doesn’t matter which byte we lose, we can always restore it. It’s a mathematical certainty!"


Again, the whole bottom line for par files is to help in possibly curbing some of the endless repost requests for missing parts of binary files posted in the newsgroups. More than once this technique has saved me from having to rely on someone to repost files when some were missing or corrupted on my news server.
-BKoT-
(Big Kahuna of Tuna)
BKoT
Member ***
 
Posts: 19
Joined: 01/01/02, 12:00 am
Location: Detroit

Postby bob » 01/02/02, 2:58 am

Hmmmm... That's interesting. So ok... I create a rar file, split it into 20 pieces. And I set my par files at 10%. So as long as I lose less than 10% of any information in the rar's, the par files can restore what I've lost?
bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
 
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis

Postby Gerry » 01/02/02, 6:02 am

What stops the par files from going missing just as easily as the split binary files?
I answer rhetorical questions for my own enjoyment.
Gerry
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
 
Posts: 5727
Joined: 12/04/01, 12:00 am
Location: Perth, Western Australia

Postby BKoT » 01/02/02, 6:03 am

Yep. Now you've got it. <IMG SRC="/phpBB/images/smiles/icon_smile.gif">

The trick is the ratio... Finding a good balance between the number of par files that are created against the number of losses you could have.

Using the example of 22 rar files and 3 parity files, if FOUR files in the rar set were missing or corrupted, then those three par files won't be able to help at all. You wouldn't even be able to recreate three of the four files because there just isn't enough data in the existing rar files to do the math.
-BKoT-
(Big Kahuna of Tuna)
BKoT
Member ***
 
Posts: 19
Joined: 01/01/02, 12:00 am
Location: Detroit

Postby bob » 01/02/02, 6:19 am

Let's say the total rar weight is 10 megs. How large would the par files be to give you a 10% protection?
bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
 
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis

Postby BKoT » 01/02/02, 6:34 am

<!-- BBCode Quote Start --><TABLE BORDER=0 ALIGN=CENTER WIDTH=85%><TR><TD><font class=postbody>Quote:</font><HR width=100% color=#333333 SIZE=1></TD></TR><TR><TD><FONT class=quote><BLOCKQUOTE>
On 2002-01-02 14:02, Gerry wrote:
What stops the par files from going missing just as easily as the split binary files?

</BLOCKQUOTE></FONT></TD></TR><TR><TD><HR width=100% color=#333333 SIZE=1></TD></TR></TABLE><!-- BBCode Quote End -->

Unfortunately, therein lies one of the problems.
I've seen plenty of instances where a complete set of par files wasn't available, for the same reason that some rar files didn't make it through.

BUT, based on how par files work, this usually isn't a problem at all.
Go back to my example of 22 rar files and 3 parity files. Knowing that I only have 3 par files, I know that I'm safe if I've lost no more than 3 rar files.
So... I'm scanning the list and see that I'm not going to be able to successfully download 1 part of the rar set. And, at the same time, I see that the two of the three par files are missing as well.

Not a problem at all.
If you've got every other file of the rar set, except for one, only one file in the par set needs to exist for you to recreate that missing rar file.

It takes awhile to really wrap your brain around the concept of just how this whole process works. When I first saw it work, I chalked up up to pure magic, but after using it for while (and being a beta tester for FSRaid, the newest par app on the block, helped a lot too), it all just falls into place.
-BKoT-
(Big Kahuna of Tuna)
BKoT
Member ***
 
Posts: 19
Joined: 01/01/02, 12:00 am
Location: Detroit

Postby BKoT » 01/02/02, 6:52 am

<!-- BBCode Quote Start --><TABLE BORDER=0 ALIGN=CENTER WIDTH=85%><TR><TD><font class=postbody>Quote:</font><HR width=100% color=#333333 SIZE=1></TD></TR><TR><TD><FONT class=quote><BLOCKQUOTE>
On 2002-01-02 14:19, bob wrote:
Let's say the total rar weight is 10 megs. How large would the par files be to give you a 10% protection?
</BLOCKQUOTE></FONT></TD></TR><TR><TD><HR width=100% color=#333333 SIZE=1></TD></TR></TABLE><!-- BBCode Quote End -->

Depends.
Again, the par files, no matter how many there are, will always be as large as the largest file added to the par set.

I just ran a test. I have a file that when compressed with rar ends up being just a little over 10 meg. When I rarred this file, I had Winrar split it into 10, 1 meg chunks. Creating a par set in this scenario using a 10% ratio will create only 1 par, 1 meg in size. So now I could afford to lose only 1 rar file. Any more than that and I'm stuck.

I then did the same thing again, only this time, I had Winrar split things into 500k chunks. Now, that 10% ratio will end up creating 3 par files, each one 500k in size. I just added a meg and a half to the total storage for this file, but if up to 3 of my rar files go missing or become corrupted, I can easily restore them.
-BKoT-
(Big Kahuna of Tuna)
BKoT
Member ***
 
Posts: 19
Joined: 01/01/02, 12:00 am
Location: Detroit

Postby bob » 01/02/02, 8:26 am

Gee... sounds holographic. How's it work anyway....
bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
 
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis

Postby Guest » 01/02/02, 9:15 am

<!-- BBCode Quote Start --><TABLE BORDER=0 ALIGN=CENTER WIDTH=85%><TR><TD><font class=postbody>Quote:</font><HR width=100% color=#333333 SIZE=1></TD></TR><TR><TD><FONT class=quote><BLOCKQUOTE>
On 2002-01-02 16:26, bob wrote:
Gee... sounds holographic. How's it work anyway....
</BLOCKQUOTE></FONT></TD></TR><TR><TD><HR width=100% color=#333333 SIZE=1></TD></TR></TABLE><!-- BBCode Quote End -->

<IMG SRC="/phpBB/images/smiles/icon_smile.gif"> I know. It all sounds like smoke and mirrors, nothing-up-the-sleeve magic, but it works.

"How" it works is all algebra. If you're aren't reduced to a pile of mush by looking at scary algebraic equations (like I am), go here for some real meat on the "how".

http://drake.ee.washington.edu/~adina/r ... slide.html


This whole process is used all the time when hard drives are configured into a RAID. But, these new par tools bring the process into software.
Guest
 

Postby BKoT » 01/02/02, 9:18 am

(Oops... Forgot to log on when making that last reply....)
-BKoT-
(Big Kahuna of Tuna)
BKoT
Member ***
 
Posts: 19
Joined: 01/01/02, 12:00 am
Location: Detroit

Postby bob » 01/02/02, 9:44 am

well,you've convinced me--I've added the app to the list for a near-future Daily Update (over the next couple weeks....) Only thing is--I don't know if I can explain it. This thread'll help a lot though.
bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
 
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis

Next

Return to Own The Machine

Who is online

Users browsing this forum: No registered users and 3 guests

cron