[Top] [All Lists]

Re: compression

To: "Josef Sipek" <jsipek@xxxxxxxxxxxxxxxxx>
Subject: Re: compression
From: "Bhagi rathi" <jahnu77@xxxxxxxxx>
Date: Fri, 14 Sep 2007 01:09:56 +0530
Cc: "Jordan Mendler" <jmendler@xxxxxxxx>, xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=4dnP2T3j83qxPe9YlPNuqGNftDDgGIcVKtQ0021a9io=; b=mVG5VXQ4ZeuYs5RjnfaLOdk+OCzoMxy3XbfRUlHfoz/gvWB00dpPzJT4QnqSFH3DfVLrBBTHl3FN/XguHOjZ0MjMeGL9vLQhHh/hcfK+IyAfWBncYjzqXIMN88opghanjXfNs4kZJIas4/9hNVyxh/I2Fqtxrz55SkeppMnVlGY=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=Y4Ve9c5ClkBrVDQFDH9O0gE5UTzokActyGpcmpEfuibAOQ7H8UCUHB8ph9DqFRxWxU/ue5blaxbHQ0QANUMGG0na5hK6/HCByMXBkevW2aplGlCHPTRMSibvfaubhLVJKF1BBxnOjmv0iW2zrp0IRkSt4LbsO7kMGtN0CH7W8EA=
In-reply-to: <20070912174216.GA5521@xxxxxxxxxxxxxxxxxxxxxxx>
References: <654e62180709111643k4700c2bdibec2a16eb5446e76@xxxxxxxxxxxxxx> <20070912174216.GA5521@xxxxxxxxxxxxxxxxxxxxxxx>
Sender: xfs-bounce@xxxxxxxxxxx
Using open source rsync seems to be smart enough to identify files which
been modified and use hard-link instead of copying the file. I am not sure
that rsync
used here is smart enough to identify the same file which haven't been

If hard-linking is already done, I believe that there is lot of duplication
of data in the same
file. It looks that open source rsync doesn't eliminate duplication of data
was already existing in older backup. It copies it again. Compressing the
for the same file across various backup snapshots can be very powerful and
my guess is that it can definitely free more that 30% of your space. Note
this is not file-system wide compression, it is compression of the same file
existing in various back-ups. Restore gets affected, but should be easy to
given that it can free lot of space.


On 9/12/07, Josef Sipek <jsipek@xxxxxxxxxxxxxxxxx> wrote:
> On Tue, Sep 11, 2007 at 04:43:20PM -0700, Jordan Mendler wrote:
> > Hi all,
> >
> > I searched the mailing list archive and could not find an answer. We are
> > currently using XFS on Linux for a 17TB Volume used for backups. We are
> > running out of space, so rather than order another array, I would like
> to
> > try to implement filesystem-level compression. Does XFS support any type
> of
> > compression? If not, are there any other ways to optimize for more space
> > storage? We are doing extensive rsyncs as our method of backups, so
> gzipping
> > on top of the filesystem is not really an option.
> Implementation-wise, one major thing to keep in mind is that offsets into
> the uncompressed copies of files in memory need to be mapped to the
> compressed ones. This is rather painful if you want to do things right
> (supporting writing as well as reading from files).
> As Eric mentioned, you may want to try to eliminate copies of identical
> files with symlinks or even hardlinks (just make sure your backup sw is
> smart enough to break links when necessary).
> Josef 'Jeff' Sipek.
> --
> The reasonable man adapts himself to the world; the unreasonable one
> persists in trying to adapt the world to himself. Therefore all progress
> depends on the unreasonable man.
>                 - George Bernard Shaw

[[HTML alternate version deleted]]

<Prev in Thread] Current Thread [Next in Thread>