Re: xfs metadata overhead

[Danny Shavit <danny@xxxxxxxxxxxxxxxxx>]

I see.

So using bulk_stat ioctl in similar way to xfs_fsr and summing allocated size will result with Âmore accurate number?

Thanks,

Danny

On Fri, Jun 24, 2016 at 4:17 PM, Eric Sandeen <sandeen@xxxxxxxxxxx> wrote:

On 6/24/16 5:34 AM, Danny Shavit wrote:
>> How do you determine allocated_size, with du?
> yes via du
>> How different? Can you show an example?
> meta data file size= 5.6 GB (*6,089,374,208*)
>
> *df:*
> Filesystem         Â1K-blocks   ÂUsed Available Use% Mounted on
> /dev/dm-39Â Â Â Â Â Â Â Â Â 2725683200Â 955900860 1769782340Â 36% /export/v1Â Â Â= 978,842,480,640 bytes
>
> *du:*Â du -s /export/v1/
> 952825644Â Â Â Â/export/v1/Â Â Â= 975,693,459,456 bytes
> 978,842,480,640 -975,693,459,456Â =
>
> Metadata size according to calculation=*3,149,021,184* bytes

Oh, right. I should have thought of this; "du" counts some metadata as
well, i.e. a directory full of zero-length files still consumes space
which is reported by du.

so your 975,693,459,456 bytes is file data as well as some metadata.

-Eric

>
> Thanks,
> Danny
>
> On Thu, Jun 23, 2016 at 9:12 PM, Eric Sandeen <sandeen@xxxxxxxxxxx <mailto:sandeen@xxxxxxxxxxx>> wrote:
>
>Â Â ÂOn 6/23/16 10:04 AM, Danny Shavit wrote:
>Â Â Â> I see. We will try this direction.
>Â Â Â> BTW: I thought that good estimate would be "volume_size -
>Â Â Â> allocated_size - free_space". But it produced quite a difference
>Â Â Â> compared to metadata dump size.
>Â Â Â> Is there a specific reason?
>
>Â Â ÂHow do you determine allocated_size, with du?
>
>  ÂHow different? Can you show an example?
>
>Â Â Â-Eric
>
>Â Â Â> Thanks,
>Â Â Â> Danny
>Â Â Â>
>Â Â Â> On Thu, Jun 23, 2016 at 1:51 AM, Dave Chinner <david@xxxxxxxxxxxxx <mailto:david@xxxxxxxxxxxxx> <mailto:david@xxxxxxxxxxxxx <mailto:david@xxxxxxxxxxxxx>>> wrote:
>Â Â Â>
>Â Â Â>Â Â ÂOn Wed, Jun 22, 2016 at 06:58:16PM +0300, Danny Shavit wrote:
>Â Â Â>Â Â Â> Hi,
>Â Â Â>Â Â Â>
>Â Â Â>Â Â Â> We are looking for a method to estimate the size of metadata overhead for a
>Â Â Â>Â Â Â> given file system.
>Â Â Â>Â Â Â> We would like to use this value as indicator for the amount of cache memory
>Â Â Â>Â Â Â> a system for faster operation.
>Â Â Â>Â Â Â> Are there any counters that are maintained in the on-disk data
>Â Â Â>Â Â Â> structures like free space for examples?
>Â Â Â>
>Â Â Â>Â Â ÂNo.
>Â Â Â>
>Â Â Â>Â Â ÂRight now, you'll need to take a metadump of the filesystem to
>Â Â Â>Â Â Âmeasure it. The size of the dump file will be a close indication of
>Â Â Â>Â Â Âthe amount of metadata in the filesystem as it only contains
>Â Â Â>Â Â Âthe filesystem metadata.
>Â Â Â>
>Â Â Â>Â Â ÂIn future, querying the rmap will enable us to calculate it on the
>Â Â Â>Â Â Âfly, (i.e. not requiring the filesystem to be snapshotted/taken off
>Â Â Â>Â Â Âline to do a metadump).
>Â Â Â>
>Â Â Â>Â Â ÂCheers,
>Â Â Â>
>Â Â Â>Â Â ÂDave.
>Â Â Â>Â Â Â--
>Â Â Â>Â Â ÂDave Chinner
>Â Â Â>Â Â Âdavid@xxxxxxxxxxxxx <mailto:david@xxxxxxxxxxxxx> <mailto:david@xxxxxxxxxxxxx <mailto:david@xxxxxxxxxxxxx>>
>Â Â Â>
>Â Â Â>
>Â Â Â>
>Â Â Â>
>Â Â Â> --
>Â Â Â> Regards,
>Â Â Â> Danny
>Â Â Â>
>Â Â Â>
>Â Â Â> _______________________________________________
>Â Â Â> xfs mailing list
>Â Â Â> xfs@xxxxxxxxxxx <mailto:xfs@xxxxxxxxxxx>
>Â Â Â> http://oss.sgi.com/mailman/listinfo/xfs
>Â Â Â>
>
>Â Â Â_______________________________________________
>Â Â Âxfs mailing list
>Â Â Âxfs@xxxxxxxxxxx <mailto:xfs@xxxxxxxxxxx>

>Â Â Âhttp://oss.sgi.com/mailman/listinfo/xfs
>
>
>
>
> --
> Regards,
> Danny

--

Regards,

Danny