xfs
[Top] [All Lists]

Re: [PATCH 07a/13] xfsprogs: add trie generator for UTF-8.

To: Roger Willcocks <roger@xxxxxxxxxxxxxxxx>
Subject: Re: [PATCH 07a/13] xfsprogs: add trie generator for UTF-8.
From: Ben Myers <bpm@xxxxxxx>
Date: Wed, 24 Sep 2014 18:11:45 -0500
Cc: linux-fsdevel@xxxxxxxxxxxxxxx, tinguely@xxxxxxx, olaf@xxxxxxx, xfs@xxxxxxxxxxx
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <1411497259.4842.163.camel@xxxxxxxxxxxxxxxxxxxxx>
References: <20140918195650.GI19952@xxxxxxx> <20140918203114.GN4482@xxxxxxx> <20140919160612.GF4482@xxxxxxx> <1411497259.4842.163.camel@xxxxxxxxxxxxxxxxxxxxx>
User-agent: Mutt/1.5.20 (2009-06-14)
Hi Roger,

On Tue, Sep 23, 2014 at 07:34:19PM +0100, Roger Willcocks wrote:
> On Fri, 2014-09-19 at 11:06 -0500, Ben Myers wrote:
> > +#define AGE_NAME       "DerivedAge.txt"
> > +#define CCC_NAME       "DerivedCombiningClass.txt"
> > +#define PROP_NAME      "DerivedCoreProperties.txt"
> > +#define DATA_NAME      "UnicodeData.txt"
> > +#define FOLD_NAME      "CaseFolding.txt"
> > +#define NORM_NAME      "NormalizationCorrections.txt"
> > +#define TEST_NAME      "NormalizationTest.txt"
> 
> Is there a reason why you're using multiple text-based data files (and
> hand-parsing them) when there's an xml formatted flat file available ?
> 
> http://www.unicode.org/Public/UCD/latest/ucdxml/

The UCD files being parsed are the authoritative source.  Check out
ucdxml.readme.txt.

> And a 2nd question - why does the trie need to encode "the the unicode
> version in which the codepoint was assigned an interpretation" ?

You need to know whether a given code point is assigned in the version
of Unicode you're normalizing for.  Unicode 8 is supposed to release
June/July 2015 (see http://www.unicode.org/versions/), but filesystems
you created this year will still need the version 7 normalization.
There is still some plumbing to do to pass the version along with the
string for normalization.

I think you bring up a good point, but we'll need to support multiple
versions in the long run.

Regards,
Ben

<Prev in Thread] Current Thread [Next in Thread>