Bogdan Costescu wrote:
On Wed, 12 Feb 2003, Chris Wedgwood wrote:
Scientific simulations that produce large amounts of data over a long
time. Of course, "large amounts" and "long time" is relative, in our case
being "several hundreds megabytes" and "days". The file is created at the
beginning of the simulation and data is appended to it for the whole
simulation duration; data is never rewritten during the simulation.
Depending on data size, simulations can finish sooner or later and
files that are of different sizes, however in order to ease later backup,
we limit the size of one file to approximately 650 MB. Writting to
not a problem, as the rate at which data is produced is very low.
reading the data becomes a problem and we need to do it to either analize
the data or transfer it from there to other computer or CD.
I am curious if the data is binary, or if it is in such a form that SQL
storage could be used? That would solve a lot of problems (or perhaps
alter them) for replication, and especially for fragmentation choices
and questions. Postgresql might be a good choice, network enabling it
and having nodes write to postgresql either via sockets or a daemon
running on your master node. If you need to sort or view non-binary data
in more than one way, SQL would be a blessing.
I am also curious what kind of Postgresql or MySQL performance
differences people have found under XFS, compared to any other file system?
D. Stimits, stimits AT attbi DOT com