| To: | Ken McDonell <kenj@xxxxxxxxxxxxxxxx> |
|---|---|
| Subject: | Re: Floating point problem |
| From: | Martin Spier <mspier@xxxxxxxxxxx> |
| Date: | Wed, 30 Jul 2014 11:35:37 -0700 |
| Cc: | "Frank Ch. Eigler" <fche@xxxxxxxxxx>, pcp@xxxxxxxxxxx, Amer Ather <aather@xxxxxxxxxxx>, Coburn Watson <cwatson@xxxxxxxxxxx>, Brendan Gregg <bgregg@xxxxxxxxxxx> |
| Delivered-to: | pcp@xxxxxxxxxxx |
| Dkim-signature: | v=1; a=rsa-sha256; c=relaxed/relaxed; d=netflix.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=SwdIxP3DC4uTgiQZFbpKZJDqPteyZCgYcOkKIt+7i5k=; b=GhLhQnGzKczzXSPb+y/EBuHHercwfUdlHayLUNqMD3Kc0YUThb+rbCYm7pLbgd1gPM bUdV6+vbFzAIYvX5ehqNDAPaLiV3lwaYBU9O42RjtQD1u7sqtBcUiwfEZh0BehEBn0qS wkh4dypaMbvo4TWu3mjZAFwZAebagKavU0YQk= |
| In-reply-to: | <53D73CBF.9090008@xxxxxxxxxxxxxxxx> |
| References: | <CAEp4+dU2kE9JJztBPc=N5oSyoEyBvN5Of19rohC3DxXGeomuRw@xxxxxxxxxxxxxx> <033501cfa8a4$fd091ed0$f71b5c70$@internode.on.net> <CAEp4+dUH6fEQ2E=o5O2q8LKfR2xUypM-AeOwQhWy9sEntvO-AQ@xxxxxxxxxxxxxx> <53D6CE6A.8030309@xxxxxxxxxxxxxxxx> <y0mbns92ek8.fsf@xxxxxxxx> <53D73CBF.9090008@xxxxxxxxxxxxxxxx> |
|
I'm not familiar with PCP's code so I might be completely wide of the mark here, but my understanding is that it's a precision problem. Aren't int64 or GMP (https://gmplib.org/) options in this case? There's also the JSON conversion problem, but one option in this case is sending out a String in the JSON response instead of a number. It's not clean and nice, but it might work. On the client side (_javascript_) I can convert that to a BigDecimal and work with it. On Mon, Jul 28, 2014 at 11:18 PM, Ken McDonell <kenj@xxxxxxxxxxxxxxxx> wrote:
|
| <Prev in Thread] | Current Thread | [Next in Thread> |
|---|---|---|
| ||
| Previous by Date: | Re: [pcp] minimal install not-as-root, Nathan Scott |
|---|---|
| Next by Date: | Re: Recent QA regressions, Dave Brolley |
| Previous by Thread: | Re: Floating point problem, Ken McDonell |
| Next by Thread: | Re: Floating point problem, Frank Ch. Eigler |
| Indexes: | [Date] [Thread] [Top] [All Lists] |