Small math question: lowest useful precision...

[Herman]

I have a small problem I can't quite figure out, I'm sure it's quite easy though. I need to find the lowest useful precision of a decimal number. For example, let's say I have a Decimal variable that contains 0.003. It's lowest useful precision is 0.001. If it's 2, the lowest useful precision is 1, 234.2456 = 0.0001, etc.. Is there a straightforward way to get this result? I tried various calculations, but I cannot seem to get it right.

Thanks.

 

[jmcilhinney]

What's the lowest useful precision of 20; 10 or 1?

 

[Herman]

It would be 1. (I know what you mean, it's not really an exact rule... Integers should always return 1.")

 

[JohnH]

This is an example how I would solve it using math and Linq as tools:

Dim values = {1, 1.1, 1.11, 1.111, 1.1111}
Dim precision = Math.Max(0, Aggregate v In values Into Max((v - Math.Truncate(v)).ToString.Length - 2))
Dim display = (1 / (10 ^ precision)).ToString

precision = 4 in this example, ie 4 decimal digits, display will be in form 0.0001 with number of digits according to precision.
Lowest number of digits will be 0 (according to your specification) and ruled by the Math.Max(0, ...) expression, display for whole numbers will therefore always show "1".

 

[Herman]

Superb, this is exactly what I was looking for!

Thanks John, great explanation!

 

[Pyth007]

I'd actually be careful with this solution... For example, if a person wants to show the lowest precision in a number and the value of that lowest precision part of the number happens to be 0, they will write that decimal with a zero in the last position to show where the lowest precision occurs (eg 1.1110 to show that the 4th decimal place (or 0.0001) is the lowest useful precision place). However VB.Net will truncate off that last 0 when converted to a number, so the following code:

        Dim values = {"1", "1.1", "1.11", "1.111", "1.1110"} ' Notice I have to enter these numbers in as strings in order to keep the zero in the final position
        Dim precision = Math.Max(0, Aggregate v In values Into Max((CDbl(v) - Math.Truncate(CDbl(v))).ToString.Length - 2)) ' And then convert the strings to a double in order to do the math
        Dim display = (1 / (10 ^ precision)).ToString
        MsgBox("Precision: " & precision & " | Decimal-Display: " & display)

will show incorrectly that the 3rd decimal position (0.001) is the lowest useful precision.

 

[JohnH]

As a number that last 0 digit is not significant. I'd rather let user override the precision with input to a NumericUpDown control, if user would prefer a different precision to what is actually needed from the actual data series.

 

[Herman]

In this case however Pyth was right. The precision is used to determine the increment on a trackbar control, so that last 0 is significant. In addition there was an issue with number larger than 9 (because of the magic "- 2") and with integers. This is what I ended up using:

Dim fractionalPart As String = (value - Math.Truncate(value)).ToString
Dim fractionalLength As Integer = If(fractionalPart.Contains("."), fractionalPart.Split(".").Last.Length, 0)

_Increment = 1 / (10 ^ fractionalLength)

 

[Pyth007]

One last thing... If this is for you (or anyone in the US), that'll work fine. However other countries (esp. in Europe) use a comma to separate whole numbers from their fractional part.

 

[JohnH]

In addition there was an issue with number larger than 9 (because of the magic "- 2")

I don't see how? Let me explain; the math removes any integer part, ending up with either a 0 value or a 0,123 fractional value. Then it takes length of number as string and subtracts 2, for fractional numbers the result is the number of decimal digits, for whole numbers the result is negative and Max(0, result) forces return 0 instead of negative when there are only whole numbers in data.

However other countries (esp. in Europe) use a comma

That's why I came up with that code, since it has no culture dependencies. There are of course culture information available in .Net, but that would increase the amount of coding needed to achieve same result.

 

[Herman]

Well I won't lie I don't remember exactly what the issue was, and your code does look fine John, but there definitely was one. The original code, although definitely pointed me in the right direction, did not work as intended originally when I stuck it in as is with some boundary cases.

Anyways this is long fixed... As far as the culture, this is designed to setup user controls programatically according to an XML template, to allow for expansion. I doubt anyone will ever edit the template directly except me, and if they do they will see it uses a period on all the other lines.