TFDV uses weird float value for sample_count of generated histograms

[liwii]

When I generate statistics from a .tfrecord file with generate_statistics_from_tfrecord, its histograms contain weird float values as the sample_counts of the buckets.
For example, in one bucket which is supposed to contain 10 samples, sample_count: 9.94000000834465 is used instead. How can I set the exact integer sample_count for each bucket?

Here's a Colab to reproduce.

[kennysong]

Is there any update (or explanation) for this behavior?

[paulgc]

TFDV currently uses an approximate method to determine the bucket boundaries in a single pass. The float values are due to this. One option would be to do some post-processing to round the values.

[kennysong]

Got it, thanks for the explanation. Are there any error bounds on the approximate counts? (i.e. it's within +-1 of the true count)

[bharatjetti]

Hi @kennysong ,

Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.

We suggest you try the latest TensorFlow Data validation version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.

Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow Data-validation space.