two case studies of NaN

IEEE-759 NaN is weird. and because of that, it's often accidentally left unaccounted for. i found two instances of this leaking into programming language design. that is, the semantics of these languages hold implicit assumptions which break with NaN.

case study 1: python

>>> from math import nan
>>> nan == nan
False
>>> [nan] == [nan]
True

as an optimization, when comparing lists for equality, elements are first compared by identity. they're only checked for equality if their identities are unequal.

in general, it's assumed that an object will always compare equal to itself. this is explicitly noted in the python 3 reference manual:

User-defined classes that customize their comparison behavior should follow some consistency rules, if possible:

• Equality comparison should be reflexive. In other words, identical objects should compare equal:

  x is y implies x == y

this isn't true for NaN. and that's fine, but because the assumption does hold for pretty much everything else, design choices are made which leave NaN unaccounted for (or at least, give NaN very weird behavior).

to be clear, i'm not necessarily saying that python's behavior here is wrong, or that this optimization is bad because NaN behaves weirdly. but i think it's at least worth pointing out.

case study 2: lua

numerical for-loops in lua look like this:

for i = 1, 10 do
    stuff()
end

the way this is supposed to work is that i starts with the initial value (here that's 1), and each iteration it increments by the step (optional third operand; defaults to 1), until the new value is greater than the limit (10), at which point the loop terminates. so this loop iterates 10 times, from 1 to 10 (inclusive).

so what happens when NaN (0/0) is thrown into the mix? here's the behavior in the reference implementation (puc-rio lua):

-- executes once
for i = 0/0, 10 do
    print(i) -- nan
end

-- executes once
for i = 0/0, 0/0 do
    print(i) -- nan
end

-- never executes
for i = 1, 10, 0/0 do
    print(i)
end

-- executes once
for i = 10, 1, 0/0 do
    print(i) -- 10.0
end

read through that code and try to figure out what's going on. every single one of those results is extremely unintuitive.

here's what's happening: the first two only execute once because the for-loop check is implemented as limit < init, but subsequent iterations check idx <= limit. so NaN passes the first test, but not the second test.

the latter two examples show that NaN is always treated as negative when used as the step, even if you do math.abs(0/0). this is because the check for whether the step is positive is 0 > step, which is of course always false for NaN.

none of this is documented btw, so this is likely a genuine oversight. but that's really interesting: using NaN causes the specific comparison operator used by the implementation to leak into the interpreter's behavior.

the simple solution here is to just check for and disallow NaN in numerical for-loops. lua already raises an error when 0 is used as the step, so it's interesting that there's no such check for NaN.

conclusion i guess

Chris Siebenmmann also previously wrote about the weird behavior of NaNs as map keys in go, so i guess that makes 3 case studies of NaN's weirdness.

NaN is weird. and that means other things behave weirdly with it.