lua: fix integer overflow in LNUM patch

lua: fix integer overflow in LNUM patch

Safely detect integer overflow in try_addint() and try_subint().
Old code relied on undefined behavior, and recent versions of GCC on x86
optimized away the if-statements.
This caused integer overflow in Lua code instead of falling back to
floating-point numbers.

Signed-off-by: Adam Bailey <aebailey@gmail.com>
(cherry picked from commit 3a2e7c30d3e6a187ba1df740cdb24c8ad84dfe48)