sha512: fix bad hardcoded constant in sha512_final()

The SHA512 implementation shipped with usign uses a wrong hardcoded
numeric constant in the final block padding code.

An additional transform step must be done when there are at least
SHA512_BLOCK_SIZE - 16 = 112 bytes in the state buffer, however the
existing code incorrectly transformed buffer data larger than or
equal to 110 bytes as well, resulting in invalid hash calculations
when exactly 110 or 111 remaining bytes were left in the buffer.

To reproduce the issue, sign a plaintext file with a size of exactly
128 * N + 64 + 110 or 128 * N + 64 + 111 bytes using signify-openbsd
and attempt to verify the signature using usign:

    $ signify-openbsd -G -n -p test.pub -s test.key
    $ dd if=/dev/zero of=test.msg bs=1 count=$((64 + 110))
    $ signify-openbsd -S -x test.sig -s test.key -m test.msg
    $ usign -V -p test.pub -x test.sig -m test.msg

Fix this issue by replacing the bad numeric constanct with a macro
expression resulting in the proper value.

The fix has been verified by cross checking the intermedia hash
results with results from OpenSSL's SHA512 implementation and by
comparing the usign SHA512 code with the hashing code shipped with
signify-openbsd.

Ref: https://forum.openwrt.org/t/signature-check-failed/41945
Signed-off-by: Jo-Philipp Wich <jo@mein.io>

diff --git a/sha512.c b/sha512.c
index 68a9e65db8f85d72e14a044dd23d257fdce326fd..d06d65b653d4ebee110521eb7dc9d7d0ed3df5b0 100644 (file)
--- a/sha512.c
+++ b/sha512.c
@@ -232,7 +232,7 @@ void sha512_final(struct sha512_state *s, uint8_t *hash)
                memset(&s->partial[last_size], 0,
                       SHA512_BLOCK_SIZE - last_size);
 
-       if (last_size > 110) {
+       if (last_size > (SHA512_BLOCK_SIZE - 16)) {
                sha512_block(s, s->partial);
                memset(s->partial, 0, sizeof(s->partial));
        }