Endianness issues in Blowfish and IDEA

[Olaf Titz <olaf@bigred.inka.de>]

I'm developing a version of CIPE which will use the modular crypto
API. From my first interoperability tests, it looks like both the
Blowfish and the IDEA modules in the current international kernel get
endian-ness in their encryption functions wrong.

CIPE historically uses Blowfish in little-endian mode. With that I
mean that the plaintext/ciphertext blocks are assumed to consist of
two 32-bit units which are fetched and stored in little-endian order.[1]
This was a result of a misunderstanding and following coding error on
my part, and it has been preserved for compatibility - I now use a
Blowfish implementation which gives the choice of little and big
endian mode and CIPE uses the little-endian one.

This means it can not talk to a regular Blowfish which by the spec
is big-endian. But it does talk to the crypto API cipher-blowfish
module, so the latter looks to me to be against the spec. Indeed, the
blowfish_encrypt routine in cipher-blowfish.c converts its data like
this:

input is const u8 *in8
this is assigned to u32 *in_blk = (u32 *)in8;
the rounds have this assignment
        yl = *(in_blk++);
        yr = *(in_blk++);
The internal variables yl, yr are supposed to be in native byte order.
Because the input is assumed to big endian, here the canonicalization
is missing. This is the same error I made with my Blowfish and I
suspect/fear it is rather common in other implementations.[2]

I know well what the author meant by the comment "This function
mustn't respect endianness", because I once ran into the same trap ;-)
Nevertheless, it is not correct. The code as it is assumes native byte
order for the plaintext/ciphertext blocks and thus will not
interoperate between big-endian and little-endian hosts.

Another problem appears to exist in the cipher-idea module: the
ideaCipher function explicitly swaps bytes in the input for
non-little-endian hosts (#ifndef LITTLE_ENDIAN_HOST), iow.
canonicalizes to little-endian, which is just the wrong direction.
This has to be a coding mistake, as even the comment at the head of
the file talks about big-endianness.

Other implementations of IDEA (e.g. the one found in OpenSSL, see the
file crypto/idea/idea_lcl.h) seem to agree with me[3] that IDEA is
assumed big-endian, too. This means the code will interoperate with
itself on any host, but not with other IDEA implementations.

As long as this is only used for encrypted file systems,
interoperability is not that vital, but for networking this will have
to be fixed somehow.

Olaf

[1] or in other words, the Blowfish encrypt routine has an initial and
final permutation of the bytes 01234567 -> 32107654.

[2] OpenSSL does it right, although the code is extremely non-obvious.

[3] It is nowhere explicitly specified, but the reference
implementation I adapted for CIPE has ntohs conversions.


Linux-crypto:  cryptography in and on the Linux system
Archive:       http://mail.nl.linux.org/linux-crypto/