well, you get integer overflow due to incorrect range check for cnt:
size_t ints_size = cnt * HASH_SIZE;
Check rather that cnt <= SIZE_MAX/HASH_SIZE
Anyways, it does not help because you use alloca(), you probably have 8 MiB stack size limit on Linux.
tree_hash_cnt could be simplified, I have
size_t cnt = (size_t)1 << __fls(count - 1);
(__fls from Linux source code)