you might want to try the vmx configs in the post above, or
try selecting the voodoo 9.5 kernel during install.

In short, if you can get the CD to boot and install, it should work on the real vm system.
The cd when it boots to install it uses 9.5 kernel.
Unfortunately vmware doesnt emulate all the same on different hardware.
you i7 processor wasnt available when iatkos made v7.

You may try the latest boot loader by copying it to usb, booting vmware on cd while usb attached.
then using the /usr/sbin/installer -pkg bootpkgname -target /Volume/yourvolume through a terminal session. (must be little unix savy)

On the other hand, try one of the other (v1) 2 bootloaders with smbios enablers on iatkos installation itself.
I dont have an exact answer, each hardware is different.
you dont have to reinstall the whole OS every time, just pick only the option you want to try and disable other packages on
the customize screen.

Do you know what line your machine reboots at? doesnt show you any of the boot messages or kernel panic messages?