Ok, so I fixed it... well did and did not fix it.

Somebody on the Xerox forum mentioned to remove all the drivers again manually, and doing this by looking at the driver details in the device manager. Much to my surprise, when I looked at the driver details, I could see that the provider of the driver is mentioned as "ESET". ESET is clearly NOT Xerox. Rather, it's the provider of my anti-virus program NOD32 (in the "smart security" suite). The driver is also shown as "not digitally signed". See below :
Strangely enough, including the image doesn't work, so just click this link

So I then deleted all the drivers AND uninstalled my ESET anti virus.
I re-installed the Windows 8.1 x64 xerox scan drivers. Didn't work!! In the device manager, it's listed as an "unknown device". I says something is wrong with the installation of the driver and it cannot continue. No amount of trying or pointing to the drivers worked!!

Then I got the idea of using the Xerox Windows 7 x64 scan driver. THAT WORKED!!! I can scan via TWAIN and WIA. Me happy!!!!

But then I re-installed ESET Smart security, and from that moment on, it quit working again. In the device manager, there is that ESET driver again blocking the access to the scanner driver.

So now I know what the problem is, but I can't really solve it. It's not really an option to work without an anti-virus program. It IS an option though to change anti-virus program, even if this ESET Smart security (NOD32) was paid for.

But before I do that, I guess I'll contact ESET first and see what they suggest...

BTW, that also explains why I did work on my Windows 7 laptop I guess, since I use Bitdefender for anti-virus on that one...