It seems as though your processor does not have hyperthreading. As per Intel's specifications http://ark.intel.com/products/37092/Intel-Xeon-Processor-E5502-4M-Cache-1_86-GHz-4_80-GTs-Intel-QPI
Having all four cores assigned to a single VM is probably causing the CPU performance issues. Have you checked the CPU Read values? If you are seeing more than 1000msec per vCPU, you can expect performance degredation.
Chaning the VM from four to one or two vCPUs shouldn't be a problem unless you have some software running which is consuming a lot of CPU time. I've run SBS2011 on a single vCPU without any problems.