I've had the same problem and the hosting company shot me down for using too many server resources. After I disabled the caching plugin the site take much faster to connect but a tiny fraction slower to load. I remember the wp-admin was being so slow, that to do any task seemed like hours.
w3 Total Cahce is a very sophisticated plugin, there are so many pieces to it, that it is very easy to get it configured incorrectly. I did the recommended configuration, and it still made my site very slow to connect.