2 experts (me, I'm none of them) 3 different opinions on that case, some say joomlas cache system is pretty buggy and causes lots of trouble, most people tell you to enable it - check it out by yourself: get firefox with firebug add.on
http://getfirebug.com/
, there's two more add-ons you can use within firebug, 1. yslow (yahoo)
http://developer.yahoo.com/yslow/
and 2. page speed (google)
http://code.google.com/speed/page-speed/
play around with the settings to check what fits best in your unique case, may be you benefit using the cache. And don't forget the "gzip" option in the global settings. Perhaps using rockgzipper
http://www.rockettheme.com/extensions-joomla/rokgzipper
is an option for you too.