Using IPSec is a good technique for securing traffic on your internal network and it generally doesn't add that much overhead to your network traffic (50 bytes for ESP, several rounds for setting up SAs, and so on). But if your users frequently download massively large files, the additional CPU usage for clients can be significant. And if your servers are under heavy load then the CPU burden for IPSec can impact them also.
A good solution in this kind of situation is to use IPSec offload cards for your clients and servers that need them. Check with Intel, 3Com, and other vendors for suitable hardware for your environment.