We have always taken the position that an XML sitemap is essential to a well-designed website. The idea is that it helps the spiders navigate the site and ensures that no pages are missed - i.e. the site gets credit for all its pages, even if its internal link structure is deficient. In addition, you can resubmit the sitemap to Google whenever you add or change pages - this is like a white-hat way to resubmit your site to the search engines.

There is another point of view, however - if your structure is deficient, you should know about it and fix it! You can actually use what the spider finds to your benefit, i.e. to fix whatever structural issues you have.
Bottom line - if you have a large site, and the resources and determination to fix the problems that are identified, then remove your sitemap and make sure that you have a well-ordered site structure - the spiders will be able to navigate without a sitemap. You could still hve an HTML sitemap for humans though. You should also keep an eye on how often the spiders revisit you, if the interval is too long you can try goosing them with an XML sitemap submission.