Don't stomp on build flags (#635)

Let CMake handle the default optimizations for various configurations. We don't need to override them. In fact, overriding them makes it impossible for users to override them themselves.