I dont cache the site when building a site. I only optimise caching on completion.
The reason I do not cache sites when building, is it can sometimes lead to caching stale content and you are scratching your head thinking why does it look like that when i have just changed it.
Same with Cloudflare too. I Just use it for DNS and not proxied when building. I try to remove any variables that can affect the clean site until the build has been completed.