That would depend if the hole you dug was previously having lots of air voids in it.
Imagine a wafer. Now, crush it! I bet you wouldn't be able to have the same volume it had prior to crushing.
As an engineer, I work at site, and digging holes has always been part of the job.
Imagine this, in contrast to the example you've given. We dig holes at site, but there are holes dug when filled back again would always have excessive soil.
Why is that? It's because the hole dug was previously purely compacted and is free of any voids, so that when filled back, the soil that got excavated became loose and in effect became more voluminous making it necessary for us to compact the excavated area again.