We need to cleanup after each execution so we can still re-use the same paths and prevent collision with the on disk validation