Fix build duration when build is not finished yet
## What does this MR do?
## Are there points in the code the reviewer needs to double check?
## Why was this MR needed?
## What are the relevant issue numbers?
Fixes#19937.
## Does this MR meet the acceptance criteria?
- [x] No CHANGELOG since it fixes a regression not yet in a stable release (I think)
- Tests
- [x] Added for this feature/bug
- [ ] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !5323
b9ed9d658a