test/system: speed up basic_{setup,teardown}()

While these are not really slow they still take about 100-250ms if I
time this locally. Given they are run for every test this adds up
quickly. Looking at CI logs I can see the timings for skipped
tests are all in 600ms range. So I think it is safe to assume that these
functions need to get faster.

We have over 670 test cases currently so we talk about over 400s spend
in these functions in CI. This allows for big gains.

Now overall this is a tricky trade of, while all tests should cleanup
after themselves there is no guarantee for that as such errors can be
leaked into other tests making debugging much harder. To work at least a
bit against this teardown checks if the test was successful and only
skips the podman commands bases on that. Without it a single flake could
cause all following tets to fail.

As such this commit does the proper setup once one suite start then only
after a test failed.

In order for this to work at all we have to fix all leaks first, see
previous commits. And then for the future keep a very strong eye on
this during reviews.

Also add a PODMAN_BATS_LEAK_CHECK option

By default test must cleanup themselves and to speed up CI we no longer
do any cleanup in teardown by default. However there is still many cases
where we might have to debug a leak so add a new PODMAN_BATS_LEAK_CHECK
env option that can be set and should cause teardown to fail if the test
did not cleanup properly.

Signed-off-by: Paul Holzinger <pholzing@redhat.com>

1 files changed, 3 insertions, 0 deletions

diff --git a/hack/bats b/hack/bats
index 20d4faaa0..3002c0199 100755
--- a/hack/bats
+++ b/hack/bats
@@ -125,6 +125,9 @@ fi
 # Used in 120-load test to identify rootless destination for podman image scp
 export PODMAN_ROOTLESS_USER=$(id -un)
 
+# Make sure to always check for leaks when running locally
+export PODMAN_BATS_LEAK_CHECK=1
+
 # Root
 if [[ "$TEST_ROOT" ]]; then
     echo "# bats ${bats_filter[*]} $TESTS"