Test2::Aggregate - Aggregate tests for increased speed
use Test2::Aggregate; use Test2::V0; # Or 'use Test::More' etc if your suite uses an other framework Test2::Aggregate::run_tests( dirs => \@test_dirs ); done_testing();
Version 0.17
Aggregates all tests specified with dirs (which can even be individual tests) to avoid forking, reloading etc. that can help with performance (dramatically if you have numerous small tests) and also facilitate group profiling. It is quite common to have tests that take over a second of startup time for milliseconds of actual runtime - Test2::Aggregate removes that overhead. Test files are expected to end in .t and are run as subtests of a single aggregate test.
dirs
A bit similar (mainly in intent) to Test::Aggregate, but no inspiration was drawn from the specific module, so simpler in concept and execution, which makes it much more likely to work with your test suite (especially if you use modern tools like Test2::Suite). It does not even try to package each test by default (there is an option), which may be good or bad, depending on your requirements.
Generally, the way to use this module is to try to aggregate sets of quick tests (e.g. unit tests). Try to iteratively add tests to the aggregator, using the lists option, so you can easily edit and remove those that do not work. Trying an entire, large, suite in one go is not a good idea, as an incompatible test can break the run making the subsequent tests fail (especially when doing things like globally redefining built-ins etc.) - see the module usage notes for help.
lists
The module can work with Test::Builder / Test::More suites, but you will have less issues with Test2::Suite (see notes).
run_tests
my $stats = Test2::Aggregate::run_tests( dirs => \@dirs, # optional if lists defined lists => \@lists, # optional if dirs defined exclude => qr/exclude_regex/, # optional include => qr/include_regex/, # optional root => '/testroot/', # optional load_modules => \@modules, # optional package => 0, # optional shuffle => 0, # optional sort => 0, # optional reverse => 0, # optional unique => 1, # optional repeat => 1, # optional, requires Test2::Plugin::BailOnFail for < 0 slow => 0, # optional override => \%override, # optional, requires Sub::Override stats_output => $stats_output_path, # optional extend_stats => 0, # optional test_warnings => 0, # optional allow_errors => 0, # optional pre_eval => $code_to_eval, # optional dry_run => 0, # optional slurp_param => {binmode => $mode} # optional );
Runs the aggregate tests. Returns a hashref with stats like this:
$stats = { 'test.t' => { 'test_no' => 1, # numbering starts at 1 'pass_perc' => 100, # for single runs pass/fail is 100/0 'timestamp' => '20190705T145043', # start of test 'time' => '0.1732', # seconds - only with stats_output 'warnings' => $STDERR # only with test_warnings on non empty STDERR } };
The parameters to pass:
dirs (either this or lists is required)
An arrayref containing directories which will be searched recursively, or even individual tests. The directories (unless shuffle or reverse are true) will be processed and tests run in order specified. Test files are expected to end in .t.
shuffle
reverse
.t
lists (either this or dirs is required)
Arrayref of flat files from which each line will be pushed to dirs (so they have a lower precedence - note root still applies, don't include it in the paths inside the list files). If the path does not exist, it will currently be silently ignored, however the "official" way to skip a line without checking it as a path is to start with a # to denote a comment.
root
#
This option is nicely combined with the --exclude-list option of yath (the Test2::Harness) to skip the individual runs of the tests you aggregated.
--exclude-list
yath
exclude (optional)
exclude
A regex to filter out tests that you want excluded.
include (optional)
include
A regex which the tests have to match in order to be included in the test run. Applied after exclude.
root (optional)
If defined, must be a valid root directory that will prefix all dirs and lists items. You may want to set it to './' if you want dirs relative to the current directory and the dot is not in your @INC.
'./'
@INC
load_modules (optional)
load_modules
Arrayref with modules to be loaded (with eval "use ...") at the start of the test. Useful for testing modules with special namespace requirements.
eval "use ..."
package (optional)
package
Will package each test in its own namespace. While it may help avoid things like redefine warnings, from experience, it can break some tests, so it is disabled by default.
override (optional)
override
Pass Sub::Override compatible key/values as a hashref.
repeat (optional)
repeat
Number of times to repeat the test(s) (default is 1 for a single run). If repeat is negative, Test2::Plugin::BailOnFail is required, as the tests will repeat until they bail on a failure. It can be combined with test_warnings in which case a warning will also cause the test run to end.
test_warnings
unique (optional)
unique
From v0.11, duplicate tests are by default removed from the running list as that could mess up the stats output. You can still define it as false to allow duplicate tests in the list.
sort (optional)
sort
Sort tests alphabetically if set to true. Provides a way to fix the test order across systems.
shuffle (optional)
Random order of tests if set to true. Will override sort.
reverse (optional)
Reverse order of tests if set to true.
slow (optional)
slow
When true, tests will be skipped if the environment variable SKIP_SLOW is set.
SKIP_SLOW
test_warnings (optional)
Tests for warnings over all the tests if set to true - this is added as a final test which expects zero as the number of tests which had STDERR output. The STDERR output of each test will be printed at the end of the test run (and included in the test run result hash), so if you want to see warnings the moment they are generated leave this option disabled.
allow_errors (optional)
allow_errors
If enabled, it will allow errors that exit tests prematurely (so they may return a pass if one of their subtests had passed). The option is available to enable old behaviour (version <= 0.12), before the module stopped allowing this.
dry_run (optional)
dry_run
Instead of running the tests, will do ok($testname) for each one. Otherwise, test order, stats files etc. will be produced (as if all tests passed).
ok($testname)
slurp_param (optional)
slurp_param
If you are using list files, by default, Path::Tiny::slurp_utf8 will be used to read them. If you would like to use slurp with your own parameters instead, like a UTF16 binmode etc, you can pass them here.
Path::Tiny::slurp_utf8
slurp
pre_eval (optional)
pre_eval
String with code to run with eval before each test. You might be inclined to do this for example:
pre_eval => "no warnings 'redefine';"
You might expect it to silence redefine warnings (when you have similarly named subs on many tests), but even if you don't set warnings explicitly in your tests, most test bundles will set warnings automatically for you (e.g. for Test2::V0 you'd have to do use Test2::V0 -no_warnings => 1; to avoid it).
use Test2::V0 -no_warnings => 1;
stats_output (optional)
stats_output
stats_output specifies a path where a file will be created to print out running time per test (average if multiple iterations) and passing percentage. Output is sorted from slowest test to fastest. On negative repeat the stats of each successful run will be written separately instead of the averages. The name of the file is caller_script-YYYYMMDDTHHmmss.txt. If '-' is passed instead of a path, then the output will be written to STDOUT. The timing stats are useful because the test harness doesn't normally measure time per subtest (remember, your individual aggregated tests become subtests). If you prefer to capture the hash output of the function and use that for your reports, you still need to define stats_output to enable timing (just send the output to /dev/null, /tmp etc).
caller_script-YYYYMMDDTHHmmss.txt
'-'
/dev/null
/tmp
extend_stats (optional)
extend_stats
This option exist to make the default output format of stats_output be fixed, but still allow additions in future versions that will only be written with the extend_stats option enabled. Additions with extend_stats as of the current version:
- starting date/time in ISO_8601.
Not all tests can be modified to run under the aggregator, it is not intended for tests that require an isolated environment, do overrides etc. For other tests which can potentially run under the aggregator, sometimes very simple changes may be needed like giving unique names to subs (or not warning for redefines, or trying the package option), replacing things that complain, restoring the environment at the end of the test etc.
Unit tests are usually great for aggregating. You could use the hash that run_tests returns in a script that tries to add more tests automatically to an aggregate list to see which added tests passed and keep them, dropping failures. See later in the notes for a detailed example.
Trying to aggregate too many tests into a single one can be counter-intuitive as you would ideally want to parallelize your test suite (so a super-long aggregated test continuing after the rest are done will slow down the suite). And in general more tests will run aggregated if they are grouped so that tests that can't be aggregated together are in different groups.
In general you can call Test2::Aggregate::run_tests multiple times in a test and even load run_tests with tests that already contain another run_tests, the only real issue with multiple calls is that if you use repeat < 0 on a call, Test2::Plugin::BailOnFail is loaded so any subsequent failure, on any following run_tests call will trigger a Bail.
Test2::Aggregate::run_tests
repeat < 0
If you haven't switched to the Test2::Suite you are generally advised to do so for a number of reasons, compatibility with this module being only a very minor one. If you are stuck with a Test::More suite, Test2::Aggregate can still probably help you more than the similarly-named Test::Aggregate... modules.
Test::Aggregate...
Although the module tries to load Test2 with minimal imports to not interfere, it is generally better to do use Test::More; in your aggregating test (i.e. alongside with use Test2::Aggregate).
Test2
use Test::More;
use Test2::Aggregate
BEGIN / END blocks will run at the start/end of each test and any overrides etc you might have set will apply to the rest of the tests, so if you use them you probably need to make changes for aggregation. An example of such a change is when you have a *GLOBAL::CORE::exit override to test scripts that can call exit(). A solution is to use something like Test::Trap:
BEGIN
END
*GLOBAL::CORE::exit
exit()
BEGIN { unless ($Test::Trap::VERSION) { # Avoid warnings for multiple loads in aggregation require Test::Trap; Test::Trap->import(); } }
Test::Class is sort of an aggregator itself. You make your tests into modules and then load them on the same .t file, so ideally you will not end up with many .t files that would require further aggregation. If you do, due to the Test::Class implementation specifics, those .t files won't run under Test2::Aggregator.
The environment variable AGGREGATE_TESTS will be set while the tests are running for your convenience. Example usage is making a test you know cannot run under the aggregator check and croak if it was run under it, or a module that can only be loaded once, so you load it on the aggregated test file and then use something like this in the individual test files:
AGGREGATE_TESTS
eval 'use My::Module' unless $ENV{AGGREGATE_TESTS};
If you have a custom test bundle, you could use the variable to do things like disable warnings on redefines only for tests that run aggregated:
use Import::Into; sub import { ... 'warnings'->unimport::out_of($package, 'redefine') if $ENV{AGGREGATE_TESTS}; }
Another idea is to make the test die when it is run under the aggregator, if, at design time, you know it is not supposed to run aggregated.
There are many approaches you could do to use Test2::Aggregate with an existing test suite, so for example you can start by making a list of the test files you are trying to aggregate:
Test2::Aggregate
find t -name '*.t' > all.lst
If you have a substantial test suite, perhaps try with a portion of it (a subdir?) instead of the entire suite. In any case, try running them aggregated like this:
use Test2::Aggregate; use Test2::V0; # Or Test::More; my $stats = Test2::Aggregate::run_tests( lists => ['all.lst'], ); open OUT, ">pass.lst"; foreach my $test (sort {$stats->{$a}->{test_no} <=> $stats->{$b}->{test_no}} keys %$stats) { print OUT "$test\n" if $stats->{$test}->{pass_perc}; } close OUT; done_testing();
Run the above with prove or yath in verbose mode, so that in case the run hangs (it can happen), you can see where it did so and edit all.lst removing the offending test.
prove
all.lst
If the run completes, you have a "starting point" - i.e. a list that can run under the aggregator in pass.lst. You can try adding back some of the failed tests - test failures can be cascading, so some might be passing if added back, or have small issues you can address.
pass.lst
Try adding test_warnings => 1 to run_tests to fix warnings as well, unless it is common for your tests to have STDERR output.
test_warnings => 1
STDERR
To have your entire suite run aggregated tests together once and not repeat them along with the other, non-aggregated, tests, it is a good idea to use the --exclude-list option of the Test2::Harness.
Test2::Harness
Hopefully your tests can run in parallel (prove/yath -j), in which case you would split your aggregated tests into multiple lists to have them run in parallel. Here is an example of a wrapper around yath, to easily handle multiple lists:
prove/yath -j
BEGIN { my @args = (); foreach (@ARGV) { if (/--exclude-lists=(\S+)/) { my $all = 't/aggregate/aggregated.tests'; `awk '{print "t/"\$0}' $1 > $all`; push @args, "--exclude-list=$all"; } else { push @args, $_ if $_; } } push @args, qw(-P...) # Preload module list (useful for non-aggregated tests) unless grep {/--cover/} @args; @ARGV = @args; } exec ('yath', @ARGV);
You would call it with something like --exclude-lists=t/aggregate/*.lst, and the tests listed will be excluded (you will have them running aggregated through their own .t files using Test2::Aggregate).
--exclude-lists=t/aggregate/*.lst
Dimitrios Kechagias, <dkechag at cpan.org>
<dkechag at cpan.org>
Please report any bugs or feature requests to bug-test2-aggregate at rt.cpan.org, or through the web interface at https://rt.cpan.org/NoAuth/ReportBug.html?Queue=Test2-Aggregate. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes. You could also submit issues or even pull requests to the github repo (see below).
bug-test2-aggregate at rt.cpan.org
https://github.com/SpareRoom/Test2-Aggregate
Copyright (C) 2019, SpareRoom.com
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install Test2::Aggregate, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Test2::Aggregate
CPAN shell
perl -MCPAN -e shell install Test2::Aggregate
For more information on module installation, please visit the detailed CPAN module installation guide.