Merging results – merge

Once .results files have been generated using the run_all, you can then merge these into useful statistics using the flitsr.merge command.

merge command line arguments

usage: merge [-h] [-R] [-r [X]] [-i DIR_ARG [DIR_ARG ...]]
             [-e DIR_ARG [DIR_ARG ...]] [-t [FILE]] [-p [FILE]] [-1] [-o FILE]
             [-d DECIMALS] [-g {metric,type}] [-m METRIC [METRIC ...]]
             [-a TYPE [TYPE ...]] [-A METRIC [METRIC ...]]
             [-c CALC [CALC ...]] [--threshold THRESHOLD THRESHOLD THRESHOLD]
             [-P CALC [CALC ...]] [-S CALC [CALC ...]] [-s [{metric,type}]]
             [-l CALC [CALC ...]] [-E CALC [CALC ...]]
             [-b BASE_TYPES [BASE_TYPES ...]] [-k] [-T]

Named Arguments

-R, --relative

Compute relative values instead of absolute values

Default: False

-r, --recurse

Activates the scripts recursive mode. This makes the script recursively look in sub-directories of the current directory for results files. An optional maximum recurse limit X can be given.

Default: False

-i, --incl

Specifies particular directories to include for the recursive option. You may optionally give a depth with each directory in the format DIR_ARG=”[<depth>:]<dir name>”, where the depth is an integer starting with 1 (the current directory). By default a depth of “*” is used, indicating any depth is valid. NOTE: the colon character (“:”) should not appear in the directory name, but if it must, then the depth must also be given. This option may be specified multiple times

Default: []

-e, --excl

Specifies particular directories to exclude for the recursive option. You may optionally give a depth with each directory (see –incl for format).This option may be specified multiple times

Default: []

-t, --tex

Specifies that an additional output file should be generated that contains the results in a LaTeX table (in .tex format). By default this is stored in results.tex, but an optional filename FILE may be given

-p, --perc@n

Specifies that an additional output file should be generated that contains the percentage-at-n results. By default this is stored in perc_at_n_results, but an optional filename FILE may be given

-1, --inline-perc@n

Instead of producing a separate percentage-at-n file, place the results inline in the results file

-o, --output

Store the results in the file with filename FILE. By default the name “results” is used

Default: results

-d, --decimals

Sets the precision (number of decimal points) for the output of all of the calculations Does not impact percentage-at-n values. (default 24, i.e. all python-stored significance).

Default: 24

-g, --grouping-order

Possible choices: metric, type

Specifies the way in which the output should be grouped. “metric” groups first by metrics and then by types, “type” does the opposite (default ‘metric’)

Default: 'metric'

-m, --metrics

Possible choices: ample, anderberg, arith_mean, barinel, cohen, dice, dstar, euclid, fleiss, geometric, goodman, gp13, hamann, hamming, harmonic, hyperbolic, jaccard, kulczynski1, kulczynski2, m1, m2, naish2, ochiai, ochiai2, overlap, rogers_tanimoto, rogot1, rogot2, russell_rao, sbi, scott, simpl_match, sokal, sorensen_dice, tarantula, wong1, wong2, wong3, zoltar, artemis, sbfl, flitsr, multi, parallel

Specify the metrics to merge results for. By default all metrics that appear in filenames of found files will be merged. Note that this option only restricts the output, all files available are still read, however files not existing are not read.

-a, --advanced-types

Specify the list of advanced types to include when merging. By default all available advanced types that appear in filenames of found files are included.

-A, -f, --advanced-metrics, --flitsrs

Possible choices: ample, anderberg, arith_mean, barinel, cohen, dice, dstar, euclid, fleiss, geometric, goodman, gp13, hamann, hamming, harmonic, hyperbolic, jaccard, kulczynski1, kulczynski2, m1, m2, naish2, ochiai, ochiai2, overlap, rogers_tanimoto, rogot1, rogot2, russell_rao, sbi, scott, simpl_match, sokal, sorensen_dice, tarantula, wong1, wong2, wong3, zoltar, artemis, sbfl, flitsr, multi, parallel

Specify the metrics for which to display advanced type values for. By default all metric’s advanced type values are shown.

-c, --calcs

Specify the list of calculations to include when merging. By default all available calculations are included. NOTE: the names of the calculations need to be found in the corresponding .results files

--threshold

Format: –threshold <calculation> {above, below} <float>. Specifies that an additional calculation should be added that counts the number of versions where the given calculation is above or below the given float threshold. The calculations are the same as for the –calcs argument.

Default: []

-P, --percentage

Specify calculations that must be intepreted as a percentage value. NOTE: the names of the calculations need to be found in the corresponding .results files

-S, --sum

Specify calculations that should be simply summed together instead of averaged. NOTE: the names of the calculations need to be found in the corresponding .results files

-s, --significance

Possible choices: metric, type

Specifies that additional significance tests should be performed to test the differences in results. The significance tests will either be conducted between metrics of the same type [metric] or between types using the same metric [type] (default type). If type is given, and the –tex option is also used, significance indicators will be added to the TeX output indicating advanced types significantly greater than their baselines (see –base-type for changing the baseline, and –less-significance for significantly less)

-l, --less-significance

Intended for use with the –significance and –tex options. Specify the calculations whose result is to be tested for significantly less than the baseline instead of significantly greater, which is the default. Affects the significance indicators for the TeX output. NOTE: the names of the calculations need to be found in the corresponding .results files

Default: []

-E, --equal-significance

Intended for use with the –significance and –tex options. Specify the calculations whose result is to be tested for significantly equal to the baseline instead of significantly greater, which is the default. Affects the significance indicators for the TeX output. NOTE: the names of the calculations need to be found in the corresponding .results files

Default: []

-b, --base-types

Intended for use with –significance and –tex options. Specify the base type that will be compared against for all other types when adding significance test annotations to the TeX output. Use the format “{}_<type>” if the baseline to compare to is dependant on the type.

-k, --keep-order

Instead of sorting the metrics and calculations by alphabetical order, keep the order that the are specified on the command line by the -m and -c options. This option does nothing to the corresponding order if either of those options are unspecified

Default: False

-T, --top-results

Additionally print out which technique performed the best for each calculation. Indicators are added to both the normal output, as well as the TeX output in the form of boldfacing.

Default: False