Base tool – flitsr

This section describes usage instructions for developers wishing to use flitsr to assist in debugging programs.

flitsr command line arguments

usage: flitsr [-h] [-o OUTPUT] [--csv] [--spectrum-csv] [-m METRIC] [-r] [-c]
              [--split] [--collapse] [-a] [-t TYPES] [-p [FORMAT_STR]]
              [--no-override] [-d DECIMALS] [--parallel]
              [--parallel-parType {bdm,msp,hwk,vwk}] [--artemis]
              [--artemis-numUniverse numUniverse]
              [--artemis-maxUniverse maxUniverse] [--artemis-p p] [--sbfl]
              [--flitsr]
              [--flitsr-internal-ranking {auto,conf,original,reverse,flitsr}]
              [--flitsr-default-metric {ample,anderberg,arith_mean,barinel,cohen,dice,dstar,euclid,fleiss,geometric,goodman,gp13,hamann,hamming,harmonic,hyperbolic,jaccard,kulczynski1,kulczynski2,m1,m2,naish2,ochiai,ochiai2,overlap,rogers_tanimoto,rogot1,rogot2,russell_rao,sbi,scott,simpl_match,sokal,sorensen_dice,tarantula,wong1,wong2,wong3,zoltar}]
              [--multi] [--multi-cutoff cutoff] [--tiebrk] [--rndm] [--otie]
              [--first] [--avg] [--med] [--last] [--weffort N] [--one-top x]
              [--all-top x] [--perc-top x] [--one-top1] [--all-top1]
              [--perc-top1] [--perc@n] [--auc] [--pauc] [--lauc]
              [--precision-at x] [--recall-at x] [--fault-num] [--fault-ids]
              [--fault-elems] [--fault-all] [--cutoff-eval MODE]
              [--cutoff-strategy STRATEGY]
              input

Positional Arguments

input

The coverage file (TCM) or directory (GZoltar) containing the coverage collected for the system over the test suite

Named Arguments

-o, --output

Specify the output file to use for all output (default: STDOUT).

Default: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>

--csv

By default FLITSR will output the ranking in it’s own FLITSR ranking format. Enabling this option will allow FLITSR to output the ranking in CSV format compatible with GZoltar’s CSV ranking format instead.

Default: False

--spectrum-csv

Enabling this option will cause FLITSR to output the spectrum in CSV format.

Default: False

-m, --metric

Possible choices: ample, anderberg, arith_mean, barinel, cohen, dice, dstar, euclid, fleiss, geometric, goodman, gp13, hamann, hamming, harmonic, hyperbolic, jaccard, kulczynski1, kulczynski2, m1, m2, naish2, ochiai, ochiai2, overlap, rogers_tanimoto, rogot1, rogot2, russell_rao, sbi, scott, simpl_match, sokal, sorensen_dice, tarantula, wong1, wong2, wong3, zoltar, artemis, sbfl, flitsr, multi, parallel

The underlying (SBFL) metric(s) to use when either ranking (if the sbfl option is given), or running the FLITSR algorithm. Option may be supplied multiple times to run multiple metrics. Specifying multiple metrics will output the results of each metric to a seperate file using the metric’s name instead of stdout. Allowed values are: [ample, anderberg, arith_mean, barinel, cohen, dice, dstar, euclid, fleiss, geometric, goodman, gp13, hamann, hamming, harmonic, hyperbolic, jaccard, kulczynski1, kulczynski2, m1, m2, naish2, ochiai, ochiai2, overlap, rogers_tanimoto, rogot1, rogot2, russell_rao, sbi, scott, simpl_match, sokal, sorensen_dice, tarantula, wong1, wong2, wong3, zoltar] (default: ochiai). When using the FLITSR advanced technique (or similar), you may also specify an advanced technique here to use as base metric, in which case the –flitsr-default_metric option may be used to likewise specify the base metric for that advanced technique. Note that it is your job to determine whether the given advanced technique is compatible with the FLITSR (or similar) algorithm, as this is not ensured for all advanced types.

-r, --ranking

Changes FLITSR’s expected input to be an SBFL ranking in Gzoltar or FLITSR format (determined automatically), instead of the usual coverage, and produces the specified calculations (or just the ranking if no calculations are given). NOTE: any non-output based options will be ignored with this option

Default: False

-c, --method

The default for FLITSR is to use the collected coverage as-is and merely produce the ranking in terms of the names/labels given to the elements. Alternatively, using this option, FLITSR can assume the coverage given is a statement level coverage, and will attempt to collapse this coverage to produce a method level coverage result. This collapse is done by constructing a coverage matrix with only the method names, where the execution of a method is determined by the union of the executions of its statements. Bugs added to the coverage are handled in a similar fashion.

Default: False

--split

When given, this option causes faults that are a combination of two or more sub-faults in mutually exclusive parts of the system to be split into separate identified faults. As a by-product this also drops faults that are not exposed by failing tests

Default: False

--collapse

Collapse dynamic basic-block groups into singular elements for the ranking an calculations

Default: False

-a, --all

Used in the evaluation of FLITSR against other techniques. Runs all metrics given in suspicious.py and both FLITSR and FLITSR* extensions over each metric. Also enables all of the above evaluation calculations. Prints the results out to files named [<flitsr method>_]<metric>.run for each FLITSR method and metric

Default: False

-t, --types

Specify the advanced type combination to use when running FLITSR. Note that this argument overrides any of the individual advanced type arguments given to FLITSR (such as –multi, –sblf, etc.).This argument may be specified multiple times to add multiple type combinations to run. The format for this argument is: “–types <type>[+<type>…]”, where each <type> is a (case-insensitive) FLITSR advanced type. Allowed types are: [‘ARTEMIS’, ‘SBFL’, ‘FLITSR’, ‘MULTI’, ‘PARALLEL’]

-p, --print-params

When producing multiple output files, specifies that the parameters for advanced types,given by the –types option, should be included in the names of these output files. Without an argument, this option produces file names with advanced type parameters of the form “{k}-{v}” where {k} is the parameter name, and {v} is the value. Other formats can be constructed by instead providing an argument using the same “{k}”, “{v}” syntax.

--no-override

By default FLITSR will override the output file(s) if they already exist, printing a warning message. This option instead allows FLITSR to leave output files that already exist, skipping that output and continuing with the rest of the outputs (if any)

Default: False

-d, --decimals

Sets the precision (number of decimal points) for the output of all of the calculations (default: 2)

Default: 2

--cutoff-eval

Possible choices: worst, best, resolve

Specifies the performance mode to use when using a multi-fault fixing cut-off strategy to produce rankings. Allowed values are: [worst, best, resolve] (default ‘worst’)

Default: 'worst'

--cutoff-strategy

Cuts off the ranking using the given strategy’s cut-off point. This affects both the rank output method and any calculations. Allowed values are: [basis, oba, mba_dominator, mba_zombie, mba_5_perc, mba_10_perc, mba_const_add, mba_optimal, aba] (default None). For basis, an optional value n may be given (e.g. basis=n) that determines the number of bases included before the cutoff

Clustering techniques

One of the following clustering techniques may be specified, along with it’s options

See flitsr advanced types for more information on flitsr advanced types.

--parallel

Run one of the parallel debugging algorithms on the spectrum to produce multiple spectrums, and process all other options on each spectrum.

--parallel-parType

Possible choices: bdm, msp, hwk, vwk

(default: ‘msp’)

Default: 'msp'

Ranking techniques

One of the following advanced ranking techniques may be specified, along with it’s options (default: flitsr)

See flitsr advanced types for more information on flitsr advanced types.

--artemis

Run the ARTEMIS technique on the spectrum to produce the ranked lists. This option may be combined with FLITSR and parallel to produce a hybrid technique.

--artemis-numUniverse

(default: 17)

Default: 17

--artemis-maxUniverse

(default: 20)

Default: 20

--artemis-p

(default: 1e-05)

Default: 1e-05

--sbfl

Disables the FLITSR algorithm so that only the base metric is used to produce the ranking. This is equivalent to using the base metric as-is, but allows the user to run these metrics within the FLITSR framework

--flitsr

Run the main FLITSR algorithm over the spectrum to produce ranked lists.

--flitsr-internal-ranking

Possible choices: auto, conf, original, reverse, flitsr

(default: ‘auto’)

Default: 'auto'

--flitsr-default-metric

Possible choices: ample, anderberg, arith_mean, barinel, cohen, dice, dstar, euclid, fleiss, geometric, goodman, gp13, hamann, hamming, harmonic, hyperbolic, jaccard, kulczynski1, kulczynski2, m1, m2, naish2, ochiai, ochiai2, overlap, rogers_tanimoto, rogot1, rogot2, russell_rao, sbi, scott, simpl_match, sokal, sorensen_dice, tarantula, wong1, wong2, wong3, zoltar

(default: ‘ochiai’)

Default: 'ochiai'

--multi

Run the FLITSR* algorithm over the spectrum to produce ranked lists.

--multi-cutoff

(default: None)

Tie breaking strategy

Specifies the tie breaking strategy to use for FLITSR and the localization

--tiebrk

Breaks ties using only execution counts

Default: Tiebrk.ORIG

--rndm

Breaks ties by a randomly ordering

--otie

Breaks ties by using the original base metric ranking (in the case of FLITSR) and by execution counts otherwise

Calculations

The following arguments replace the default ranking output of FLITSR with evaluation calculations. Multiple of the following arguments can be given in the same call

--first

Display the wasted effort to the first fault

Default: []

--avg, --average

Display the wasted effort to the average fault

--med, --median

Display the wasted effort to the median fault

--last

Display the wasted effort to the last fault

--weffort

Display the wasted effort to the Nth fault

--one-top

Display the expected value of finding at least one fault in the top x elements (elements with the highest suspiciousness).

Default: []

--all-top

Display the expected value of the total number of faults found in the top x elements (elements with the highest suspiciousness)

--perc-top

Display the expected value of the percentage of faults found in the top x elements (elements with the highest suspiciousness)

--one-top1

(Deprecated: Use –one-top instead) Display the expected value of finding at least one fault in the first element in the ranking.

--all-top1

(Deprecated: Use –all-top instead) Display the expected value of the total number of faults found in the first element in the ranking

--perc-top1

(Deprecated: Use –perc-top instead) Display the percentage of faults found in the first element in the ranking

--perc@n, --percent-at-n

Produces the percentage-at-N values (i.e. the percentage of faults found at N% of code inspected). The output of the perc@n calculation is a list of ranks of all found faults, preceded by the number of elements in the system, which can be used to generate percentage-at-N/recall graphs

Default: []

--auc, --area-under-curve

Dislpays the area under the curve produced by the percentage-at-N calculation

--pauc, --percent-area-under-curve

Dislpays the area under the curve produced by the percentage-at-N calculation as a percentage of the maximum possible value (i.e. closest to perfect recall)

--lauc, --log-area-under-curve

Dislpays the area under the curve produced by the percentage-at-N calculation as a logarithmic percentage of the maximum possible value (i.e. closest to perfect recall). The logarithmic effect causes lower ranks to have a greater effect on the value, which corresponds to the lower ranks being more useful to the developer

--precision-at

Displays precision values at a given rank x. Precision is the amount of faults f found within the cut-off point x, out of the number of elements seen (i.e. f/x). Can be specified multiple times

Default: []

--recall-at

Displays recall values at a given rank x. Recall is the amount of faults f found within the cut-off point x, out of the total number of faults n (i.e. f/n). Can be specified multiple times

Default: []

--fault-num

Display the number of faults in the program

Default: []

--fault-ids

Display the IDs of the faults in the program

--fault-elems

Display the elements that are faulty in the program

--fault-all

Display all info of the faults in the program