Extending flitsr calculations

By default, flitsr provides many of the most commonly used measures and metrics for the evaluation of fault localization techniques. However, if there are any other evaluation metrics you would like to use that are not yet implemented, flitsr provides the functionality for you to easily create your own in the form of calculations.

Calculation format

A custom evaluation metric or calculation is simply a function which takes a Ties object and bool for whether to collapse ambiguity groups, as well as any other custom parameters, and returns any form of output (usually a float). The signature of a calculation function is:

example_calculation(ties: Ties, collapse: bool, ...)

The function must be decorated with the calculation decorator as specified below, which informs flitsr of the new calculation.

flitsr.calculations.calculation(print_name: str | Callable[[...], str], desc: str, arg_name: str, *alt_arg_names: str) → Callable

A decorator for methods that perform calculations to specify that they should be made available as calculations. NOTE: this decorator must be executed after (i.e., must be placed above) any other custom decorators which are intended to be processed when the function is run (this does not include the parameter decorator, which may be placed in any order).

Parameters:

  • print_name – str: The name of the calculation when printing out. Can be a string, or function that takes the calculations parameters and returns a string.

  • desc – str: A breif description of the calculation to show in the help messages

  • arg_name – str: The main command line argument name.

  • alt_arg_names – str: Alternate command line argument names.

An example calculation function using the above format would thus be:

from flitsr.calculations import calculation

@calculation(print_name='custom_calc', desc='My custom calculation',
             'c-calc', 'custom-calculation')
def example_calculation(ties: Ties, collapse: bool, my_param: str):
  ...

The above format would add a new flitsr calculation, which would have command-line options --c-calc and --custom-calculation in the main flitsr application (with the help message "My custom calculation"). These options will take one str argument, corresponding to my_param. When this calculation is performed, the output produced by flitsr will be of the form "custom_calc: <output>".

Creating a calculation plugin

To create a calculation plugin, use the flitsr.calculation entry point, for example:

pyproject.toml
[project.entry-points.'flitsr.calculation']
my_calc = "my-package.custom_calc"

Where custom_calc is a python module containing the calculation function using the calculation decorator.

Calculation parameters

As noted in Calculation format above, a calculation can take any number of other arbitrary parameters, which will be supplied by the command-line option as arguments. flitsr will automatically detect any of these parameters in the calculation function’s signature, as well as any type annotations and defaults given there. These type annotations and defaults will be used to construct the command-line option arguments.

Attention

Due to limitations in the argparse module, defaults given to the function signature will only be used if there is a single argument.

For parameters with simple primitive types (int, str, bool, float, and Path), flitsr will automatically convert the command-line agrument into the required type to be given to the function.

For more complex types or conversions, or for parameters which may only take on a finite set of values, the parameter decorator as given below may be used.

flitsr.calculations.parameter(name: str, type: Callable[[str], Any] | None = None, choices: Collection[Any] | None = None) → Callable

A decorator for functions that perform calculations to specify additional information for a parameter of that calculation. Note: this decorator is only required for parameters where you need to convert the type, or specify choices. Other parameters will be automatically detected in the function signature, including default values.

Parameters:

  • name – The name of the paramter. Must be one of the parameters in the method signature.

  • type – A function to convert the given parameter from a string into the type required. Will be used by the command-line parser.

  • choices – A Collection of values which the given parameter can take on.

Using this decorator it is possible (using type) to supply a custom function which takes the string argument which would be given on the command-line, and return the argument converted to be usable in your calculation. You may also specify (using choices), a collection of pre-defined choices which are valid for the given parameter (cf. flitsr.advanced.attributes.choices).

Note

As specified in the argparse documentation, choices are checked after any type conversions have been performed, so objects given by choices should match the type specified, as the function given by type will not be run on them.

Using the example given in Calculation format above, a example use of the parameter decorator to specify pre-defined choices would be:

from flitsr.calculations import calculation, parameter

@parameter('my_param', choices=['choice1', 'choice2'])
@calculation(print_name='custom_calc', desc='My custom calculation',
             'c-calc', 'custom-calculation')
def example_calculation(ties: Ties, collapse: bool, my_param: str):
  ...