nebula.addons.trustworthiness.calculation

nebula.addons.trustworthiness.calculation#

Attributes#

dirname

logger

R_L1

R_L2

R_LI

Functions#

get_mapped_score(score_key, score_map)

Finds the score by the score_key in the score_map.

get_normalized_scores(scores)

Calculates the normalized scores of a list.

get_range_score(value, ranges[, direction])

Maps the value to a range and gets the score by the range and direction.

get_map_value_score(score_key, score_map)

Finds the score by the score_key in the score_map and returns the value.

get_true_score(value, direction)

Returns the negative of the value if direction is 'desc', otherwise returns value.

get_scaled_score(value, scale, direction)

Maps a score of a specific scale into the scale between zero and one.

get_value(value)

Get the value of a metric.

check_properties(*args)

Check if all the arguments have values.

get_cv([list, std, mean])

Get the coefficient of variation.

get_global_privacy_risk(dp, epsilon, n)

Calculates the global privacy risk by epsilon and the number of clients.

get_elapsed_time(scenario)

Calculates the elapsed time during the execution of the scenario.

get_bytes_models(models_files)

Calculates the mean bytes of the final models of the nodes.

get_bytes_sent_recv(bytes_sent_files, bytes_recv_files)

Calculates the mean bytes sent and received of the nodes.

get_avg_loss_accuracy(loss_files, accuracy_files)

Calculates the mean accuracy and loss models of the nodes.

get_feature_importance_cv(model, test_sample)

Calculates the coefficient of variation of the feature importance.

get_clever_score(model, test_sample, nb_classes, ...)

Calculates the CLEVER score.

stop_emissions_tracking_and_save(tracker, outdir, ...)

Stops emissions tracking object from CodeCarbon and saves relevant information to emissions.csv file.

Module Contents#

nebula.addons.trustworthiness.calculation.dirname#
nebula.addons.trustworthiness.calculation.logger#
nebula.addons.trustworthiness.calculation.R_L1 = 40#
nebula.addons.trustworthiness.calculation.R_L2 = 2#
nebula.addons.trustworthiness.calculation.R_LI = 0.1#
nebula.addons.trustworthiness.calculation.get_mapped_score(score_key, score_map)#

Finds the score by the score_key in the score_map.

Parameters:

  • score_key (string) – The key to look up in the score_map.

  • score_map (dict) – The score map defined in the eval_metrics.json file.

Returns:

The normalized score of [0, 1].

Return type:

float

nebula.addons.trustworthiness.calculation.get_normalized_scores(scores)#

Calculates the normalized scores of a list.

Parameters:

scores (list) – The values that will be normalized.

Returns:

The normalized list.

Return type:

list

nebula.addons.trustworthiness.calculation.get_range_score(value, ranges, direction='asc')#

Maps the value to a range and gets the score by the range and direction.

Parameters:

  • value (int) – The input score.

  • ranges (list) – The ranges defined.

  • direction (string) – Asc means the higher the range the higher the score, desc means otherwise.

Returns:

The normalized score of [0, 1].

Return type:

float

nebula.addons.trustworthiness.calculation.get_map_value_score(score_key, score_map)#

Finds the score by the score_key in the score_map and returns the value.

Parameters:

  • score_key (string) – The key to look up in the score_map.

  • score_map (dict) – The score map defined in the eval_metrics.json file.

Returns:

The score obtained in the score_map.

Return type:

float

nebula.addons.trustworthiness.calculation.get_true_score(value, direction)#

Returns the negative of the value if direction is ‘desc’, otherwise returns value.

Parameters:

  • value (int) – The input score.

  • direction (string) – Asc means the higher the range the higher the score, desc means otherwise.

Returns:

The score obtained.

Return type:

float

nebula.addons.trustworthiness.calculation.get_scaled_score(value, scale, direction)#

Maps a score of a specific scale into the scale between zero and one.

Parameters:

  • value (int or float) – The raw value of the metric.

  • scale (list) – List containing the minimum and maximum value the value can fall in between.

  • direction (str)

Returns:

The normalized score of [0, 1].

Return type:

float

nebula.addons.trustworthiness.calculation.get_value(value)#

Get the value of a metric.

Parameters:

value (float) – The value of the metric.

Returns:

The value of the metric.

Return type:

float

nebula.addons.trustworthiness.calculation.check_properties(*args)#

Check if all the arguments have values.

Parameters:

args (list) – All the arguments.

Returns:

The mean of arguments that have values.

Return type:

float

nebula.addons.trustworthiness.calculation.get_cv(list=None, std=None, mean=None)#

Get the coefficient of variation.

Parameters:

  • list (list) – List in which the coefficient of variation will be calculated.

  • std (float) – Standard deviation of a list.

  • mean (float) – Mean of a list.

Returns:

The coefficient of variation calculated.

Return type:

float

nebula.addons.trustworthiness.calculation.get_global_privacy_risk(dp, epsilon, n)#

Calculates the global privacy risk by epsilon and the number of clients.

Parameters:

  • dp (bool) – Indicates if differential privacy is used or not.

  • epsilon (int) – The epsilon value.

  • n (int) – The number of clients in the scenario.

Returns:

The global privacy risk.

Return type:

float

nebula.addons.trustworthiness.calculation.get_elapsed_time(scenario)#

Calculates the elapsed time during the execution of the scenario.

Parameters:

scenario (object) – Scenario required.

Returns:

The elapsed time.

Return type:

float

nebula.addons.trustworthiness.calculation.get_bytes_models(models_files)#

Calculates the mean bytes of the final models of the nodes.

Parameters:

models_files (list) – List of final models.

Returns:

The mean bytes of the models.

Return type:

float

nebula.addons.trustworthiness.calculation.get_bytes_sent_recv(bytes_sent_files, bytes_recv_files)#

Calculates the mean bytes sent and received of the nodes.

Parameters:

  • bytes_sent_files (list) – Files that contain the bytes sent of the nodes.

  • bytes_recv_files (list) – Files that contain the bytes received of the nodes.

Returns:

The total bytes sent, the total bytes received, the mean bytes sent and the mean bytes received of the nodes.

Return type:

4-tupla

nebula.addons.trustworthiness.calculation.get_avg_loss_accuracy(loss_files, accuracy_files)#

Calculates the mean accuracy and loss models of the nodes.

Parameters:

  • loss_files (list) – Files that contain the loss of the models of the nodes.

  • accuracy_files (list) – Files that contain the acurracies of the models of the nodes.

Returns:

The mean loss of the models, the mean accuracies of the models, the standard deviation of the accuracies of the models.

Return type:

3-tupla

nebula.addons.trustworthiness.calculation.get_feature_importance_cv(model, test_sample)#

Calculates the coefficient of variation of the feature importance.

Parameters:

  • model (object) – The model.

  • test_sample (object) – One test sample to calculate the feature importance.

Returns:

The coefficient of variation of the feature importance.

Return type:

float

nebula.addons.trustworthiness.calculation.get_clever_score(model, test_sample, nb_classes, learning_rate)#

Calculates the CLEVER score.

Parameters:

  • model (object) – The model.

  • test_sample (object) – One test sample to calculate the CLEVER score.

  • nb_classes (int) – The nb_classes of the model.

  • learning_rate (float) – The learning rate of the model.

Returns:

The CLEVER score.

Return type:

float

nebula.addons.trustworthiness.calculation.stop_emissions_tracking_and_save(tracker, outdir, emissions_file, role, workload, sample_size=0)#

Stops emissions tracking object from CodeCarbon and saves relevant information to emissions.csv file.

Parameters:

  • tracker (object) – The emissions tracker object holding information.

  • outdir (str) – The path of the output directory of the experiment.

  • emissions_file (str) – The path to the emissions file.

  • role (str) – Either client or server depending on the role.

  • workload (str) – Either aggregation or training depending on the workload.

  • sample_size (int) – The number of samples used for training, if aggregation 0.

Contents