detector_benchmark.pipeline¶

Submodules¶

Classes¶

`CreateDatasetPipeline`	Abstract class for an experiment pipeline.
`ExperimentTestDetectorPipeline`	Abstract class for an experiment pipeline.
`TextQualityPipeline`	Pipeline to evaluate the quality of text.

Package Contents¶

class detector_benchmark.pipeline.CreateDatasetPipeline(cfg: dict, dataset_loader: detector_benchmark.dataset_loader.FakeTruePairsDataLoader, attack: detector_benchmark.generation.ArticleGenerator, experiment_path: str, batch_size: int = 1, skip_cache: bool = False, skip_train_split: bool = False)¶

Bases: detector_benchmark.pipeline.experiment_pipeline.ExperimentPipeline

Abstract class for an experiment pipeline.

cfg¶

dataset_loader¶

attack¶

experiment_path¶

batch_size¶

generator_name¶

experiment_name¶

skip_cache¶

skip_train_split¶

log¶

create_experiment_dataset() → datasets.Dataset¶

Create the fake true dataset for the experiment by generating fake articles using the generator.

Returns:¶

dataset: Dataset
The generated fake true dataset.

run_pipeline()¶: Main function of the class, runs the pipeline. Creates the dataset for the experiment and saves it.

class detector_benchmark.pipeline.ExperimentTestDetectorPipeline(cfg: dict, detector: detector_benchmark.detector.Detector, experiment_path: str, non_attack_experiment_path: str, dataset_experiment_path: str, batch_size: int = 1)¶

Bases: detector_benchmark.pipeline.experiment_pipeline.ExperimentPipeline

Abstract class for an experiment pipeline.

cfg¶

experiment_path¶

non_attack_experiment_path¶

batch_size¶

detector_name¶

dataset_experiment_path¶

dataset_experiment_name¶

experiment_name¶

target_fpr¶

log¶

detector¶

find_threshold(eval_set: datasets.Dataset, target_fpr: float) → float¶

Find the detection threshold for the target false positive rate (FPR) on the evaluation set. Uses the fpr and thresholds obtained with the roc_curve function from sklearn.

Parameters:¶

eval_set: Dataset
The evaluation set to use for finding the threshold.

target_fpr: float
The target false positive rate to find the threshold for.

Returns:

threshold: float: The detection threshold for the target FPR.

evaluate_detector(preds: list[int], logits: list[float], preds_at_threshold: list[int], labels: list[int], dataset: datasets.Dataset, detection_threshold: float, data_split: str = 'test')¶

Use the predictions and labels to compute the metrics of the detector and save them.

Parameters:¶

preds: list[int]
The predictions of the detector.

logits: list[float]
The logits of the detector.

preds_at_threshold: list[int]
The predictions of the detector at the given threshold.

labels: list[int]
The true labels of the dataset.

dataset: Dataset
The dataset used for testing the detector.

detection_threshold: float
The detection threshold used for the predictions at the given threshold.

data_split: str
The data split used for testing the detector. Default is “test”.

run_pipeline()¶: Main function of the pipeline, runs the pipeline. First, find the detection threshold for the target FPR on the evaluation set. Then, test the detector on the test set using the detection threshold and save the results.

class detector_benchmark.pipeline.TextQualityPipeline(scorer: detector_benchmark.text_quality_evaluation.Scorer, dataset_path: str, dataset_path_compare: str | None = None, batch_size: int = 64, return_loss_lists: bool = False, eval_human: bool = False)¶

Bases: detector_benchmark.pipeline.experiment_pipeline.ExperimentPipeline

Pipeline to evaluate the quality of text.

scorer¶

dataset¶

batch_size¶

return_loss_lists¶

eval_human¶

run_pipeline()¶