detector_benchmark.text_quality_evaluation

Submodules

• detector_benchmark.text_quality_evaluation.scorer
• detector_benchmark.text_quality_evaluation.scorer_utils

Classes

Scorer	Helper class that provides a standard way to create an ABC using
SelfScorer	Helper class that provides a standard way to create an ABC using
RefScorer	Helper class that provides a standard way to create an ABC using
BertScoreScorer	Helper class that provides a standard way to create an ABC using
SemScoreScorer	Helper class that provides a standard way to create an ABC using
IDFScorer	Helper class that provides a standard way to create an ABC using
PrometheusScorer	Helper class that provides a standard way to create an ABC using
PPLScorer	Helper class that provides a standard way to create an ABC using

Package Contents

class detector_benchmark.text_quality_evaluation.Scorer(name)

Bases: abc.ABC

Helper class that provides a standard way to create an ABC using inheritance.

name

abstract score() → float

class detector_benchmark.text_quality_evaluation.SelfScorer(name)

Bases: Scorer

Helper class that provides a standard way to create an ABC using inheritance.

name

abstract score(eval_texts: str) → float

class detector_benchmark.text_quality_evaluation.RefScorer(name)

Bases: Scorer

Helper class that provides a standard way to create an ABC using inheritance.

name

abstract score(eval_texts: str, ref_text: str) → float

class detector_benchmark.text_quality_evaluation.BertScoreScorer(name)

Bases: RefScorer

Helper class that provides a standard way to create an ABC using inheritance.

model = 'microsoft/deberta-xlarge-mnli'

num_layers = 40

score(eval_text: str, ref_text: str) → float

score_batch(eval_texts: list[str], ref_texts: list[str], batch_size) → float

class detector_benchmark.text_quality_evaluation.SemScoreScorer(name)

Bases: RefScorer

Helper class that provides a standard way to create an ABC using inheritance.

tokenizer

model

mean_pooling(model_output, attention_mask)

score(eval_text: str, ref_text: str) → float

class detector_benchmark.text_quality_evaluation.IDFScorer(name, corpus: list[str])

Bases: SelfScorer

Helper class that provides a standard way to create an ABC using inheritance.

corpus

filtered_corpus

word_to_idf

remove_stopwords(corpus: list[str])

score(eval_text: str) → float

score_batch(eval_texts: list[str], batch_size=1) → float

class detector_benchmark.text_quality_evaluation.PrometheusScorer(name, compare_human_to_ai: bool = False)

Bases: CompareScorer

Helper class that provides a standard way to create an ABC using inheritance.

name

model

judge

compare_human_to_ai

score(eval_text1: str, eval_text2: str, ref_text: str | None = None) → float

shuffle_positions(text_list1: list[str], text_list2: list[str]) → tuple[list[str], list[str]]

reassign_scores(scores: list[int], text_list1: list, text_list2: list) → list[int]

score_batch(eval_texts1: list[str], eval_texts2: list[str], ref_texts: list[str], instructions: list[str], rubric: str, batch_size=1) → float

class detector_benchmark.text_quality_evaluation.PPLScorer(name, model, tokenizer)

Bases: SelfScorer

Helper class that provides a standard way to create an ABC using inheritance.

name

scorer_model

scorer_tokenizer

device

metric

score(eval_text: str) → float

score_batch(eval_texts: list[str], batch_size=1, return_loss_lists=False) → float

See https://huggingface.co/spaces/evaluate-measurement/perplexity/blob/main/perplexity.py