How to generate the benchmarkĀ¶

cf. bash_scripts/generating_datasets/generate_no_watermark_dataset.sh (see other scripts in bash_scripts/generating_datasets for other examples)

How to generate the benchmark without watermarkingĀ¶

attack="generation_base"

generator_name="llama3_instruct_3.1"
prefix_size=10
dataset_size=5000
batch_size=128
experiment_name="test"
data_folder="data/generated_datasets"
skip_train_split=True
max_sample_len=100
max_new_tokens=50
min_new_tokens=40
temperature=0.8
top_p=0.95
repetition_penalty=1
do_sample=True
top_k=50

create_dataset generation=$attack watermark=$watermark_scheme generation.generator_name=$generator \
        generation.dataset_size=$dataset_size generation.experiment_name=$experiment_name \
        generation.prefix_size=$prefix_size \
        generation.skip_train_split=$skip_train_split generation.skip_cache=$skip_cache \
        generation.batch_size=$batch_size generation.data_folder=$data_folder \
        generation.max_sample_len=$max_sample_len generation.max_new_tokens=$max_new_tokens generation.min_new_tokens=$min_new_tokens \
        generation.temperature=$temperature generation.top_p=$top_p generation.repetition_penalty=$repetition_penalty \
        generation.do_sample=$do_sample generation.top_k=$top_k

Configure the benchmark

This section outlines the parameters used for generating datasets with the specified configurations. See the configuration files under conf/generation for the complete list. Note that all default values assume that attack=ā€generation_baseā€ which means that no attack is used.

  • attack: Sets the base hydra configuration file for the generation used to generate the fake samples (base parameters). Here, ā€œgeneration_baseā€ corresponds to the base file which means no attack is used. This will determine the default parameter values. See here for the list of supported attacks and here to add yours. Default value: ā€œgeneration_baseā€

  • batch_size: Specifies the number of samples to be generated in parallel by the GPU. Default value: 2

  • data_folder: The directory where the generated datasets will be stored. Here, it is set to ā€œdata/generated_datasetsā€. Default value: ā€œdata/generated_datasetsā€

  • dataset_name: Base dataset used for the true samples and the prefixes Default value: cnn_dailymail

  • dataset_size: Sets the total number of samples to be generated. This includes the train, eval and test split (80/10/10 split). This means that for dataset_size=5000, there will be a test split of size 500. Default value: 100

  • do_sample: Whether to use top_p sampling or greedy decoding. Default value: True

  • experiment_name: A string that names the experiment. In this case, it is set to ā€œtestā€, which can be useful for tracking results. Default value: base

  • generator_name: Specifies the name of the generator to be used, in this case, ā€œllama3_instruct_3.1ā€. This determines the model that will generate the text. See here for the list of supported generators and here to add yours. Default value: qwen2_chat_0_5B

  • max_new_tokens: The maximum number of new tokens to generate. Default value: 220

  • max_sample_len: The maximum length of each of the fake/true samples in number of characters. All samples fake or true are cut to this value. Default value: 500

  • min_new_tokens: The minimum number of new tokens to generate, set to 40. This ensures that each generated sample has a minimum length. Default value: 200

  • prefix_size: Number of first words to take from the true samples that will be forced into the fake samples to start the generation. Default value: 10

  • repetition_penalty=1: Repetition penalty value used for the generation. Default value: 1

  • skip_train_split: A boolean parameter that, when set to True, indicates that the training split of the dataset should be skipped. It will still create a train split, but the fake samples will be empty. Note that there is an eval split used for finding the correct threshold for a given target FPR and the test split used to test the detector on that threshold. The train split could be used to train a detector on that dataset. Default value: False

  • temperature: Temperature used for the generation. Default value: 0.8

  • top_k: top k value used for the generation. Default value: 50

  • top_p: top p value used for the generation. Default value: 0.95

How to generate the benchmark with watermarkingĀ¶

attack="generation_base"

generator_name="llama3_instruct_3.1"
watermark_scheme="watermark_base"
prefix_size=10
dataset_size=5000
batch_size=128
experiment_name="test"
data_folder="data/generated_datasets"
skip_train_split=True
max_sample_len=100
max_new_tokens=50
min_new_tokens=40
temperature=0.8
top_p=0.95
repetition_penalty=1
do_sample=True
top_k=50

python create_dataset.py generation=$attack watermark=$watermark_scheme generation.generator_name=$generator \
        generation.dataset_size=$dataset_size generation.experiment_name=$experiment_name \
        generation.prefix_size=$prefix_size \
        generation.skip_train_split=$skip_train_split generation.skip_cache=$skip_cache \
        generation.batch_size=$batch_size generation.data_folder=$data_folder \
        generation.max_sample_len=$max_sample_len generation.max_new_tokens=$max_new_tokens generation.min_new_tokens=$min_new_tokens \
        generation.temperature=$temperature generation.top_p=$top_p generation.repetition_penalty=$repetition_penalty \
        generation.do_sample=$do_sample generation.top_k=$top_k

Configure the benchmark

The parameters here are the same as above except for the following:

  • watermark_scheme: Indicates the hydra config file for the watermarking scheme to be applied when generating the text. ā€œwatermark_baseā€ means no watermark is used (base generation). See here for the full list of watermarking schemes. Default value: ā€œwatermark_baseā€.

Configure the watermark:Ā¶

  • TODO

Extra parameters depending on the attack:Ā¶

  • TODO