Privacy Metrics

class synthyverse.evaluation.privacy.DCR(discrete_features=None, subsample_test_size=True, max_rows=50000, random_state=0)[source]

Bases: object

Distance to Closest Record (DCR) privacy metric.

Measures whether synthetic records are more often closer (by Gower distance) to a training record than an independent test record. Low scores indicate risk that synthetic data overfits to the training set, and therefore privacy risk.

Parameters:

  • discrete_features (list) – List of discrete/categorical feature names. Default: [].

  • subsample_test_size (bool) – Whether to subsample the training set and synthetic set to the test set size. Prevents biasing DCR due to different sample sizes. If used, multiple iterations of the DCR score are computed and aggregated, to ensure the metric is based on all training and synthetic records. Default: True.

  • max_rows (int) – Maximum number of rows to use for computation to limit evaluation time. Default: 50000.

  • random_state (int)

Example

>>> import pandas as pd
>>> from synthyverse.evaluation import DCR
>>>
>>> # Prepare data
>>> X_train = pd.DataFrame(...)
>>> X_test = pd.DataFrame(...)
>>> X_syn = pd.DataFrame(...)
>>> discrete_features = ["category_col"]
>>>
>>> # Create metric
>>> metric = DCR(
...     discrete_features=discrete_features,
...     max_rows=50000,
...     subsample_test_size=True
... )
>>>
>>> # Evaluate
>>> results = metric.evaluate(X_train, X_test, X_syn)
evaluate(train, test, sd)[source]

Evaluate synthetic data privacy using DCR metric.

DCR score is aggregated as min(1, dcr.closer_to_test * 2) to ensure higher scores indicate better privacy.

Parameters:

  • train (DataFrame) – Training data as a pandas DataFrame.

  • test (DataFrame) – Test data as a pandas DataFrame.

  • sd (DataFrame) – Synthetic data as a pandas DataFrame.

Returns:

Dictionary with keys:

  • ”dcr.score”: DCR score

  • ”dcr.closer_to_train”: Proportion closer to train

  • ”dcr.closer_to_test”: Proportion closer to test

Return type:

dict

Raises:

AssertionError – If test set is larger than train set.

class synthyverse.evaluation.privacy.DOMIAS(discrete_features=None, ref_prop=0.5, member_prop=1.0, n_components=0.95, random_state=0, metric='roc_auc', predict_top=0.5)[source]

Bases: object

DOMIAS membership inference attack metric.

Measures vulnerability to membership inference attacks using density-based methods. Lower scores indicate better privacy. Uses KDE on PCA-transformed data for density estimation.

For threshold-based metrics we also include a naive baseline, i.e., all records are predicted as members.

Based on the paper “Membership inference attacks against synthetic data through overfitting detection” by van Breugel et al. (2023).

Parameters:

  • discrete_features (list) – List of discrete/categorical feature names. Default: [].

  • ref_prop (float) – Proportion of test set to use as reference for density estimation. Default: 0.5.

  • member_prop (float) – Proportion of train set to use as members. Default: 1.0.

  • n_components (Union[int, float]) – Number of PCA components. Float in (0,1] = variance target, int = exact components. Default: 0.95.

  • random_state (int) – Random seed for reproducibility. Default: 0.

  • metric (str) – Evaluation metric. One of “roc_auc”, “f1”, “accuracy”, “precision-recall”. Default: “roc_auc”.

  • predict_top (float) – Proportion of top predictions to consider as members. Can be lowered to increase attack precision at the cost of recall. Only relevant for threshold-based metrics, e.g., “precision-recall”. Default: 0.5.

Example

>>> import pandas as pd
>>> from synthyverse.evaluation import DOMIAS
>>>
>>> # Prepare data
>>> X_train = pd.DataFrame(...)
>>> X_test = pd.DataFrame(...)
>>> X_syn = pd.DataFrame(...)
>>> discrete_features = ["category_col"]
>>>
>>> # Create metric
>>> metric = DOMIAS(
...     discrete_features=discrete_features,
...     ref_prop=0.5,
...     n_components=0.95,
...     metric="roc_auc",
...     random_state=42
... )
>>>
>>> # Evaluate
>>> results = metric.evaluate(X_train, X_test, X_syn)
evaluate(train, test, syn)[source]

Evaluate synthetic data privacy using DOMIAS membership inference attack.

Parameters:

  • train (DataFrame) – Training data as a pandas DataFrame.

  • test (DataFrame) – Test data as a pandas DataFrame.

  • syn (DataFrame) – Synthetic data as a pandas DataFrame.

Returns:

Dictionary with DOMIAS metric scores. Keys include metric name with

configuration parameters in the name.

Return type:

dict