Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 27 additions & 8 deletions docs/api_reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Domain Specification (``dpsynth.domain``)
The ``domain`` module provides dataclasses for describing the schema of a
tabular dataset. Each column is represented by one of the attribute types
below. Pass a mapping of column names to attribute objects as the ``domains``
argument to :class:`~dpsynth.TabularSynthesizer`.
argument to :class:`~dpsynth.TabularConfig`.

.. autosummary::
:toctree: _autosummary
Expand All @@ -59,7 +59,7 @@ Cross-Attribute Constraints (``dpsynth.constraints``)
The ``constraints`` module lets you express known relationships between columns
so that the synthetic data honours them. Pass a list of
:class:`~dpsynth.constraints.Constraint` objects as
``cross_attribute_constraints`` to :class:`~dpsynth.TabularSynthesizer`.
``cross_attribute_constraints`` to :class:`~dpsynth.TabularConfig`.

.. autosummary::
:toctree: _autosummary
Expand Down Expand Up @@ -88,21 +88,38 @@ protocol shared by all DPSynth mechanisms.

----

Tabular Synthesis (``dpsynth``)
===============================

.. currentmodule:: dpsynth

The primary entry point for generating differentially private synthetic data from standard tabular datasets (such as Pandas DataFrames).

.. autosummary::
:toctree: _autosummary
:nosignatures:
:template: autosummary/class.rst

TabularConfig
TabularMechanism

----

Discrete Mechanisms (``dpsynth.discrete_mechanisms``)
======================================================

.. currentmodule:: dpsynth.discrete_mechanisms

Discrete mechanisms operate on pre-discretized integer datasets
(:class:`mbi.Dataset`). :class:`TabularSynthesizer` applies them internally
(:class:`mbi.Dataset`). :class:`~dpsynth.TabularConfig` applies them internally
after encoding your DataFrame. Use them directly only if you already have a
discrete dataset.

Mechanism Configs
-----------------

Each config class corresponds to a published DP synthesis algorithm. Pass one
as the ``discrete_mechanism`` argument to :class:`~dpsynth.TabularSynthesizer`,
as the ``discrete_mechanism`` argument to :class:`~dpsynth.TabularConfig`,
or use :class:`~dpsynth.discrete_mechanisms.DiscreteConfig` to add one-way
marginal measurement and domain compression.

Expand All @@ -118,16 +135,18 @@ marginal measurement and domain compression.
SWIFTConfig
AIMGDPConfig

DiscreteConfig
--------------
DiscreteConfig and DiscreteMechanism
------------------------------------

:class:`DiscreteConfig` wraps any of the mechanism configs above with one-way
marginal pre-measurement and optional domain compression. It is the recommended
entry point when you have a pre-discretized table.
marginal pre-measurement and optional domain compression. When calibrated, it
produces a runnable :class:`DiscreteMechanism`. This is the recommended entry
point when you have a pre-discretized table.

.. autosummary::
:toctree: _autosummary
:nosignatures:
:template: autosummary/class.rst

DiscreteConfig
DiscreteMechanism
132 changes: 82 additions & 50 deletions docs/in_memory_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@ within a single machine's RAM.

--------------------------------------------------------------------------------

## Python API: `dpsynth.TabularSynthesizer`
## Python API: `dpsynth.TabularConfig`

The primary entry point for in-memory synthesis is
`dpsynth.TabularSynthesizer`. It accepts a dictionary of attribute domains,
is calibrated with a privacy budget, and generates a fully synthetic,
differentially private DataFrame matching the exact schema and data types of
your input.
`dpsynth.TabularConfig`. It accepts a dictionary of attribute domains and
mechanism options, is calibrated with a privacy budget to produce a
`dpsynth.TabularMechanism`, and generates a fully synthetic, differentially
private DataFrame matching the exact schema and data types of your input.

### Usage

Expand All @@ -27,43 +27,45 @@ from dpsynth import discrete_mechanisms
import numpy as np
import pandas as pd

synth = dpsynth.TabularSynthesizer(
config = dpsynth.TabularConfig(
domains=domains,
discrete_mechanism=discrete_mechanisms.MSTConfig(),
)
result = synth.calibrate(
epsilon=1.0,
delta=1e-6,
)(np.random.default_rng(), sensitive_df)
mechanism = config.calibrate(epsilon=1.0, delta=1e-6)
result = mechanism(np.random.default_rng(), sensitive_df)
synthetic_df = result.synthetic_data
```

### Key Arguments
### Key Configuration Arguments

When initializing `dpsynth.TabularConfig`:

* `data`: The sensitive input `pd.DataFrame`.
* `domains`: Mapping of column names to domain specifications
([`CategoricalAttribute`, `NumericalAttribute`, or `OpenSetCategoricalAttribute`](data_and_terminology.md)).
Every key must exist in `data.columns`.
* `epsilon`, `delta`: Total differential privacy budget parameters.
* `discrete_mechanism`: Configuration object specifying which DP synthesis
mechanism to run (e.g., `MSTConfig()`, `AIMConfig()`,
`IndependentConfig()`).
* `numerical_bins`: Number of equal-frequency quantile buckets used to
discretize continuous numerical columns (default: `32`).
* `one_way_marginal_budget_fraction`: Fraction of total `(epsilon, delta)`
allocated for one-way marginal measurements and domain compression (default:
`0.1`).
* `skip_compression`: If `True`, bypasses the rare-category merging phase.
Note: Compression cannot currently be used simultaneously with
cross-attribute constraints.
* `init_budget_fraction`: Fraction of total `(epsilon, delta)` budget
allocated for per-column initialization such as bounds computation and
partition selection (default: `0.1`).
* `cross_attribute_constraints`: Optional sequence of constraints to enforce
on generated data.

When calling `config.calibrate(...)`:

* `epsilon`, `delta`: Total differential privacy budget parameters. Returns a
runnable `TabularMechanism`.

--------------------------------------------------------------------------------

## End-to-End Python Example
## Standalone End-to-End Python Example

Here is a complete Python script demonstrating how to load data, parse a domain
YAML file, configure the AIM mechanism, set a fixed random seed, and generate
synthetic records.
Here is a complete, self-contained Python script demonstrating how to specify a
domain, set up a `TabularConfig`, calibrate the mechanism with a privacy budget,
load sensitive data, synthesize records, and print the first few rows.

```python
import dpsynth
Expand All @@ -72,34 +74,64 @@ from dpsynth import domain
import numpy as np
import pandas as pd

# 1. Load sensitive tabular data into Pandas
sensitive_df = pd.read_csv("sensitive_transactions.csv")

# 2. Load domain schema from YAML
attribute_domains = domain.from_yaml_file("transaction_domain.yaml")
# 1. Domain Specification: Define the schema of the tabular dataset
attribute_domains = {
"age": domain.NumericalAttribute(lower_bound=18, upper_bound=90),
"workclass": domain.CategoricalAttribute(
allowed_values=["Private", "Self-emp", "Gov", "Other"]
),
"education": domain.CategoricalAttribute(
allowed_values=["HS-grad", "Bachelors", "Masters", "PhD"]
),
}

# 3. Configure and calibrate the synthesizer (AIM)
synth = dpsynth.TabularSynthesizer(
# 2. Setup Config: Configure synthesizer with domain and mechanism choices
config = dpsynth.TabularConfig(
domains=attribute_domains,
discrete_mechanism=discrete_mechanisms.AIMConfig(
max_rounds=50,
pgm_iters=1000,
),
)
calibrated = synth.calibrate(
epsilon=1.0,
delta=1e-6,
numerical_bins=16, # Use 16 quantile buckets for numerical columns
discrete_mechanism=discrete_mechanisms.MSTConfig(),
numerical_bins=16,
)

# 4. Generate Differentially Private synthetic data
seed = 42
result = calibrated(np.random.default_rng(seed), sensitive_df)
# 3. Calibrate Mechanism: Allocate privacy budget to get runnable mechanism
mechanism = config.calibrate(epsilon=1.0, delta=1e-5)

# 4. Load Data: Create sensitive input DataFrame matching the domain schema
sensitive_df = pd.DataFrame({
"age": [25, 42, 30, 55, 62, 29, 38, 47, 51, 33],
"workclass": [
"Private",
"Gov",
"Private",
"Self-emp",
"Other",
"Private",
"Gov",
"Private",
"Self-emp",
"Private",
],
"education": [
"Bachelors",
"Masters",
"HS-grad",
"PhD",
"HS-grad",
"Bachelors",
"HS-grad",
"Masters",
"Bachelors",
"HS-grad",
],
})

# 5. Synthesize Data: Run the calibrated mechanism on the sensitive data
rng = np.random.default_rng(seed=42)
result = mechanism(rng, sensitive_df)
synthetic_df = result.synthetic_data

# 5. Save the synthetic dataframe
synthetic_df.to_csv("synthetic_transactions.csv", index=False)
print("Synthetic data successfully generated!")
# 6. Print the first few rows of the generated synthetic dataset
print("Generated Synthetic Data:")
print(synthetic_df.head())
```

--------------------------------------------------------------------------------
Expand Down Expand Up @@ -141,7 +173,7 @@ python3 bin/main.py \

## Under the Hood: The In-Memory Lifecycle

When you invoke `TabularSynthesizer`, the library performs the following
When you configure and run `TabularConfig`, the library performs the following
single-machine pipeline:

1. **Discretization**: Continuous numerical columns are bucketed into
Expand All @@ -152,11 +184,11 @@ single-machine pipeline:
3. **Domain Compression**: DPSynth measures 1-way marginals with Gaussian noise
and merges rare categories into an `"Other"` bucket, producing an un-noised
discrete dataset (`mbi.Dataset`).
4. **Mechanism Execution**: Calls `discrete_mechanisms.run_mechanism()` to
execute the selected algorithm (`AIM`, `MST`, etc.) on the discrete dataset.
The mechanism fits a Markov Random Field (`mbi.MarkovRandomField`) via
Private-PGM mirror descent.
4. **Mechanism Execution**: Calls the configured discrete mechanism (`AIM`,
`MST`, etc.) on the discrete dataset. The mechanism fits a Markov Random
Field (`mbi.MarkovRandomField`) via Private-PGM mirror descent.
5. **Sampling & Inversion**: Samples synthetic integer records from the
graphical model, unpacks `"Other"` categories, and inverts the integer
encoding back to original Pandas dtypes (strings, integers, floating
points).

29 changes: 28 additions & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ dataframes to massive distributed datasets across computing clusters:
└────────────────────────────────────────┘
```

### 1. In-Memory DataFrame API (`dpsynth.TabularSynthesizer`)
### 1. In-Memory DataFrame API (`dpsynth.TabularConfig`)

Optimized for rapid prototyping, research experimentation, and datasets that
easily fit within single-machine memory.
Expand Down Expand Up @@ -128,3 +128,30 @@ APIs:
## Contact

--------------------------------------------------------------------------------

```{toctree}
:maxdepth: 2
:caption: Getting Started
:hidden:

data_and_terminology
in_memory_api
scalable_pipeline_api
```

```{toctree}
:maxdepth: 2
:caption: Guides
:hidden:

processing_lifecycle
contributors_guide
```

```{toctree}
:maxdepth: 2
:caption: API Reference
:hidden:

api_reference
```
40 changes: 0 additions & 40 deletions docs/index.rst

This file was deleted.

4 changes: 2 additions & 2 deletions docs/sitemap.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

* [Why DPSynth?](index.md#why-dpsynth)
* [Core APIs and Execution Models](index.md#core-apis-and-execution-models)
* [1. In-Memory DataFrame API (`dpsynth.TabularSynthesizer`)](index.md#1-in-memory-dataframe-api-dpsynth-tabularsynthesizer)
* [1. In-Memory DataFrame API (`dpsynth.TabularConfig`)](index.md#1-in-memory-dataframe-api-dpsynth-tabularconfig)
* [2. Scalable PipelineBackend API (`dpsynth.data_generation`)](index.md#2-scalable-pipelinebackend-api-dpsynthdata_generation)
* [Documentation Sitemap & Navigation](index.md#documentation-sitemap--navigation)
* [Supported Synthesis Algorithms](index.md#supported-synthesis-algorithms)
Expand Down Expand Up @@ -46,7 +46,7 @@
<details>
<summary>📁 <a href="in_memory_api.md">In-Memory DataFrame API Guide</a></summary>

* [Python API: `dpsynth.TabularSynthesizer`](in_memory_api.md#python-api-dpsynth-tabularsynthesizer)
* [Python API: `dpsynth.TabularConfig`](in_memory_api.md#python-api-dpsynth-tabularconfig)
* [Function Signature](in_memory_api.md#function-signature)
* [Key Arguments](in_memory_api.md#key-arguments)
* [End-to-End Python Example](in_memory_api.md#end-to-end-python-example)
Expand Down
2 changes: 2 additions & 0 deletions dpsynth/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@
from dpsynth.data_generation_v3 import TabularConfig
from dpsynth.data_generation_v3 import TabularMechanism
from dpsynth.data_generation_v3 import TabularSynthesizer
from dpsynth.discrete_mechanisms.discrete import DiscreteConfig
from dpsynth.discrete_mechanisms.discrete import DiscreteMechanism
from dpsynth.domain import CategoricalAttribute
from dpsynth.domain import FreeFormTextAttribute
from dpsynth.domain import NumericalAttribute
Expand Down
Loading