# Diff Gnome

Diff Gnome is a CLI-first tool for verifying that two datasets actually match.

It is built for real cross-database verification and repair workflows: compare two tables, dry-run or apply syncs, preflight risky jobs with doctor, clean duplicate rows with dedupe, restore backup exports, and batch everything through config files.

Diff Gnome supports:

- MySQL
- PostgreSQL
- SQL Server
- text output
- JSON output
- Markdown output
- schema comparison
- row counts
- chunked row comparison
- recursive drill-down for mismatched aligned chunks
- sample missing, extra, and changed rows
- apply/dry-run sync workflows
- doctor preflight checks
- dedupe and restore workflows
- batch run configs

## Usage

```bash
./diff-gnome compare \
  --left-driver mysql \
  --left-dsn 'user:pass@tcp(localhost:3306)/appdb' \
  --right-driver postgres \
  --right-dsn 'postgres://user:pass@localhost:5432/warehouse?sslmode=disable' \
  --table customer_orders \
  --key id
```

Activate a license on this machine:

```bash
./diff-gnome license activate \
  --key YOUR-LICENSE-KEY
```

Check the stored license and force a live entitlement refresh:

```bash
./diff-gnome license status --check
```

Deactivate the local machine instance:

```bash
./diff-gnome license deactivate
```

Open the purchase page in your default browser:

```bash
./diff-gnome license purchase
```

JSON output:

```bash
./diff-gnome compare \
  --left-driver mysql \
  --left-dsn 'user:pass@tcp(localhost:3306)/appdb' \
  --right-driver postgres \
  --right-dsn 'postgres://user:pass@localhost:5432/warehouse?sslmode=disable' \
  --table customer_orders \
  --key id \
  --output json
```

Fail the command when any schema, count, or row differences are found:

```bash
./diff-gnome compare \
  --left-driver mysql \
  --left-dsn 'user:pass@tcp(localhost:3306)/appdb' \
  --right-driver postgres \
  --right-dsn 'postgres://user:pass@localhost:5432/warehouse?sslmode=disable' \
  --table customer_orders \
  --key id \
  --fail-on-diff
```

Export every mismatch to a JSONL file while keeping the main report compact:

```bash
./diff-gnome compare \
  --left-driver mysql \
  --left-dsn 'user:pass@tcp(localhost:3306)/appdb' \
  --right-driver sqlserver \
  --right-dsn 'sqlserver://user:pass@sqlhost:1433?database=appdb&encrypt=disable' \
  --table customer_orders \
  --key id \
  --mismatch-file reports/customer-orders-mismatches.jsonl
```

Optional filter:

```bash
./diff-gnome compare \
  --left-driver mysql \
  --left-dsn 'user:pass@tcp(localhost:3306)/appdb' \
  --right-driver postgres \
  --right-dsn 'postgres://user:pass@localhost:5432/warehouse?sslmode=disable' \
  --table customer_orders \
  --key id \
  --where "created_at >= '2026-01-01'"
```

SQL Server example:

```bash
./diff-gnome compare \
  --left-driver sqlserver \
  --left-dsn 'sqlserver://user:pass@sqlhost:1433?database=appdb&encrypt=disable' \
  --right-driver mysql \
  --right-dsn 'user:pass@tcp(localhost:3306)/appdb' \
  --table customer_orders \
  --key id
```

Using environment variables instead of inline credentials:

```bash
export DIFFGNOME_LEFT_DSN='user:pass@tcp(source-host:3306)/appdb'
export DIFFGNOME_RIGHT_DSN='postgres://user:pass@localhost:5432/targetdb?sslmode=disable'

./diff-gnome compare \
  --left-driver mysql \
  --right-driver postgres \
  --table customer_orders \
  --key id
```

Or point at custom env var names:

```bash
export SOURCE_DB_DSN='user:pass@tcp(source-host:3306)/appdb'
export SOURCE_DB_DRIVER='mysql'

./diff-gnome compare \
  --left-driver-env SOURCE_DB_DRIVER \
  --left-dsn-env SOURCE_DB_DSN \
  --right-driver-env TARGET_DB_DRIVER \
  --right-dsn-env TARGET_DB_DSN \
  --table customer_orders \
  --key id
```

Composite key example:

```bash
./diff-gnome compare \
  --left-driver mysql \
  --left-dsn 'user:pass@tcp(localhost:3306)/appdb' \
  --right-driver sqlserver \
  --right-dsn 'sqlserver://user:pass@sqlhost:1433?database=appdb&encrypt=disable' \
  --table customer_order_lines \
  --key order_id,line_number
```

Different table names on each side:

```bash
./diff-gnome compare \
  --left-driver mysql \
  --left-dsn 'user:pass@tcp(localhost:3306)/appdb' \
  --right-driver sqlserver \
  --right-dsn 'sqlserver://user:pass@sqlhost:1433?database=appdb&encrypt=disable' \
  --left-table customer_orders_archive \
  --right-table customer_orders \
  --key id
```

Keyless compare example:

```bash
./diff-gnome compare \
  --left-driver mysql \
  --left-dsn 'user:pass@tcp(left-host:3306)/appdb' \
  --right-driver mysql \
  --right-dsn 'user:pass@tcp(right-host:3306)/appdb' \
  --table event_journal
```

Ignore precision-only timestamp differences:

```bash
./diff-gnome compare \
  --left-driver mysql \
  --left-dsn 'user:pass@tcp(left-host:3306)/appdb' \
  --right-driver sqlserver \
  --right-dsn 'sqlserver://user:pass@right-host:1433?database=appdb&encrypt=disable' \
  --table customer_orders \
  --ignore-column changedon
```

Normalize date/time columns to the calendar date:

```bash
./diff-gnome compare \
  --left-table reporting.activity_log_archive \
  --right-table activity_log \
  --ignore-column changedon \
  --date-only-column changedate
```

Treat empty strings and nulls as equivalent during matching:

```bash
./diff-gnome compare \
  --table customer_orders \
  --key id \
  --empty-string-as-null
```

Treat transformed display values as equivalent for one column:

```bash
./diff-gnome compare \
  --table promo_rules \
  --key docid \
  --equivalent-value 'applydiscount:1=Yes|0='
```

Share one equivalence rule across several columns:

```bash
./diff-gnome compare \
  --table promo_rules \
  --key docid \
  --equivalent-value 'applydiscount,billdiscount,showdiscount:1=Yes|0='
```

Keyless insert-only example:

```bash
./diff-gnome sync \
  --left-driver mysql \
  --left-dsn 'user:pass@tcp(left-host:3306)/appdb' \
  --right-driver mysql \
  --right-dsn 'user:pass@tcp(right-host:3306)/appdb' \
  --table event_journal \
  --insert-only \
  --apply
```

Apply a sync only if the planned writes stay under your thresholds:

```bash
./diff-gnome sync \
  --left-driver mysql \
  --left-dsn 'user:pass@tcp(left-host:3306)/appdb' \
  --right-driver mysql \
  --right-dsn 'user:pass@tcp(right-host:3306)/appdb' \
  --table customer_orders \
  --key id \
  --apply \
  --max-writes 5000 \
  --max-deletes 100
```

Write a rollback-friendly backup export before applying a sync:

```bash
./diff-gnome sync \
  --left-driver mysql \
  --left-dsn 'user:pass@tcp(left-host:3306)/appdb' \
  --right-driver mysql \
  --right-dsn 'user:pass@tcp(right-host:3306)/appdb' \
  --table customer_orders \
  --key id \
  --backup-file reports/customer-orders-sync-backup.sql \
  --backup-format sql \
  --apply
```

Force destructive runs to include a filter:

```bash
./diff-gnome sync \
  --left-driver mysql \
  --left-dsn 'user:pass@tcp(left-host:3306)/appdb' \
  --right-driver mysql \
  --right-dsn 'user:pass@tcp(right-host:3306)/appdb' \
  --table customer_orders \
  --key id \
  --where "updated_at >= '2026-01-01'" \
  --require-where \
  --apply
```

Run a preflight doctor before syncing a risky table:

```bash
./diff-gnome doctor \
  --left-driver sqlserver \
  --left-dsn 'sqlserver://user:pass@sqlhost:1433?database=targetdb&encrypt=disable' \
  --right-driver mysql \
  --right-dsn 'user:pass@tcp(source-host:3306)/appdb' \
  --table monthly_statement_items \
  --key account_id,statement_month,recipient_id,invoice_number \
  --where "statement_month >= '2026-01'" \
  --apply \
  --delete-missing \
  --max-writes 5000 \
  --max-deletes 250
```

Run multiple jobs from one config file:

```bash
./diff-gnome run --config examples/run-config.yaml
```

Scan a single table for duplicate key groups:

```bash
./diff-gnome dedupe \
  --driver sqlserver \
  --dsn 'sqlserver://user:pass@sqlhost:1433?database=appdb&encrypt=disable' \
  --table monthly_statement_items \
  --key account_id,statement_month,recipient_id,invoice_number \
  --ignore-column updated
```

Delete exact duplicate copies while keeping one row per identical set:

```bash
./diff-gnome dedupe \
  --driver sqlserver \
  --dsn 'sqlserver://user:pass@sqlhost:1433?database=appdb&encrypt=disable' \
  --table monthly_statement_items \
  --key account_id,statement_month,recipient_id,invoice_number \
  --ignore-column updated \
  --where "statement_month >= '2026-01'" \
  --require-where \
  --max-deletes 250 \
  --apply
```

Export the duplicate rows that will be removed before applying dedupe:

```bash
./diff-gnome dedupe \
  --driver sqlserver \
  --dsn 'sqlserver://user:pass@sqlhost:1433?database=appdb&encrypt=disable' \
  --table monthly_statement_items \
  --key account_id,statement_month,recipient_id,invoice_number \
  --ignore-column updated \
  --backup-file reports/monthly-statement-items-dedupe-backup.sql \
  --backup-format sql \
  --apply
```

The `dedupe` command runs against a single datasource, so you can point it at either side you want to clean up. `--ignore-column` lets you exclude noise columns such as `updated` from duplicate matching. `--apply` currently supports SQL Server, MySQL, and PostgreSQL.

Restore a backup export back into one datasource:

```bash
./diff-gnome restore \
  --driver sqlserver \
  --dsn 'sqlserver://user:pass@sqlhost:1433?database=targetdb&encrypt=disable' \
  --input-file reports/monthly-statement-items-dedupe-backup.sql \
  --input-format sql \
  --max-statements 500 \
  --transaction \
  --apply
```

Use `restore` with SQL backups for direct replay, or with JSONL backups when you want Diff Gnome to translate the structured backup records back into row-level inserts, updates, and deletes.
Use `--verify-only` for a stronger read-only check; for JSONL backups it validates the referenced tables and columns against the target datasource without replaying anything.

Override the config-file parallelism from the CLI:

```bash
./diff-gnome run --config examples/run-config.yaml --parallel 4
```

For multi-connection setups, prefer `diff-gnome run --config ...` over hidden shell state.

## Doctor

`diff-gnome doctor` is a read-only preflight command for two-sided runs. It checks:

- schema compatibility using the same loose/strict type rules as compare and sync
- filtered row counts on both sides
- duplicate-key groups on both sides for the key you provided
- whether the chosen key is actually backed by an index on each side
- whether the `where` columns are backed by leading index columns on each side
- how many indexes exist on the left/target table, to flag likely slow writes
- left-side triggers that would fire during `sync --apply` or `dedupe --apply`
- safety warnings based on `--apply`, `--delete-missing`, `--max-writes`, `--max-deletes`, and `--require-where`

It does not compare every row and it never writes data. Use it to catch weak keys, duplicate-key risk, missing key indexes, trigger-heavy targets, and obviously unsafe apply settings before you run a real sync.

## Run Config Files

`diff-gnome run` accepts YAML or JSON. The file supports:

- top-level `parallel` to control how many jobs run at once
- top-level `continue_on_error` to keep going after a job fails
- top-level `animate` to show animated batch progress on stderr
- top-level `output_dir` for generated job reports and mismatch exports
- named `connections` so you do not repeat DSNs
- `defaults` for any shared job settings
- a `jobs` list containing `mode: compare`, `mode: sync`, `mode: dedupe`, `mode: restore`, or `mode: doctor`

The job fields mirror the CLI flags using snake_case names, for example:

- `left_driver`, `left_driver_env`, `left_dsn`, `left_dsn_env`
- `right_driver`, `right_driver_env`, `right_dsn`, `right_dsn_env`
- `connection`, `driver`, `driver_env`, `dsn`, `dsn_env`
- `table`, `left_table`, `right_table`, `key`
- `input_file`, `input_format`
- `ignore_columns`, `date_only_columns`, `equivalent_values`, `empty_string_as_null`, `where`
- `output`, `output_file`, `event_file`, `mismatch_file`, `fail_on_diff`, `chunk_size`, `sample_limit`
- `backup_file`, `backup_format`
- `write_batch_size`, `max_statements`, `max_writes`, `max_deletes`, `require_where`, `transaction`, `verify_only`, `apply`, `delete_missing`, `insert_only`
- `strict_types`

Two-sided jobs use `left` and `right` connection references.
Single-datasource jobs such as `dedupe` and `restore` can use `connection`.
You can store connection strings inline with `dsn`, or reference environment variables with `dsn_env`.
Type matching is loose by default; set `strict_types: true` or pass `--strict-types` to require exact type matches.
`output` can be `text`, `json`, or `markdown`. The same applies to direct CLI runs via `--output markdown`.
When `output_file: true`, Diff Gnome writes a per-job report to `output_dir/<job-name>.txt`, `.json`, or `.md` based on `output`.
When `event_file: true`, Diff Gnome writes a machine-readable JSONL event stream to `output_dir/<job-name>-events.jsonl`.
When `mismatch_file: true` on a compare job, Diff Gnome writes every mismatch as one JSON object per line to `output_dir/<job-name>-mismatches.jsonl`.
If a compare job finds no mismatches, no mismatch file is created.
When `fail_on_diff: true` on a compare job, Diff Gnome still writes the report and event log, but the job is marked failed if any schema, count, or row differences are found.
`defaults.output_file` applies to all jobs.
`defaults.mismatch_file` applies only to compare jobs; sync jobs ignore that default.
`defaults.backup_file` applies only to sync and dedupe jobs; compare, restore, and doctor jobs ignore that default.
`backup_format` can be `jsonl` or `sql`; managed backup outputs use `.jsonl` or `.sql` accordingly.

Example:

```yaml
parallel: 2
continue_on_error: true
animate: true
output_dir: reports

connections:
  target_sqlserver:
    driver: sqlserver
    dsn_env: DIFFGNOME_LEFT_DSN
  source_mysql:
    driver: mysql
    dsn_env: DIFFGNOME_RIGHT_DSN

defaults:
  left: target_sqlserver
  right: source_mysql
  output: text
  chunk_size: 10000
  require_where: true
  output_file: true
  mismatch_file: true
  backup_file: true
  backup_format: sql

jobs:
  - name: audit-trail-compare
    mode: compare
    left_table: reporting.activity_log_archive
    right_table: activity_log
    ignore_columns: changedon
    date_only_columns: changedate

  - name: candidate-doctor
    mode: doctor
    table: candidate_records
    key: record_id
    where: "updated_at >= '2026-01-01'"
    apply: true
    max_writes: 5000
    max_deletes: 100

  - name: candidate-sync
    mode: sync
    table: candidate_records
    key: record_id
    where: "updated_at >= '2026-01-01'"
    apply: true
    write_batch_size: 1000
    output: json
```

If you want readable doctor/sync reports plus machine-readable job events by default:

```yaml
defaults:
  output: markdown
  output_file: true
  event_file: true
  fail_on_diff: true
```

That will produce files like:

- `reports/candidate-doctor.md`
- `reports/candidate-doctor-events.jsonl`

Inline DSNs work too:

```yaml
connections:
  target_sqlserver:
    driver: sqlserver
    dsn: 'sqlserver://user:password@sqlhost:1433?database=targetdb&encrypt=disable'
  source_mysql:
    driver: mysql
    dsn: 'user:password@tcp(mysqlhost:3306)/appdb'
```

## How It Works

1. Diff Gnome inspects both schemas.
2. It compares row counts.
3. It streams both tables ordered by the configured key columns.
4. When aligned key chunks match, it skips row-by-row comparison with a chunk hash.
5. When a chunk hash differs, it recursively splits the chunk until it finds the changed rows.
6. When keys diverge, it records rows missing from the right side or extra on the right side.

## Current Constraints

- Every key column must be non-null and sortable.
- `compare` can run without a key by matching rows on full row contents.
- `sync --insert-only` can run without a key by matching rows on full row contents and inserting rows that exist only on the right.
- Use `--left-table` and `--right-table` when the source and target table names differ.
- Use `--ignore-column` to exclude audit or precision-mismatched columns such as `changedon` from schema comparison and row matching.
- Use `--date-only-column` when one side stores full timestamps and the other stores only dates for the same logical field.
- Use `--equivalent-value 'column[,column]:value=value|value=value'` or `equivalent_values: 'column[,column]:value=value|value=value'` when one side rewrites display values during ETL, for example `applydiscount,billdiscount:1=Yes|0=`.
- Use `--empty-string-as-null` or `empty_string_as_null: true` when one side stores missing values as empty strings and the other stores real `NULL`s.
- Numeric columns are normalized before comparison, so equivalent values like `1.16` and `1.160` compare as equal.
- Common punctuation variants are normalized during text comparison, so characters like `‚` and `,` compare as equal.
- `diff-gnome run` supports YAML or JSON config files with many compare/sync jobs and bounded parallel execution.
- In loose mode, known temporal type pairs such as `SMALLDATETIME`, `DATETIME`, `DATETIME2`, `TIMESTAMP`, and `TIMESTAMPTZ` are treated as compatible across databases.
- In loose mode, date and datetime text values are normalized when they parse cleanly, so cases like `VARCHAR` vs `DATE` and SQL Server/MySQL timestamp formatting differences can compare as equal.
- In loose mode, common integer type families such as `INT`, `UNSIGNED INT`, and `TINYINT`, and text families such as `VARCHAR` and `TEXT`, are treated as compatible across databases.
- Use `--strict-types` or `strict_types: true` when you want exact type matches instead.
- Table and column names must be bare identifiers or dot-qualified identifiers.
- The `--where` clause is treated as trusted SQL and passed through to both databases unchanged.
- Only one table is supported per individual compare or sync job.
- SQL Server support paginates with `TOP n` ordered by the configured key columns.
