Browse documentationOpen

byrcsc/laravel-data-sync · 1.x

Sensitive data.

What the package copies where, how to turn failed-row payload capture off per definition or globally, and what you give up when you do.

A sync makes copies. Knowing which ones exist, and for how long, is the whole of this page.

CopyWhere it livesRetention
The staged filedata-sync.staging diskNot pruned — see below
The archived filedata-sync.archive diskretention.archive_days (30)
The preserved failed filedata-sync.failed diskretention.failed_files_days (14)
Failed-row payloadssync_row_failures.payloadretention.row_failures_days (30)
Ledger metadatasync_filesNever pruned

Staging is not swept. A dry run discards its staged copy, and a failed run moves its copy to the failed disk, but a successfully processed file leaves its staged copy behind, and sync:prune does not touch the staging disk. Over time the staging area holds a copy of every file the package has imported. Sweep it yourself — a scheduled find, an object-lifecycle rule on the bucket — sized to how far back a retry needs to reach.

The ledger holds a checksum, a path, a filename, a size, and timestamps — no row contents. Everything else that carries actual data is on a retention window.

Failed-row payloads

When a row fails, its payload is recorded in sync_row_failures alongside the validation errors. That is what makes sync:status --failures readable and sync:retry --failed-rows possible.

For a feed carrying personal or financial data, it is also a copy of that data in a second table, with a different retention window and probably different access controls than the destination model.

Turn it off for one definition:

public function captureFailedPayloads(): bool
{
    return false;
}

Or, when you build the definition yourself:

$definition = (new PayrollSync)->withoutPayloadCapture();

Turn it off everywhere:

// config/data-sync.php
'store_failed_payloads' => false,

The global toggle wins. A definition cannot re-enable capture that config has turned off.

What you keep, and what you lose

With capture off, failures still record the row number, the validation errors, and the exception message — everything except the data itself. You keep the counts, the reasons, and the histogram in error_summary.

You lose two things:

  • sync:status --failures prints [payload capture disabled] instead of the row.
  • sync:retry --failed-rows has nothing to replay and refuses to run. Retrying the whole archived file still works.

The archived file is the bigger copy

Payload capture covers failed rows. The archive covers every row: it is a byte-for-byte copy of the supplier's file, kept for retention.archive_days. For a sensitive feed that is usually the copy worth thinking about first.

Options, in order of how much they cost you:

  • Shorten retention.archive_days. Retries and after-the-fact diagnosis stop working past that window, but the ledger still prevents reprocessing.
  • Point the archive, staging, and failed disks at encrypted storage — a server-side-encrypted bucket, an encrypted volume. The package writes through Flysystem, so this is a disk configuration concern rather than a package one.
  • Restrict access to those disks separately from the application's public storage. Nothing in the package serves archived files to users.

Note that the failed disk holds a copy for retention.failed_files_days after a failure, and the staging disk keeps one indefinitely unless you sweep it. Point all three somewhere appropriate, not just the archive.

Sensitive columns you do not want at all

Two mechanisms keep data out of the destination model:

  • Map only the fields you need. Unmapped columns are ignored by default, and are never written — only their names are recorded, on the run's unmapped_columns.
  • Do not enable UnknownColumnPolicy::Capture on a feed with sensitive extras. Capture writes every unmapped column into a JSON attribute, which is the opposite of what you want here. If you need it anyway, cast the target with encrypted:array.

Neither removes the data from the archived file, which is a copy of what the supplier sent.