›
byrcsc/laravel-data-sync · 1.x
Control where imported files and failed row payloads are stored and retained.
A sync makes copies. Knowing which ones exist, and for how long, is the whole of this page.
| Copy | Where it lives | Retention |
|---|---|---|
| The staged file | data-sync.staging disk | Not pruned, see below |
| The archived file | data-sync.archive disk | retention.archive_days (30) |
| The preserved failed file | data-sync.failed disk | retention.failed_files_days (14) |
| Failed-row payloads | sync_row_failures.payload | retention.row_failures_days (30) |
| Ledger metadata | sync_files | Never pruned |
Staging is not swept. A dry run discards its staged copy, and a failed run moves its copy to the failed disk, but a successfully processed file leaves its staged copy behind, and
sync:prunedoes not touch the staging disk. Over time the staging area holds a copy of every file the package has imported. Sweep it yourself, a scheduledfind, an object-lifecycle rule on the bucket , sized to how far back a retry needs to reach.
The ledger holds a checksum, a path, a filename, a size, and timestamps, no row contents. Everything else that carries actual data is on a retention window.
When a row fails, its payload is recorded in sync_row_failures alongside the
validation errors. That is what makes sync:status --failures readable and
sync:retry --failed-rows
possible.
For a feed carrying personal or financial data, it is also a copy of that data in a second table, with a different retention window and probably different access controls than the destination model.
Turn it off for one definition:
public function captureFailedPayloads(): bool
{
return false;
}Or, when you build the definition yourself:
$definition = (new PayrollSync)->withoutPayloadCapture();Turn it off everywhere:
// config/data-sync.php
'store_failed_payloads' => false,The global toggle wins. A definition cannot re-enable capture that config has turned off.
With capture off, failures still record the row number, the validation errors,
and the exception message, everything except the data itself. You keep the
counts, the reasons, and the histogram in error_summary.
You lose two things:
sync:status --failures prints [payload capture disabled] instead of the
row.sync:retry --failed-rows has nothing to replay and refuses to run. Retrying
the whole archived file still works.Payload capture covers failed rows. The archive covers every row: it is a
byte-for-byte copy of the supplier's file, kept for retention.archive_days.
For a sensitive feed that is usually the copy worth thinking about first.
Options, in order of how much they cost you:
retention.archive_days. Retries and after-the-fact diagnosis
stop working past that window, but the ledger still prevents reprocessing.Note that the failed disk holds a copy for retention.failed_files_days after a
failure, and the staging disk keeps one indefinitely unless you sweep it. Point
the archive, staging, and failed copies in appropriate storage.
Two mechanisms keep data out of the destination model:
unmapped_columns.UnknownColumnPolicy::Capture
on a feed with sensitive extras. Capture writes every unmapped column into a
JSON attribute, which is the opposite of what you want here. If you need it
anyway, cast the target with encrypted:array.Neither removes the data from the archived file, which is a copy of what the supplier sent.