Browse documentationOpen

byrcsc/laravel-data-sync · 1.x

Retention and pruning.

The four independent retention windows sync:prune enforces, what it protects, what it never touches, and how retention interacts with retries.

php artisan sync:prune

sync:prune is scheduled daily whenever schedule registration is on. It enforces four independent windows:

KeyDefaultWhat it removes
retention.runs_days90Finished run records
retention.row_failures_days30Failed-row records and their payloads
retention.archive_days30Archived copies of processed files
retention.failed_files_days14Preserved copies of failed files
// config/data-sync.php
'retention' => [
    'runs_days' => 365,
    'row_failures_days' => 30,
    'archive_days' => 90,
    'failed_files_days' => 30,
],

Each value is a number of days, or null to disable that window entirely. Negative or non-integer values are rejected at boot.

What is protected

  • The sync_files ledger is never pruned. It is the record of what has been processed, and deleting from it would let an old file be imported a second time. Ledger rows are narrow — a checksum, a path, a filename, a size, and timestamps — so growth is slow and bounded by how many distinct files you receive.
  • Runs that are still running are never removed, however old.
  • Files referenced by a running run are skipped when pruning archives and failed files, so a long import cannot have its own working copy deleted underneath it.

Row-failure records belonging to a running run are left alone too, even past their window.

Independence, and what it implies

The four windows are deliberately separate. The common shape is a long run history and short file retention: runs_days at a year for reporting, archive_days at 30 because the files themselves are large and probably sensitive.

The consequence to keep in mind is that a run can outlive the file it points at. sync_runs.archive_path will name a file that no longer exists, and sync:retry on that run reports that the archived copy is gone. Retention is what bounds how far back a retry can reach — not the run history.

Likewise, row_failures_days bounds how long sync:retry --failed-rows remains possible, independently of the run itself.

What sync:prune does not clean

The staging disk. A dry run discards its staged copy and a failed run moves its copy to the failed disk, but a successfully processed file leaves its staged copy in place, and pruning never looks there. On a busy installation the staging area grows to hold a copy of every file ever imported.

Sweep it yourself, sized to how far back a retry needs to reach:

find storage/app/data-sync/staging -type f -mtime +30 -delete

or, on object storage, a lifecycle rule on the staging prefix. Nothing in the package reads a staged file after its run finalizes, other than a retry — which restores from the archive or failed disk rather than from staging.

Framework batch records are also outside its scope. Prune those with Laravel's own queue:prune-batches.

Reading the result

Pruned 412 runs, 1204 row failures, 88 archives, and 3 failed files.
use ByRcsc\LaravelDataSync\Actions\PruneHistory;

$report = app(PruneHistory::class)();

$report->runs;
$report->rowFailures;
$report->archives;
$report->failedFiles;

Pruning files is a listing plus a delete per expired path, so a very large archive disk makes the command proportionally slower. If it starts running long, shorten archive_days or move archives to storage with its own lifecycle rules and set archive_days to null.

A failure while listing or deleting stops the command with a message naming the disk config key at fault, and exits non-zero. Nothing is half-deleted beyond the files it had already removed.

Turning it off

Setting a window to null disables it. Setting data-sync.register_schedules to false stops the daily invocation along with everything else the package schedules — wire sync:prune up yourself if you do that, or history and archives grow without bound.