›
›
›
  1. docs
  2. ›
  3. byrcsc/laravel-data-sync
1.x
Browse documentationOpenClose

Getting started

  • Introduction
  • Installation and setup
  • Quick start

Core concepts

  • File identity and runs
  • Sync definitions
  • Sources and connections
  • File formats and readers
  • Field mapping
  • Matching and writing
  • Validation and hooks

Running syncs

  • Running a sync
  • Queues and workers
  • Scheduling

Transfers

  • File transfers

Operations

  • Run history and status
  • Events and listeners
  • Failures and retries
  • Sensitive data
  • Health checks
  • Retention and pruning

Reference

  • Configuration
  • Console commands
  • Testing
  • Troubleshooting

Getting started

  • Introduction
  • Installation and setup
  • Quick start

Core concepts

  • File identity and runs
  • Sync definitions
  • Sources and connections
  • File formats and readers
  • Field mapping
  • Matching and writing
  • Validation and hooks

Running syncs

  • Running a sync
  • Queues and workers
  • Scheduling

Transfers

  • File transfers

Operations

  • Run history and status
  • Events and listeners
  • Failures and retries
  • Sensitive data
  • Health checks
  • Retention and pruning

Reference

  • Configuration
  • Console commands
  • Testing
  • Troubleshooting

byrcsc/laravel-data-sync · 1.x

Retention and pruning.

Prune run history, row failures, archives, and failed files on separate schedules.

php artisan sync:prune

sync:prune is scheduled daily whenever schedule registration is on. It enforces four independent windows:

KeyDefaultWhat it removes
retention.runs_days90Finished run records
retention.row_failures_days30Failed-row records and their payloads
retention.archive_days30Archived copies of processed files
retention.failed_files_days14Preserved copies of failed files
// config/data-sync.php
'retention' => [
    'runs_days' => 365,
    'row_failures_days' => 30,
    'archive_days' => 90,
    'failed_files_days' => 30,
],

Each value is a number of days, or null to disable that window entirely. Negative or non-integer values are rejected at boot.

What is protected

  • The sync_files ledger is never pruned. It is the record of what has been processed, and deleting from it would let an old file be imported a second time. Ledger rows are narrow, a checksum, a path, a filename, a size, and timestamps, so growth is slow and bounded by how many distinct files you receive.
  • Runs that are still running are never removed, however old.
  • Files referenced by a running run are skipped when pruning archives and failed files, so a long import cannot have its own working copy deleted underneath it.

Row-failure records belonging to a running run are left alone too, even past their window.

Independence, and what it implies

The four windows are deliberately separate. The common shape is a long run history and short file retention: runs_days at a year for reporting, archive_days at 30 because the files themselves are large and probably sensitive.

The consequence to keep in mind is that a run can outlive the file it points at. sync_runs.archive_path will name a file that no longer exists, and sync:retry on that run reports that the archived copy is gone. Retention is what bounds how far back a retry can reach, not the run history.

Likewise, row_failures_days bounds how long sync:retry --failed-rows remains possible, independently of the run itself.

What sync:prune does not clean

The staging disk. A dry run discards its staged copy and a failed run moves its copy to the failed disk, but a successfully processed file leaves its staged copy in place, and pruning never looks there. On a busy installation the staging area grows to hold a copy of every file ever imported.

Sweep it yourself, sized to how far back a retry needs to reach:

find storage/app/data-sync/staging -type f -mtime +30 -delete

or, on object storage, a lifecycle rule on the staging prefix. Nothing in the package reads a staged file after its run finalizes, other than a retry, which restores from the archive or failed disk rather than from staging.

Framework batch records are also outside its scope. Prune those with Laravel's own queue:prune-batches.

Reading the result

Pruned 412 runs, 1204 row failures, 88 archives, and 3 failed files.
use ByRcsc\LaravelDataSync\Actions\PruneHistory;

$report = app(PruneHistory::class)();

$report->runs;
$report->rowFailures;
$report->archives;
$report->failedFiles;

Pruning files is a listing plus a delete per expired path, so a very large archive disk makes the command proportionally slower. If it starts running long, shorten archive_days or move archives to storage with its own lifecycle rules and set archive_days to null.

A failure while listing or deleting stops the command with a message naming the disk config key at fault, and exits non-zero. Nothing is half-deleted beyond the files it had already removed.

Turning it off

Setting a window to null disables it. Setting data-sync.register_schedules to false stops the daily invocation along with everything else the package schedules, wire sync:prune up yourself if you do that, or history and archives grow without bound.

What to read next

  • Sensitive data for what each copy contains.
  • Failures and retries for what retention costs you.
  • Scheduling for the daily registration.
PreviousHealth checksNextConfiguration
View source

On this page

  1. What is protected
  2. Independence, and what it implies
  3. What sync:prune does not clean
  4. Reading the result
  5. Turning it off
  6. What to read next