byrcsc/laravel-data-sync · 1.x

File identity and runs.

How the checksum ledger decides what has already been processed, what a run records, and the full path a file takes from discovery to archive.

Two records carry the audit trail: a file in the sync_files ledger, and a run in sync_runs for each attempt at that file.

A file is its contents

Discovery computes the SHA-256 checksum of every candidate file and keys the ledger on (definition, checksum). That has three consequences worth internalizing:

Renaming changes nothing. products-final.csv and products.csv with identical bytes are the same file.
One changed byte makes a new file. A supplier that re-exports the same data with a new timestamp column produces a new checksum and a new run.
The same file can belong to two definitions. The ledger is per definition, so two syncs reading the same directory each track it separately.

sync_files also records the path, filename, size, mtime, first-seen time, and last-processed time, but none of those decide identity.

File status	Meaning
`pending`	Discovered and staged, not yet processed
`processing`	A run is in flight
`completed`	A run finished; the checksum is now closed
`failed`	The last attempt failed; it is retried freely

Only a completed checksum blocks reprocessing. A failed file is picked up again on the next sync:run without --force.

sync:prune removes runs, failed rows, archived files, and preserved failed files. It never removes ledger rows, because a deleted checksum is an old file that can be imported a second time. Ledger growth is one narrow row per file per definition.

A run is one attempt

Every processing attempt creates a run — including forced reprocessing and every retry — so history shows each attempt rather than overwriting the last.

Column	What it carries
`ulid`	The public identifier used by `sync:status` and `sync:retry`
`definition`	The registered name
`status`	See below
`write_path`	`bulk`, `eloquent`, or null for a transfer
`dry_run`	Whether the run wrote anything
`batch_id`	The queued batch, when the run fanned out
`rows_total` … `rows_failed`	Counters
`unmapped_columns`	Header columns no field mapped
`error_summary`	The failure message, or a JSON histogram of row errors
`warnings`	Non-fatal problems, such as a failed source cleanup
`archive_path`	The archived copy, or the preserved failed copy

Run status	Meaning
`pending`	Recorded, waiting on a worker
`running`	In progress
`completed`	Every row written
`completed_with_errors`	Finished; some rows failed and were recorded
`failed`	The run did not finish
`cancelled`	The queued batch was cancelled

The last four are terminal. sync:retry only accepts a terminal run.

The path a file takes

Discovery takes a cache lock per definition and lists the source directory. Files younger than minFileAge() are deferred. Files whose path, size, and modification time match an already-completed ledger row are skipped without being read.
Staging streams each remaining file to the staging disk under a temporary name, checksums it, and moves it to <staging path>/<definition>/<checksum>-<filename>. If that checksum has already completed, the staged copy is deleted and the file is counted as skipped.
A run is recorded for each file that survives, with its write path resolved from the definition.
Processing reads from staging — never from the source. Queued runs plan the file first, then fan chunks out through Bus::batch(); --now and atomic() runs read and write in one process. See running a sync.
Finalizing archives the staged file to the archive disk and verifies the copy's checksum. A mismatch deletes the copy and fails the run. Only after a verified archive does the configured source action run.
Source cleanup applies archiveTo(), moveSourceTo(), or deleteFromSource(). A failure here does not fail the run; it is recorded in the run's warnings.

A failed run takes a different exit: the staged file is copied to the failed disk, verified, and removed from staging, and its path is stored on the run so sync:retry can restore it later.

Dry runs

A dry run records a run and reports counts, then discards the staged file. It does not archive, does not touch the source, and does not mark the checksum processed, so the same file is discovered again on the next run.

Reading the records

Both models are ordinary Eloquent models and can be queried directly:

use ByRcsc\LaravelDataSync\Models\SyncFile;
use ByRcsc\LaravelDataSync\Models\SyncRun;

$run = SyncRun::query()->where('ulid', $ulid)->firstOrFail();

$run->file;                 // the ledger row
$run->failures;             // SyncRowFailure records
$run->status;               // a RunStatus enum
$run->write_path;           // a WritePath enum, or null for transfers

SyncFile::query()
    ->where('definition', 'products')
    ->where('status', 'failed')
    ->get();

Table names come from data-sync.tables, so both models resolve their table at runtime rather than hard-coding it.