Browse documentationOpen

byrcsc/laravel-data-sync · 1.x

Running a sync.

Run definitions from the console or from code, choose between the queued and in-process paths, force a reprocess, preview with a dry run, and understand the discovery lock.

php artisan sync:run                 # every enabled definition
php artisan sync:run products        # one definition
php artisan sync:run products --now  # process in this process, no queue
php artisan sync:run products --force
php artisan sync:run products --dry-run

With no name, every enabled definition runs in registration order. A definition disabled through data-sync.overrides.<name>.enabled is skipped; one named explicitly runs regardless.

What a run does

  1. Takes a cache lock for the definition and lists the source directory.
  2. Defers files younger than minFileAge(), and skips files already recorded as completed.
  3. Streams each remaining file to the staging disk and checksums it.
  4. Records a run per file.
  5. Processes it — queued by default, in this process with --now.

The command prints one line per run plus a summary:

Running [products]...
  File run [01JQ8Z3M0K2N4P6R8T0V2X4Y6A] (bulk) ... 2 created, 0 updated, 0 skipped, 0 failed
[products] finished: 1 discovered, 0 skipped, 0 deferred.

discovered counts files that produced a run, skipped counts checksums already completed, and deferred counts files held back by minFileAge().

On the queued path the per-run counters are the state at dispatch — usually all zeros — because nothing has been written yet. Use sync:status for the outcome.

Queued by default

sync:run records a run per discovered file and dispatches a job for each. That job plans the file, then fans its chunks out through Bus::batch() so several workers can share one file. Nothing is written until a worker picks the jobs up.

The queued path needs a worker and the job_batches table. See queues and workers.

--now

--now skips the queue and does everything in the command: discovery, reading, mapping, writing, archiving. It is the right choice for small feeds, for local development, for one-off backfills, and for anywhere without a worker.

The two paths write identically. What differs is parallelism, and where an error surfaces: an in-process run rethrows, so the command prints the message and exits non-zero, while a queued run records the failure on the run and fails the job.

--force

--force processes a checksum that has already completed. Without it, discovery skips such files twice over: once by a path, size, and modification-time match that avoids reading the file at all, and again after checksumming.

A forced run creates a new run rather than rewriting the old one, so the history shows both attempts. It does not delete or reconcile anything that the first run wrote — for an Upsert definition it simply writes the same rows again.

--dry-run

php artisan sync:run products --dry-run

A dry run validates and counts without writing models, running write hooks, archiving the file, cleaning up the source, or marking the checksum processed. It always runs in-process; it never queues, even without --now.

The run is still recorded, and sync:status marks it as a dry run. Because nothing is marked processed, the same file is discovered again on the next run.

Two caveats:

  • Dry runs skip prepare(), so a definition that reshapes rows there can report slightly different numbers than the real run.
  • Dry runs still dispatch events. A SyncCompleted listener that notifies people should check $event->run->dry_run.

The discovery lock

Discovery takes a cache lock per definition for five minutes, so two overlapping sync:run invocations do not stage or process the same file twice. The second one reports that discovery is already running and exits successfully:

Skipped [products] because discovery is already running.

Exiting zero is deliberate: an overlapping scheduled run is a normal condition, not a failure worth paging anyone about. The lock covers discovery only — once files are dispatched, processing runs as long as it needs to.

The lock lives in the cache, so the cache store must be one that supports atomic locks. sync:doctor verifies it.

Running from code

use ByRcsc\LaravelDataSync\Facades\DataSync;

$report = DataSync::run('products');              // in-process, like --now
$report = DataSync::run('products', force: true);
$report = DataSync::run('products', dryRun: true);

$report = DataSync::queue('products');            // discover and dispatch, like plain sync:run
$report = DataSync::queue('products', force: true);

run() and queue() are the two paths the command exposes as --now and the default. Both return a RunReport:

$report->definition;   // 'products'
$report->discovered;   // files that produced a run
$report->skipped;      // checksums already completed
$report->deferred;     // files held back by minFileAge()
$report->locked;       // true when discovery was already running
$report->runs;         // list<SyncReport>, one per file

Each SyncReport carries the SyncRun model, its status and write path, and the total, created, updated, skipped, and failed counters. On the queued path those counters are the state at dispatch, so a queue() report tells you what was discovered, not what was written.

Three more methods read the registry:

DataSync::registered();          // ['products' => ProductsSync, ...]
DataSync::enabled();             // minus definitions disabled in config
DataSync::definition('products');

registered() validates every definition as it resolves, so calling it is also a cheap way to assert your configuration is sound. definition() throws an InvalidDefinitionException for a name that is not registered.

Those five methods are the whole programmatic surface. The facade resolves ByRcsc\LaravelDataSync\SyncManager, so app(SyncManager::class) works identically where you would rather inject it.