byrcsc/laravel-data-sync · 1.x

Installation and setup.

Install the package, publish its configuration and migration, prepare the queue and Flysystem adapters, and verify the installation with sync:doctor.

Install the package

composer require byrcsc/laravel-data-sync
php artisan vendor:publish --tag="data-sync-config"
php artisan vendor:publish --tag="data-sync-migrations"
php artisan migrate

Publish the configuration before the migration when you want custom table names. The migration reads them from data-sync.tables, so changing the names afterwards means renaming tables by hand.

The migration creates three tables:

Table	What it holds
`sync_files`	The ledger: one row per definition and checksum, never pruned
`sync_runs`	One row per attempt, with counters, write path, batch id, archive path
`sync_row_failures`	One row per failed row, with its errors and optional payload

Queue batches

Queued syncs fan a file's chunks out through Bus::batch(), so the framework's job_batches table must exist. It ships with Laravel's default migrations; older applications add it with:

php artisan make:queue-batches-table
php artisan migrate

sync:doctor reports a missing job_batches table as a failure. Without it, only sync:run --now works.

Flysystem adapters

FTP and SFTP sources need a Flysystem adapter, and neither is installed by default:

composer require league/flysystem-ftp        # FTP sources
composer require league/flysystem-sftp-v3    # SFTP sources

A data-sync.connections entry whose driver is ftp or sftp throws an InvalidConfigurationException at boot when its adapter is missing, naming the package to install.

Disks

The package uses three working areas, all configured as filesystem disks:

// config/data-sync.php
'staging' => [
    'disk' => 'local',
    'path' => 'data-sync/staging',
],

'archive' => [
    'disk' => 'local',
    'path' => 'data-sync/archive',
],

'failed' => [
    'disk' => null,                 // falls back to the staging disk
    'path' => 'data-sync/failed',
],

Staging is where a discovered file is copied before it is read. Everything downstream — planning, chunk jobs, archiving, transfers — reads from staging, never from the source.

Archive holds a checksum-verified copy of every successfully processed file, under <path>/<definition>/<Y/m/d>/<run ulid>-<checksum>-<filename>.

Failed holds a preserved copy of the staged file for runs that failed or were cancelled. sync:retry restores from whichever of the two applies.

Staging is shared state. sync:run stages files in the dispatching process, and the worker that processes them reads from staging. When the dispatcher and the worker are on different machines, a local staging disk means the worker looks for a file that is not there. See queues and workers.

Sources

A source is either a filesystem disk from config/filesystems.php:

Source::disk('incoming')->path('products');

or an entry in data-sync.connections, which is an inline Flysystem configuration built on demand — useful for a supplier SFTP server that does not deserve a permanent disk:

// config/data-sync.php
'connections' => [
    'supplier-sftp' => [
        'driver' => 'sftp',
        'host' => env('SUPPLIER_SFTP_HOST'),
        'username' => env('SUPPLIER_SFTP_USERNAME'),
        'privateKey' => env('SUPPLIER_SFTP_KEY_PATH'),
        'root' => '/outbound',
    ],
],

Connection arrays take the same keys as the matching driver in config/filesystems.php. See sources and connections.

Registering definitions

Definitions are registered by class name:

// config/data-sync.php
'syncs' => [
    App\Syncs\ProductsSync::class,
    App\Syncs\StatementsTransfer::class,
],

The list is validated at boot. A class that does not exist, does not extend SyncDefinition or TransferDefinition, or collides with another definition's name throws an InvalidConfigurationException immediately rather than at run time.

Queue routing

Package jobs run on the default queue connection unless you say otherwise:

// config/data-sync.php
'queue' => [
    'connection' => 'redis',
    'queue' => 'imports',
],

Per-definition routing is available through overrides and through the definition itself. See queues and workers.

Verify the installation

php artisan sync:doctor

Run it after installing and in your deploy pipeline. It checks configuration validity, required tables, definition resolution, source and destination reachability, disk writability, the cache lock store, matchOn indexes, and stuck runs. --strict turns warnings into a non-zero exit. See health checks.

Scaffolding a definition

php artisan make:sync ProductsSync
php artisan make:sync StatementsTransfer --transfer

Both write to app/Syncs/. The Sync or Transfer suffix is appended when you leave it off, so make:sync Products and make:sync ProductsSync generate the same class.