Introduction.

An external system sends your application a CRM export, supplier price list, or bank statement. The file must be validated and written to your database without importing the same delivery twice.

Laravel Data Sync turns that process into a definition class. The class names the source, file format, destination model, field mapping, and matching columns. The package finds files, reads rows, writes models, and records each attempt.

Your application keeps ownership of its models, its business rules, and whatever it does with the data afterwards.

The package follows semantic versioning: upgrading within 1.x is safe. Source and issues live at github.com/byrcsc/laravel-data-sync.

The three nouns

A sync definition is a class. Behavior lives in the class; config/data-sync.php holds only infrastructure: which disks, which queue, which retention windows. Registering a definition is one line of config.

A file is identified by the SHA-256 checksum of its contents, not by its name. Every checksum the package has seen is recorded in the sync_files ledger, which is never pruned. A checksum that has completed successfully is never processed again, so re-dropping the same file is a no-op and a failed file is retried on the next run. Rename a file and nothing changes; change one byte and it is a new file.

A run is one attempt at one file. Runs carry the counters, the write path, the batch id, the archive path, the warnings, and the error summary. Every forced or retried attempt creates a new run, so history shows each attempt rather than overwriting the last one. Runs are pruned on a retention schedule; the ledger they point at is not.

What is included

• Class-based sync definitions with a declarative field mapping, per-row validation rules, and four write policies.
• Streaming readers for CSV and other delimited text, XLSX, JSON, NDJSON, and fixed-width files.
• Queued processing that fans a file's chunks out through Bus::batch() so several workers share one file, plus an in-process mode for small feeds.
• A checksum ledger that makes reprocessing a no-op, and checksum-verified archives for completed files and preserved copies for failed ones.
• Run history with per-row failure records, an error summary, and queued batch progress, surfaced through sync:status.
• Retries: reprocess a whole finished run from its preserved file, or replay only the rows that failed.
• File transfers that move files to a destination disk with checksum verification, filename metadata, and templated destination paths.
• Schedule registration from the definition or from config, retention pruning, and a sync:doctor health check for the whole installation.

What it does not do

The package draws its edges deliberately. What follows describes what it sets out to do rather than what it might do later: treat none of it as planned work, and none of it as ruled out forever.

• Model pruning for absent rows. The package never deletes application records that stop appearing in a feed. It is written for incremental and delta feeds rather than full snapshots.
• PDF and OCR extraction. Transfers move PDFs; nothing reads inside them.
• AI-assisted extraction. Mapping is declarative and deterministic.
• Two-way sync. Data flows from files into models, never back out.
• API connectors. Sources are filesystems. An API feed has to be written to a disk first.
• Merge semantics for captured JSON. UnknownColumnPolicy::Capture overwrites the target attribute; it does not merge with what is there.
• Retry from a date. sync:retry takes a run, not a time range.
• Glob and pattern discovery. A source names either a directory, including files in its subdirectories, or a single file. There is no filtering by name or extension.

A first sync

use App\Models\Product;
use ByRcsc\LaravelDataSync\Contracts\Reader as ReaderContract;
use ByRcsc\LaravelDataSync\Definitions\Field;
use ByRcsc\LaravelDataSync\Definitions\Reader;
use ByRcsc\LaravelDataSync\Definitions\Source;
use ByRcsc\LaravelDataSync\Definitions\SyncDefinition;

final class ProductsSync extends SyncDefinition
{
    public function source(): Source
    {
        return Source::disk('incoming')->path('products');
    }

    public function format(): ReaderContract
    {
        return Reader::csv();
    }

    public function model(): string
    {
        return Product::class;
    }

    public function fields(): array
    {
        return [
            Field::make('sku', from: 'SKU'),
            Field::make('name', from: 'Description'),
        ];
    }

    public function matchOn(): array
    {
        return ['sku'];
    }
}

Register the class in data-sync.syncs, then run php artisan sync:run products. The name comes from the class: ProductsSync becomes products.

Design boundaries

Four decisions shape almost everything else in the package.

Content, not filenames, decides what has been processed. Discovery checksums every candidate file. The ledger is keyed on definition and checksum, and it is never pruned, because deleting from it would let an old file be imported a second time.

Chunks commit independently. A chunk that fails rolls back its own rows and leaves earlier chunks committed, which is what lets a large file make progress instead of losing an hour of work to row 900,000. Whole-file atomicity is opt-in through atomic().

Write hooks change the write path. A definition with no beforeSave() or afterSave() writes rows through a bulk upsert and dispatches no model events. Declaring either hook switches the definition to per-row Eloquent writes. The package does not guess. See the write path tradeoff.

The package never touches your source until the copy is verified. A file is staged, processed, and copied to the archive disk first. Only when that copy's checksum matches does the configured source action, whether that is archive, move, or delete, run against the original file.

What to read next

• Installation and setup to create the package tables and configure its disks.
• Quick start to import a CSV into an Eloquent model.
• File identity and runs to understand duplicate detection and import history.