Introduction.

An external system sends your application a CRM export, supplier price list, or bank statement. The file must be validated and written to your database without importing the same delivery twice.

Laravel Data Sync turns that process into a definition class. The class names the source, file format, destination model, field mapping, and matching columns. The package finds files, reads rows, writes models, and records each attempt.

A definition can also pull from an HTTP API or from another database, such as an ERP. The package pages through the endpoint or the table, saves what it read as a file, and imports that file the same way.

Your application keeps ownership of its models, its business rules, and whatever it does with the data afterwards.

Version 2.0 renamed several classes and methods from 1.0.0. Upgrading from 1.0 lists every change. Source and issues live at github.com/byrcsc/laravel-data-sync.

The three nouns

A sync definition is a class. Behavior lives in the class; config/data-sync.php holds only infrastructure: which disks, which queue, which retention windows. Registering a definition is one line of config.

A file is identified by the SHA-256 checksum of its contents, not by its name. Every checksum the package has seen is recorded in the sync_files ledger, which is never pruned. A checksum that has completed successfully is never processed again, so re-dropping the same file is a no-op and a failed file is retried on the next run. Rename a file and nothing changes; change one byte and it is a new file. An API sync or a database sync writes each fetch to a file, so the same rules apply to what an API or a table returns.

A run is one attempt at one file. Runs carry the counters, the write strategy, the batch id, the archive path, the warnings, and the error summary. Every forced or retried attempt creates a new run, so history shows each attempt rather than overwriting the last one. Runs are pruned on a retention schedule; the ledger they point at is not.

What is included

• Class-based sync definitions with a declarative field mapping, per-row validation rules, and four write policies.
• Streaming readers for CSV and other delimited text, XLSX, JSON, NDJSON, and fixed-width files.
• Queued processing that fans a file's chunks out through Bus::batch() so several workers share one file, plus an in-process mode for small feeds.
• A checksum ledger that makes reprocessing a no-op, and checksum-verified archives for completed files and preserved copies for failed ones.
• Run history with per-row failure records, an error summary, and queued batch progress, surfaced through sync:status.
• Retries: reprocess a whole finished run from its preserved file, or replay only the rows that failed.
• API syncs that page through an HTTP endpoint, save each fetch as an NDJSON snapshot, and resume from a watermark that moves only after a completed run.
• Database syncs that read any connection in config/database.php in key-ordered pages, with a timestamp watermark, or a rowversion watermark on SQL Server.
• Definition providers that register definitions at runtime, such as one per database each account connects.
• File transfers that move files to a destination disk with checksum verification, filename metadata, and templated destination paths.
• Schedule registration from the definition or from config, retention pruning, and a sync:doctor health check for the whole installation.

What it does not do

The package draws its edges deliberately. What follows describes what it sets out to do rather than what it might do later: treat none of it as planned work, and none of it as ruled out forever.

• Model pruning for absent rows. The package never deletes application records that stop appearing in a feed. It is written for incremental and delta feeds rather than full snapshots.
• PDF and OCR extraction. Transfers move PDFs; nothing reads inside them.
• AI-assisted extraction. Mapping is declarative and deterministic.
• Two-way sync. Data flows into models. Sending anything back to a remote system or a source database is your application's job. See send acknowledgements back.
• Hard deletes in a source database. A watermark only sees rows that still exist, so the package never reads a row that the source deleted.
• The package's own tables on SQL Server. A database sync can read from SQL Server, but the package tests its ledger and history tables on SQLite, MySQL, and PostgreSQL only.
• Cross-connection joins. A definition's query runs on one connection.
• Webhooks and inbound push. An API sync polls on a schedule. Nothing listens for a remote system to call your application.
• OAuth token refresh. An API sync's request() receives a PendingRequest, and your application applies credentials it already manages.
• Merge semantics for captured JSON. UnknownColumnPolicy::Capture overwrites the target attribute; it does not merge with what is there.
• Retry from a date. sync:retry takes a run, not a time range.
• Glob and pattern discovery. A source names either a directory, including files in its subdirectories, or a single file. There is no filtering by name or extension.

A first sync

use App\Models\Product;
use ByRcsc\LaravelDataSync\Contracts\Reader;
use ByRcsc\LaravelDataSync\Definitions\Field;
use ByRcsc\LaravelDataSync\Definitions\Format;
use ByRcsc\LaravelDataSync\Definitions\Source;
use ByRcsc\LaravelDataSync\Definitions\SyncDefinition;

final class ProductsSync extends SyncDefinition
{
    public function source(): Source
    {
        return Source::disk('incoming')->path('products');
    }

    public function format(): Reader
    {
        return Format::csv();
    }

    public function model(): string
    {
        return Product::class;
    }

    public function fields(): array
    {
        return [
            Field::make('sku', from: 'SKU'),
            Field::make('name', from: 'Description'),
        ];
    }

    public function matchOn(): array
    {
        return ['sku'];
    }
}

Register the class in data-sync.syncs, then run php artisan sync:run products. The name comes from the class: ProductsSync becomes products.

Design boundaries

Four decisions shape almost everything else in the package.

Content, not filenames, decides what has been processed. Discovery checksums every candidate file. The ledger is keyed on definition and checksum, and it is never pruned, because deleting from it would let an old file be imported a second time.

Chunks commit independently. A chunk that fails rolls back its own rows and leaves earlier chunks committed, which is what lets a large file make progress instead of losing an hour of work to row 900,000. Whole-file atomicity is opt-in through atomic().

Write hooks change the write strategy. A definition with no beforeSave() or afterSave() writes rows through a bulk upsert and dispatches no model events. Declaring either hook switches the definition to per-row Eloquent writes. The package does not guess. See the write strategy tradeoff.

The package never touches your source until the copy is verified. A file is staged, processed, and copied to the archive disk first. Only when that copy's checksum matches does the configured source action, whether that is archive, move, or delete, run against the original file.

What to read next

• Installation and setup to create the package tables and configure its disks.
• Quick start to import a CSV into an Eloquent model.
• File identity and runs to understand duplicate detection and import history.
• Syncing from an API when the data sits behind an HTTP endpoint.
• Syncing from a database when the data sits in another database.