›
›
›
  1. docs
  2. ›
  3. byrcsc/laravel-data-sync
2.x
Browse documentationOpenClose

Getting started

  • Introduction
  • Installation and setup
  • Quick start
  • Upgrading from 1.0

Core concepts

  • File identity and runs
  • Sync definitions
  • Sources and connections
  • File formats and readers
  • Field mapping
  • Matching and writing
  • Validation and hooks

Running syncs

  • Running a sync
  • Queues and workers
  • Scheduling

APIs and databases

  • Syncing from an API
  • Syncing from a database
  • Definition providers

Transfers

  • File transfers

Operations

  • Run history and status
  • Events and listeners
  • Failures and retries
  • Sensitive data
  • Health checks
  • Retention and pruning

Reference

  • Configuration
  • Console commands
  • Testing
  • Troubleshooting

Getting started

  • Introduction
  • Installation and setup
  • Quick start
  • Upgrading from 1.0

Core concepts

  • File identity and runs
  • Sync definitions
  • Sources and connections
  • File formats and readers
  • Field mapping
  • Matching and writing
  • Validation and hooks

Running syncs

  • Running a sync
  • Queues and workers
  • Scheduling

APIs and databases

  • Syncing from an API
  • Syncing from a database
  • Definition providers

Transfers

  • File transfers

Operations

  • Run history and status
  • Events and listeners
  • Failures and retries
  • Sensitive data
  • Health checks
  • Retention and pruning

Reference

  • Configuration
  • Console commands
  • Testing
  • Troubleshooting

byrcsc/laravel-data-sync · 2.x

Health checks.

Check configuration, disks, connections, tables, and indexes before a sync runs.

php artisan sync:doctor
php artisan sync:doctor --strict

sync:doctor inspects the installation and its operational dependencies and reports two kinds of finding. Failures exit non-zero. Warnings do not, unless --strict is given.

Run it after installing, after changing disks or connections, and in your deploy pipeline. It is designed to turn a silent 2 a.m. failure into a deploy-time one.

Failures

CheckFails when
Configurationdata-sync config is invalid, an unknown definition class, a duplicate name, an unknown override key, a bad cron expression, a bad retention value, a connection with no driver, or an FTP/SFTP driver whose Flysystem adapter is missing
Tablessync_files, sync_runs, sync_row_failures, or job_batches is missing, or sync_watermarks is missing while an API or database sync is registered
DefinitionsA registered definition cannot be resolved or fails validation
ConnectionsA data-sync.connections entry cannot list files
SourcesA definition's source cannot be listed. For an API or database sync, that is its directory in the staging area
Source databasesA database sync's connection cannot be reached, its driver's PDO extension is not loaded, or its source table lacks a mapped, key, or watermark column
Transfer destinationsA transfer's destination disk cannot be listed
Working disksThe staging or archive disk rejects a write-and-delete probe
Cache locksThe cache store cannot acquire and release a lock

The probe writes a small file into the configured staging and archive paths and removes it again, so a misconfigured bucket policy shows up here rather than half way through a 400,000-row import.

Failures are deduplicated, so one broken connection shared by three definitions is reported once.

For each database sync, the doctor reads the source table's columns with one schema query and reads no rows. A missing PDO extension is reported with the command that installs it, such as pecl install sqlsrv pdo_sqlsrv for SQL Server. The doctor also checks every definition that a definition provider returns.

Warnings

CheckWarns when
matchOn indexesA definition's match columns have no matching unique or primary index on the destination table
Stuck runsA run has been running for more than an hour
Staging driverThe staging disk uses the local driver
Stalled watermarkAn API or database sync's last 5 runs all finished without completed

All four are worth reading rather than silencing.

A missing unique index on matchOn() means concurrent chunk jobs can insert the same logical row twice, and upsert() has nothing to conflict on. The index columns must match the matchOn() set exactly, in any order.

A stuck run is almost always a worker that died mid-batch. Restart the worker; the batch resumes with whatever chunks have not run. If the batch is gone, retry the run.

A stalled watermark means one or more rows fail on every run. The watermark moves only after a completed run, so each run reads the same window again and the window grows. List the failed rows with sync:status <name> --failures, fix the cause, then replay them with sync:retry. Change the run count with data-sync.stalled_watermark_runs.

A local staging disk is a single-machine configuration. On one box it is correct and the warning is expected noise. Across machines it means a worker will look for a staged file that is not there. See queues and workers.

--strict in a pipeline

php artisan sync:doctor --strict

--strict makes any warning a non-zero exit, which is what you want in CI or a deploy gate, with one caveat: a single-machine deployment will always warn about the local staging disk, so --strict there fails permanently. Either point staging at shared storage, or use plain sync:doctor and read the warnings.

What it does not check

  • Whether files are actually arriving. A source that is reachable and empty passes. Alert on run history for that, not on the doctor.
  • Whether a definition's mapping matches today's file. Schema validation happens when a file is read, because a header is a property of a file rather than of a configuration. Use --dry-run for that.
  • Queue workers. It verifies job_batches exists, not that anything is consuming the queue.
  • API endpoints. It checks that an API sync's endpoint() is an http or https URL, and never sends it a request. A health check must not add load to another system's API. A --dry-run of the sync is the way to test the endpoint.
  • Retention drift. It validates the retention values, not whether pruning has been running.

Running it from code

use ByRcsc\LaravelDataSync\Actions\InspectInstallation;

$report = app(InspectInstallation::class)();

$report->failures;   // list<string>
$report->warnings;   // list<string>

Useful for a /health endpoint or a scheduled check that posts to Slack. It performs real I/O against every source and disk, so run it on a schedule rather than per request.

What to read next

  • Installation and setup for the dependencies being checked.
  • Configuration for the keys that must be valid.
  • Troubleshooting for what to do about specific findings.
PreviousSensitive dataNextRetention and pruning
View source

On this page

  1. Failures
  2. Warnings
  3. --strict in a pipeline
  4. What it does not check
  5. Running it from code
  6. What to read next