›
byrcsc/laravel-data-sync · 2.x
Configure sync classes, disks, queues, connections, and retention windows.
php artisan vendor:publish --tag="data-sync-config"config/data-sync.php holds infrastructure only, which classes are
registered, which disks, which queue, which retention windows. Behavior belongs
on the definition.
return [
'syncs' => [],
'connections' => [],
'staging' => [
'disk' => 'local',
'path' => 'data-sync/staging',
],
'archive' => [
'disk' => 'local',
'path' => 'data-sync/archive',
],
'failed' => [
'disk' => null,
'path' => 'data-sync/failed',
],
'queue' => [
'connection' => null,
'queue' => null,
],
'register_schedules' => true,
'overrides' => [],
'retention' => [
'runs_days' => 90,
'row_failures_days' => 30,
'archive_days' => 30,
'failed_files_days' => 14,
],
'capture_failed_payloads' => true,
'stalled_watermark_runs' => 5,
'tables' => [
'files' => 'sync_files',
'runs' => 'sync_runs',
'row_failures' => 'sync_row_failures',
'watermarks' => 'sync_watermarks',
],
];| Key | Default | Purpose |
|---|---|---|
syncs | [] | Registered definition and definition provider class names |
connections | [] | Inline Flysystem configurations for Source::connection() |
staging.disk / staging.path | local, data-sync/staging | Where discovered files are copied for processing |
archive.disk / archive.path | local, data-sync/archive | Where processed files are kept |
failed.disk / failed.path | null, data-sync/failed | Where failed files are preserved; falls back to the staging disk |
queue.connection / queue.queue | null | Default routing for package jobs |
register_schedules | true | Whether the package registers scheduled commands |
overrides | [] | Per-definition enabled, schedule, queue, chunk_size |
retention.runs_days | 90 | Days finished runs are kept |
retention.row_failures_days | 30 | Days failed-row records and payloads are kept |
retention.archive_days | 30 | Days archived files are kept |
retention.failed_files_days | 14 | Days preserved failed files are kept |
capture_failed_payloads | true | Global failed-row payload capture toggle |
stalled_watermark_runs | 5 | Runs without completed before the stalled-watermark warning |
tables.files / .runs / .row_failures | sync_files, sync_runs, sync_row_failures | Table names, read by the migrations and the models |
tables.watermarks | sync_watermarks | The table that holds each API and database sync's watermark |
'syncs' => [
App\Syncs\ProductsSync::class,
App\Syncs\StatementsTransfer::class,
],Each entry must name an existing class that extends SyncDefinition or
TransferDefinition, or implements Contracts\DefinitionProvider. No two
definitions may resolve to the same name.
Violations throw an InvalidConfigurationException.
The package checks a class entry at boot. It never calls a provider at boot,
because a provider may read tables that do not exist yet, such as during a
first migrate. It checks the definitions a provider returns when a command,
a queue worker, or the scheduler first resolves them. See definition
providers.
Inline Flysystem configurations, built on demand by
Source::connection('<key>'):
'connections' => [
'supplier-sftp' => [
'driver' => 'sftp',
'host' => env('SUPPLIER_SFTP_HOST'),
'username' => env('SUPPLIER_SFTP_USERNAME'),
'privateKey' => env('SUPPLIER_SFTP_KEY_PATH'),
'root' => '/outbound',
],
],Each entry must be an array with a non-empty driver. An ftp or sftp driver
requires its Flysystem adapter, league/flysystem-ftp or
league/flysystem-sftp-v3, and throws at boot when it is missing. Keys beyond
driver are passed straight to the driver, exactly as in
config/filesystems.php.
All three name a disk from config/filesystems.php and a path root within it.
failed.disk falls back to the staging disk when it is null.
See installation for what each holds, and queues and
workers for why a local
staging disk is a single-machine configuration.
'queue' => [
'connection' => 'redis',
'queue' => 'imports',
],The default routing for every package job. Resolution order, most specific first:
Connection, overrides.<name>.queue.connection, then the definition's
queueConnection(), then queue.connection.
Queue, overrides.<name>.queue.queue, then the definition's queue(),
then overrides.<name>.queue when it is a bare string, then queue.queue.
Per-definition operational settings, keyed by definition name. Only four keys are accepted; anything else throws at boot, which is deliberate, behavior belongs on the class, not in config.
'overrides' => [
'products' => [
'enabled' => true,
'schedule' => '0 * * * *',
'queue' => ['connection' => 'redis', 'queue' => 'imports'],
'chunk_size' => 500,
],
],| Key | Accepts | Notes |
|---|---|---|
enabled | true / false | false removes the definition from sync:run with no name, and from scheduling |
schedule | A cron expression, or false | Wins over the definition's schedule(); false disables scheduling for it |
queue | ['connection' => …, 'queue' => …] or a queue name string | See above |
chunk_size | An integer of at least 1 | Wins over the definition's chunkSize() |
The key must name a registered definition. An override for an unknown name is a boot-time error rather than a silent no-op, which catches renamed definitions. When a definition provider is registered, the package checks override names once the provider has returned its definitions, not at boot.
true by default. It registers sync:prune daily, plus sync:run <name> for
every enabled definition that declares a schedule() method or has a cron
override. Set it to false to register nothing and wire the commands up
yourself.
Four independent windows, in days. null disables one. See retention and
pruning.
1.0 called this key store_failed_payloads. The package fails at boot while a
published config still has the old name. See upgrading from
1.0.
The global switch for recording the raw payload of a failed row. It wins over a
definition's captureFailedPayloads(), so turning it off here turns it off
everywhere. See sensitive data.
5 by default. An API or database sync moves its watermark only when a run
finishes as completed. When a definition's last N runs all finished without
completed, sync:status and sync:doctor warn that its watermark has
stopped moving. The value must be an integer of at least 1. See when the
watermark moves.
'tables' => [
'files' => 'data_sync_files',
'runs' => 'data_sync_runs',
'row_failures' => 'data_sync_row_failures',
'watermarks' => 'data_sync_watermarks',
],Read by both the migration and the models, so custom names work without subclassing anything.
Publish the config before the migration if you intend to change these. The migration reads the values at publish time; changing them afterwards means renaming tables by hand.
The whole file is validated when the package boots, and again by
sync:doctor. An invalid configuration throws
InvalidConfigurationException immediately rather than during a 2 a.m. run:
syncs entries exist, are the right type, and have unique names. A name
that a definition provider returns may use only letters, digits, hyphens, and
underscores, and starts with a letter or digit.overrides keys name registered definitions and use only the four accepted
settings, with a valid cron expression and a positive integer chunk size.retention values are null or non-negative integers.connections entries are arrays with a driver, and their adapters are
installed.stalled_watermark_runs is an integer of at least 1.The package ships no env() calls of its own, the published config is plain
values. Wire your own where they help:
'staging' => [
'disk' => env('DATA_SYNC_STAGING_DISK', 'local'),
'path' => env('DATA_SYNC_STAGING_PATH', 'data-sync/staging'),
],Remember that config:cache freezes the result, so env() outside config files
returns null in production.