›
›
›
  1. docs
  2. ›
  3. byrcsc/laravel-data-sync
2.x
Browse documentationOpenClose

Getting started

  • Introduction
  • Installation and setup
  • Quick start
  • Upgrading from 1.0

Core concepts

  • File identity and runs
  • Sync definitions
  • Sources and connections
  • File formats and readers
  • Field mapping
  • Matching and writing
  • Validation and hooks

Running syncs

  • Running a sync
  • Queues and workers
  • Scheduling

APIs and databases

  • Syncing from an API
  • Syncing from a database
  • Definition providers

Transfers

  • File transfers

Operations

  • Run history and status
  • Events and listeners
  • Failures and retries
  • Sensitive data
  • Health checks
  • Retention and pruning

Reference

  • Configuration
  • Console commands
  • Testing
  • Troubleshooting

Getting started

  • Introduction
  • Installation and setup
  • Quick start
  • Upgrading from 1.0

Core concepts

  • File identity and runs
  • Sync definitions
  • Sources and connections
  • File formats and readers
  • Field mapping
  • Matching and writing
  • Validation and hooks

Running syncs

  • Running a sync
  • Queues and workers
  • Scheduling

APIs and databases

  • Syncing from an API
  • Syncing from a database
  • Definition providers

Transfers

  • File transfers

Operations

  • Run history and status
  • Events and listeners
  • Failures and retries
  • Sensitive data
  • Health checks
  • Retention and pruning

Reference

  • Configuration
  • Console commands
  • Testing
  • Troubleshooting

byrcsc/laravel-data-sync · 2.x

Upgrading from 1.0.

Rename the 1.0 classes, methods, and config key, run the 2.0 migration, and drain the queue before you deploy.

Version 2.0 adds API syncs. It also renames classes and methods whose 1.0 names described the wrong thing. The package removes the old names instead of deprecating them, so code written against 1.0.0 needs the changes on this page before it runs on 2.x.

Version 2.1 adds database syncs and definition providers. Both are additions, so an application on 2.0 upgrades to 2.1 with composer update and no other change.

Do the steps in this order:

  1. Rename the classes and methods in your definitions.
  2. Rename the config key.
  3. Let the queue drain.
  4. Deploy and run the migration.

Rename classes and methods

1.02.x
Definitions\ReaderDefinitions\Format
Enums\WritePathEnums\WriteStrategy
SyncRun::$write_pathSyncRun::$write_strategy
Data\SyncReportData\RunResult
SyncReport::$writePathRunResult::$writeStrategy
Source::archiveTo()Source::copyTo()
Source::moveSourceTo()Source::moveTo()
Source::deleteFromSource()Source::deleteOriginal()
Source::sourceArchivePath()Source::copyPath()
Source::sourceMovePath()Source::movePath()
Source::deletesSource()Source::deletesOriginal()
Source::directory()Source::pathName()
TransferDefinition::matchFilename()TransferDefinition::filenamePattern()
Field::source()Field::from()
SyncDefinition::withoutPayloadCapture()SyncDefinition::withoutFailedPayloadCapture()

Most definitions only need the format() change. Contracts\Reader keeps its name, and only the factory class is renamed, so a definition no longer needs an import alias:

// 1.0
use ByRcsc\LaravelDataSync\Contracts\Reader as ReaderContract;
use ByRcsc\LaravelDataSync\Definitions\Reader;

public function format(): ReaderContract
{
    return Reader::csv();
}

// 2.x
use ByRcsc\LaravelDataSync\Contracts\Reader;
use ByRcsc\LaravelDataSync\Definitions\Format;

public function format(): Reader
{
    return Format::csv();
}

RunReport::$runs now holds RunResult objects. Code that reads $report->runs[0]->created keeps working. Code that type-hints SyncReport or reads writePath needs the new names.

Rename the config key

In config/data-sync.php, rename store_failed_payloads to capture_failed_payloads:

'capture_failed_payloads' => true,

The package fails at boot while the old key is present:

The [data-sync.store_failed_payloads] config was renamed to [data-sync.capture_failed_payloads] in 1.1. Rename it in config/data-sync.php.

The package does not ignore the old key, because the new key defaults to true. An application that had turned payload capture off would start storing row payloads again without notice.

If you use custom table names, add the new watermarks key to data-sync.tables before you run the migration:

'tables' => [
    'files' => 'data_sync_files',
    'runs' => 'data_sync_runs',
    'row_failures' => 'data_sync_row_failures',
    'watermarks' => 'data_sync_watermarks',
],

Without it, the table is named sync_watermarks.

Let the queue drain

A ProcessChunkJob queued by 1.0.0 carries a selectedRows property that 2.0 renamed to keptRows. A 2.x worker that picks up one of those jobs fails it.

Before you deploy:

  1. Stop new syncs from starting. Stop the scheduler, or set enabled to false in data-sync.overrides for each definition.
  2. Wait until sync:status shows no pending or running runs and the queue holds no package jobs.
  3. Deploy.

If you deployed with package jobs still queued, check sync:status for runs that did not finish and retry them.

Run the migration

php artisan vendor:publish --tag="data-sync-migrations"
php artisan migrate

Publishing again adds upgrade_data_sync_tables_to_v2 and leaves the migration you published for 1.0 as it is. The new migration:

  • creates the sync_watermarks table;
  • adds a nullable pending_watermark column to sync_files;
  • renames sync_runs.write_path to write_strategy.

Run it even if you have no API or database syncs. 2.x writes write_strategy on every run, so syncs fail until the column is renamed. Version 2.1 adds no migration of its own.

sync:doctor reports a missing sync_watermarks table only when an API or database definition is registered.

Output that changed

Scripts that parse command output need two updates:

  • The sync:status column Status · path is now Status · strategy.
  • The line after the table now reads ... used the bulk write strategy. instead of ... used the bulk write path.

What to read next

  • Syncing from an API for the feature 2.0 adds.
  • Syncing from a database for the feature 2.1 adds.
  • Health checks to verify the installation after the migration.
PreviousQuick startNextFile identity and runs
View source

On this page

  1. Rename classes and methods
  2. Rename the config key
  3. Let the queue drain
  4. Run the migration
  5. Output that changed
  6. What to read next