Manage records from a system which does not provide time-based change tracking or webhooks

Ideally, every business system should provide a mechanism to filter or sort data records by the time of their creation and change. Otherwise, integrating systems will be very resource-heavy because you would have to move all available datasets from one application to another every time at least one data record was created or updated. In well-designed APIs, for example, such mechanism is ensured by pagination and modification timestamps.

But every once in a while, integrators have to deal with systems – oftentimes, legacy ones, – that don’t have a built-in time-based change tracking or allow to set up webhook calls for each freshly created or updated data record.

To enable integration with such systems, we’ve developed a component that learns about the creation of new records, updates to existing ones, and deletions of records even in the absence of time-based change tracking mechanisms – the Delta Detection component.

About Delta Detection connector

This is elastic.io’s own private connector, designed to function with all other connectors in your integration workflows. This connector is part of a standard platform installation.

The Delta Detection connection currently only provides the action functionality, which means that you will be able to select it only after there is at least one connector already available in your flow.

Follow the Documentation link above to find what you need to do to set it up. You will also find there the expected input and output metadata.

You cannot edit or expand this connector yourself since it is covered by a proprietary license. Please contact our pre-sales team at ps[@]elastic.io if you have any questions.

PLEASE NOTE:You need to be registered on the elastic.io integration platform in order to use the Delta Detection connector. You can sign up for a free 14-day trial of elastic.io here if you’re new to the platform.

How to add and use Delta Detection on elastic.io

Step 1
In your Dashboard or Flows section of the platform, click on “Add new Flow” and first, select the system of records that you would like to connect to the Delta Detection connector.

Once you’ve done this, you can select the Delta Detection connector.

Step 2
The way how Delta Detection works is that it pulls all records from a system on a regular basis, e.g. once a day, and then runs the entire dataset through its own filter. After that, the connector performs upsert operations (I.e. updating an existing record or if a record doesn’t exist, create a new one), in another system of your choice.

Illustration of a common mechanism of Delta Detection 

Step 3
In the configuration step, you will need to work with objectID and objectData parameters, which set unique identifier for storing object information at external object-storage and set object data that help detect changes.

For more details on how to work with the Delta Detection component, please check the dedicated Documentation page following the link above.