Despite August being one of the hottest summer months, we were busy building nice new features. Our goal was to give user a better visibility of the status of integration workflows, more transparency on containers and scheduling, and faster feedback to user actions.
In other words, we wanted to deliver you a better user experience.
And so here is the overview of what’s new on the elastic.io integration platform, and how it now allows you to be even more efficient in your work.
New Feature: Near Real-Time Update Notifications of the Run Tables
This feature offers a new way of communication with the browser. Typically, in order to get updates, one has to either click on a Refresh button or reload a page. We used to have such a button too.
Now your elastic.io dashboard page actively receives information from the platform, giving you automatic updates on your integration workflows.
With near real-time update notifications, every task performs reload automatically, while the live records tracker shows you how many data records are passing through your integration workflows. The task that has been executed the latest is always on top of the list in the run log.
With this new feature, the Refresh button, as you might have expected, has become obsolete and been, therefore, removed.
Now you can be sure that you’re always up-to-date on all the latest changes in your integration workflows.
Improvement: Tasks Status
This is my personal favourite one. The thing is that previously, there was not so much visibility as to what happened with your integration workflows, whether they were running or not. Surely, you could determine that based on certain information, but the process of accessing this information was quite far from being obvious at a glance.
The new visual element shows you immediately, as soon as you log in to the dashboard, what the status of your tasks is: Whether they are active and running, or inactive, or maybe sleeping. The latter, by the way, means that the task is active, but no containers are running at this precise moment. This is the case when tasks are scheduled to wake up and check for data in timed intervals, e.g. once an hour.
This major improvement to visibility leads us to the next feature.
New Feature: The ‘Run now’ Button
We have been receiving requests for this feature every now and then, and it’s easy to see why. Sometimes, you simply don’t want to wait until the next scheduled run and need to execute the task right now – e.g. while debugging errors. Earlier it was possible only if you opened your task and started it anew.
The new button ‘Run now’ allows you to execute integration workflows right in the dashboard at any moment, without having to revisit them.
Quick tip: You can see on the picture above, that if you hover your mouse over this button, you’ll see when the next run is supposed to happen. Pretty handy, ha?
Note: If your integration workflows start with the Webhook component, when you click on the Run now button, you will get an instruction to how you do this; in other words, what command line to use to run your webhook.
New Option: Enabling/Disabling Long-Running Tasks
You surely know about our long-running tasks, or as the official name goes, Real-Time Flows: Our Docker containers in this type of tasks don’t ever stop, enabling mega-fast data processing between the components. This is especially important when your workflows rely on the most recent data delivered as fast as possible.
As you might remember, Real-Time Flows used to be in beta, and you had to contact our support in order to have it activated for each single task of yours. Well, it’s in the past.
Now you can turn your regular tasks into long-running on your own with only one click. Just choose the “Real-time flow” option for the particular task, while it’s in the inactive mode:
You can see that your task has become a long-running one when there is a check next to “Real-time flow” and a cute rocket icon next to the name of your task.
Note please: Since containers in long-running tasks are always up, they will consume your quota faster than regular tasks. Just keep this in mind;-)
New Feature: Alerting
Last (but not least) new feature is Alerting. This is another feature for which we received lots of requests, because previously, if an error occurred there was no way to know that until you would log in to the dashboard.
Now, you can activate alerting specifically by choosing the „Subscribe to errors“ option for each task:
You will know that this feature is activated if there is a check mark next to it.
As soon as an error in your integration workflows occurs, you will receive a notification with a couple of details via email to the email address you’ve registered with on the elastic.io integration platform.
Now you might wonder how often you will get those error notifications, and whether you get 100 emails all of a sudden if your task, by any chance, has produced 100 errors. Rest assured, you will not. Well, not necessarily:) If the error message contains the same text, then you will get only one email regardless of the number of errors. And you will get that email only once an hour, even if your task is scheduled to run every three minutes.
You can always unsubscribe from Alerting by clicking on the „Subscribe to errors“ option one more time.
Please test this new feature out and let us know what you think of it.
Improvements & Enhancements: Miscellaneous
In addition to a set of new features and options, we also have a few improvements and enhancements.1 We have added an extra layer of security to our Webhook component. Now this component has also a field where you can enter your HMAC verification shared secret. This field allows to verify that the data that a webhook receives is coming from a particular user, and from anyone else. If the signature doesn’t concur, an error will be shown.
This field is not required, however, if you process specifically sensitive information, we recommend you use it.2 Our Step by Step feature is out of beta now and activated for everyone. Not only that, it also now supports static metadata. In other words, if your component has only field objects without any values, they will be shown in the Data Mapper just the same as values (this part was a bit problematic previously). 3 You must have already noticed our updated UI design. There are no major changes yet, for now you just have the menu on the left-hand side. But we have already received some positive feedback about this change. If you have one too, please do not hesitate to leave it below the post.
4 We did some major improvements to the Request/Reply component. Maybe you know that the older version of the component that enabled the Request/ Reply message exchange pattern had to be at the end of integration workflows.
Now you can add a component after the Response/Reply component. For example, in the sample integration workflow below, we are able to receive data from Salesforce and update the Database with this data almost simultaneously. This is possible due to the fact that, thanks to the Request/Reply component within the flow, we would receive Salesforce data via webhook without having to wait for the database to update.
Also, it is now possible to customize HTTP headers for the Request/Reply component while configuring it.
5 The last but not least, here is an important fix for our components in general. Previously, if a component was built or configured not entirely correctly so that it would eat up all the memory, there was no way to learn about that. Even the logs wouldn’t have shown the root of the problem; according to them, the component would have started as usual but wouldn’t have worked for some unknown reason.
Now it is fixed, both for Node.js and Java. The logs will show you exactly that a component is not running because it is out of memory.
Oh, almost forgot. We have also added support for OAuth1. You can read more about it in our documentation.
What to expect in September
Well, without giving out too much information: In September we are going to work on making the installation process of our platform considerably easier for other developers. In addition to that, we will continue on improving the new features and options issued in August.
As for other things to expect in September, as the Germans say “Lassen Sie sich überraschen” (which means literally “Let yourself be surprised”).