When there is a talk about what method to use in order to seamlessly sync data between applications, platforms and databases (in other words, application integration), APIs are often named as a today’s best-practices method for handling this. Unlike ETL-based integrations, which are not scalable, quite expensive and a bad fit for dynamic changes in business needs, APIs allow for considerably more visibility into data consumption as well as more flexibility per se.
Yet using APIs to sync data between applications and Co can sometimes become rather a wasteful and inefficient way of using resources. Here is when Webhooks come to the fore.
In this blog article, we would like to touch upon the difference between how APIs and Webhooks “work” when used to sync data between applications, why API-based integration can sometimes become quite messy, and when exactly Webhooks are actually a better fit.
When to prefer APIs for application integration, and why
As you know, in order to obtain data updates through API, you need to “call” it on a regular basis. So, one can say that APIs are a great fit for application integration when you know you will have a constant change of data.
The huge advantage of polling APIs for data is that you are the captain of your own ship. If you want more data, you just increase the paging size. If you know that right now you have a traffic jam somewhere on your server, then you, logically, reduce the paging size. This is practically an implicit flow control, which is very important when it comes to high data volumes and retaining the performance capability.[clickToTweet tweet=”When used in #applicationintegration, #APIs have the power of giving you flow control. #DevOps #EAI” quote=”When used in application integration, APIs have the power of giving you an implicit flow control”]
If you have rapidly changing data, for example, from outdoor sensors in an IoT integration scenario or shipment tracking systems in an eCommerce-related scenario, then you can be sure that data updates will be waiting for your call whenever you make it. In other words, you won’t find yourself empty-handed when polling API in such scenarios.
However, there is no guarantee that every call you make to API will have something ready for you on the other end. What if your data is not updated constantly and regularly? What if out of 8 API calls to a server, for example, to an inventory management system, only one gets an actual response? Just imagine this: Zapier is said to have made about 30 million poll requests over a substantial amount of time, out of which only 460,000 returned some results, which is only 1,5 percent.[clickToTweet tweet=”Wasteful #APIs: only about 1,5% of all poll requests return some results. #automation #DevOps” quote=”Wasteful APIs: only about 1,5 percent of all poll requests return some results.”]
In addition to that, as protection measures, most good APIs pose a rate limit on the number of calls per certain period of time, say, per hour. You certainly don’t want to waste these already limited resources by making empty calls.
This is when you can turn to additionally setting up Webhooks instead of simply polling APIs.
When Webhooks integration is a better fit to sync data between applications
The main difference between how Webhooks and APIs generally work is that, while APIs place calls without knowing whether they get any data update as a response or not, Webhooks receive calls through HTTP POSTs from external systems only when those have some data updates.[clickToTweet tweet=”#APIs place calls even if there is no data, #Webhooks sit and wait to be called instead. #automation” quote=”APIs place calls even if there is no data, Webhooks sit and wait to be called instead.”]
Then, Webhooks can “activate” the whole integration workflow to act on these updates. The main point here is, however, that this activation happens for a reason (i.e. because there is a real update to process), and not just because it’s time to make yet another check for an update.
Another good thing about Webhooks is that they provide other applications and services with (near) real-time information. This is possible through APIs only too, but it involves then an extra configuration step and is extremely resources-consuming. Webhooks make real-time really simple.
However, one of the dangers of using a Webhook is that you may never learn about any changes if the other system that sends updates to your Webhook goes offline at the wrong time for some reason.
In addition to that, you’ll have considerably less control over the flow of data because you’ll have to accept as much data as you get. Of course, there are ways to tackle this aspect effectively, for example storing data in a message queue. Yet this is by far not so efficient as being able to e.g. scale the paging size freely according to your immediate needs.
So, when would you prefer a Webhook to APIs in order to sync data between applications, platforms or databases?
I guess, one of the most obvious scenarios would be is when need real-time data updates but you don’t want to waste your resources by constantly polling APIs.
Another very common scenario is when your integration flow includes some external application or system that has a bad API or no API at all. A good example would be a logistics company that sends shipment status updates only via a message.[clickToTweet tweet=”#Webhooks are indispensable in #applicationintegration if #APIs are bad or absent. #automation ” quote=”Webhooks are indispensable in application integration if APIs are bad or absent”]
In this case, using a Webhook is probably the only way to sync data between applications automatically. The message will be sent directly to the Webhook that is configured specifically to “dissect” it and map its “contents” to another application, e.g. a fulfilment management system.
In a nutshell, Webhooks integration can be actually regarded as a perfect, and sometimes even essential complement to API-based application integration.
If you’re thinking about how to design a great Webhook or a great API for other developers to use them to effectively sync data between applications, or if you’re just wondering how to recognise such a Webhook or API, here are the first places to check out:
6 Characteristics That Make APIs Fit for Application Integration – our own quick guide about APIs and what they should provide to enable an easy integration with them
Webhooks do’s and dont’s: what we learned after integrating +100 APIs – a nice article focused on webhook features that are important for developers who want to use them to sync data between applications and Co.
What Are Webhooks? And Why Should You Get Hooked? – if you want to “backpedal” a bit and get some basics first, this article does an excellent job at explaining in simple terms what webhooks are and how they work – accompanied by some hands-on examples and a beautiful infographic.