Have you gotten data from external sources through API's recently? You send a GET or POST request to a URL with your authentication key and a request for data, and back comes XML or JSON. You parse that XML or JSON, store it in your database, and keep going.
You probably do this so much that you don't even realize there might be a better way. Instead, you think it's normal that:
- You have to build a different API for every source.
- The API is periodically unavailable, so you're down as well.
- The API could change and break your app.
- The API could simply be taken away from you, to "upgrade to a better API", to "improve security", or for some other perfectly good reason that leaves you scrambling for an alternative.
- You have to store a local copy of the data retrieved from the API, in case it goes away.
- You had to work out a procedure for audit and verification of the data obtained from the API, in case anybody thinks you might've altered or changed it.
- You have to keep up with security to protect your authentication keys.
While a new and better Javascript web framework seems to pop up every week, not much has changed in API integration land for years. We're all so used to doing it this way, we think this is just our lot in life.
Unfortunately, these API integrations are expensive and time consuming to build and maintain, so they don't scale in scope. When we're talking about building for Carbon Accounting and Certification Working Group and potentially integrating data from a large number of sources, it's just not a good way to do it.