How to organize APIs better to improve efficiency by 100% - Cloud Authentication Hell

Setting up enterprise-level APIs is a rough, and no-one enjoys it. It's not to say it's not necessary and required for a safe internet, but there's a level of work which stops us engineers from "quicking spinning up a cloud integration/pipeline in a few minutes" in the same way we'd do for a quick local shell script. The cloud is tricky. 

For my Python cloud data platform, Your Data Place, I'm doing a lot of planning around how it may be used and what it should be used for, which I need to stop doing. The reality is our platform will not be used for any given specific task, but whatever tasks are required to be streamlined from within a company. To do that most effectively, I must provide the most wanted to be used integrations for each project, and marry it with the ease of use of a simple Python library which you pip install, and quickly import and use.

So far, my list is limited in cloud (or shorthand) integrations which satisfy this need. For example, the main integration I've put in so far is Google related APIs. Of the Google APIs, I've put Sheets, Maps, Analytics and Gmail. You can see the UI component below:


On clicking a Google product for the first time, the user will be sent to an OAuth2 consent screen, which then syncs with the YDP backend server. This allows one to then import the YDP SDK in Python, and utilize the raw service provided by the API, or use the provided YDP "simple SDK" in order to do common jobs, quickly. For example, a Google Sheet with the ID "abcdefg" could be read into a Python list using:

import ydp
data = ydp.sheets.open("abcdefg")
print(data) # ["a", "b", "c"]

This has already been implemented, along with write functions, append funtions, edit functions, and so-forth. Improvements would include the ability to search by ID by default, but also add name="x" to search by the name of a sheet.

But what other implementations (Auth + SDK) would be helpful?

#1 - AWS Comprehend

The ability to quickly authorize or input your API credentials, then immediately import in the ability to comprehend text information (sentiment, keywords, etc) on existing datasets would allow developers to deploy smart pipelines and smart parsing capabilities. For example:

import ydp
comments = ydp.sheets.open(name="Blogger Comments")
for comment in comments:
    negative_sentiment = ydp.comprehend.sentiment(comment).negative
    print(f"'{comment[0:10]}' negative sentiment: {negative_sentiment}")

Which would then allow developers to ignore the headers required for "bringing in comprehend", and instead focus on their own business logic. Write a Twitter extractor, Reddit extractor, and plug into comprehend will become easier than writing the hard extract logic.


Comments

Popular posts from this blog

The Petabyte Project, the Most Valuable Idea

Shareable data and code in YDP Projects

Siesta of YDP