My thoughts on data platform bolt-ons and how they may apply to Your Data Place

When I set out to build Your Data Place, I didn't want to build a conventional data platform, but wanted to get away from the trend-following practices that all data platforms seem to adhere to at the moment. One of those which I didn't seem fond of was bolt-ons, as they feel more like a hosting offer, and the bolt-on apps can quickly become obscure.

I've started reassessing though, as I've begun to learn Docker. With Docker, I've realized the ability to create images, which serve as shareable layers to runtimes. This means I can create isolated, safe, groups of programs which can run, and not use up precious chunks of RAM more than they need. From there, now realizing the Python runtimes will HAVE to be in Docker to offer the functionality I want, I realized something further: so could bolt-ons.

I had this concept in my head that I could offer a semi-database service to the YDP Python SDK, where developers could upload/retrieve data using some sort of syntax that follows:

ydp.db.insert("users", {username: "jackhales", password: "powpow"}, async=True)

I quickly wrote off the idea, because what am I then trying to do, build a new database technology? No one needs that, and it would suck from a efficiency/scale standpoint! And I don't want to be a database company beyond the basic key-value use-cases.

But now, armed with the knowledge of Docker and how, say, a MongoDB or MySQL container can sit on my server once, and then have a full container using those images and have resources be fully utilized to the inch for users? That's VERY interesting.

A user could (theoretically) click some page that sets up some sort of database management technology. From there, each project they create may have an option for the use of a database (to share them), and so-forth. I'll come back to this idea in the future to expand on it further.


Comments

Popular posts from this blog

Siesta of YDP

Shareable data and code in YDP Projects

The start: a week of Rust