My thoughts on data platform bolt-ons and how they may apply to Your Data Place

-

When I set out to build Your Data Place, I didn't want to build a conventional data platform, but wanted to get away from the trend-following practices that all data platforms seem to adhere to at the moment. One of those which I didn't seem fond of was bolt-ons, as they feel more like a hosting offer, and the bolt-on apps can quickly become obscure.

I've started reassessing though, as I've begun to learn Docker. With Docker, I've realized the ability to create images, which serve as shareable layers to runtimes. This means I can create isolated, safe, groups of programs which can run, and not use up precious chunks of RAM more than they need. From there, now realizing the Python runtimes will HAVE to be in Docker to offer the functionality I want, I realized something further: so could bolt-ons.

I had this concept in my head that I could offer a semi-database service to the YDP Python SDK, where developers could upload/retrieve data using some sort of syntax that follows:

ydp.db.insert("users", {username: "jackhales", password: "powpow"}, async=True)

I quickly wrote off the idea, because what am I then trying to do, build a new database technology? No one needs that, and it would suck from a efficiency/scale standpoint! And I don't want to be a database company beyond the basic key-value use-cases.

But now, armed with the knowledge of Docker and how, say, a MongoDB or MySQL container can sit on my server once, and then have a full container using those images and have resources be fully utilized to the inch for users? That's VERY interesting.

A user could (theoretically) click some page that sets up some sort of database management technology. From there, each project they create may have an option for the use of a database (to share them), and so-forth. I'll come back to this idea in the future to expand on it further.


Comments

Post a Comment

What did you think about this post? Do you have any ideas you think would be helpful?

Popular posts from this blog

Shareable data and code in YDP Projects

-
Image
Datasets, everywhere... One major ability I want in YDP is the ability to create datasets that can be interpreted as global, reproducable datasets which were created as part of a dataset generator. Although I presume datasets will primarily be used inside their own projects, I think there's great value in the ability to make these datasets global, and accessible by any other project. The Problem At the moment, the problem is in the implementation. This was no issue when YDP was a sandbox test project in my local filesystem, where I could easily WRITE a file here, then READ a file here under the same name. But, in a real production architecture, we're using Docker as of this moment to manage our independent filesystems. When it comes to being able to have a dataset be generated in Project 1's file system (FS) then wants to be used in Project 2's FS, what has to occur is something along the lines of the following: Flag a Python script as a generator for a dataset. Execute
Read more

Siesta of YDP

-
Unfortunately, I've had to take a siesta in Your Data Place. I'm still extremely passionate, but most of my spare time has been consumed by a half-work half-play project which is niche in my industry which will allow me some more future freedom if it ends up working correctly. I'm using some of the innovations I've outlined here in that project, to make it something completely unique and separate to the current competitors. It's in the pharmaceutical space, and I may start to think through concepts in this blog, as some of the ideas are quite abstract but need to be represented simply and easily to the users. In the time since I've last written, I have not learnt Rust to a proficcient level. I've learnt Firebase and become better at frontend web design as well as better at software application building using Python. I will return to Rust, and YDP, as these are my raw passions. I believe my experience in this next project will greatly inspire more ideas in YD
Read more

The start: a week of Rust

-
Image
I've taken a small break to focus on my primary work obligations, and in that time I've had lots of time to ponder. During this time, I flew to Tasmania (737-800) and read through a few preliminary chapters of Manning Publications'  "Math for Programmers"  (Paul Orland). This book is a real workhorse for programmers who want and know  they wish to be able to implement more mathematics into their code, but find the conventional teaching methods difficult to fast-track. Home now, I plan to spend a week giving Rust a real go from the other Manning Publications book, "Rust in Action"  (Tim McNamara). In this time, I'll learn everything with a noble aspiration: "How could this benefit the users of YDP?" (as well as my primary work, and other interests of mine like genetics). Rust is fast, but from what I've experienced, it's cracked and goated with the sauce. What I find interesting about Rust, is the author's intent is as clear as d
Read more