Your Data Place Projects - Shareable Common Libraries

-

Sharing libraries is core to Pythons swift useability, but Python on steroids is the ability is to share them in every project, with one click.

One big feature I want to have built out before initial launch of Your Data Place is the ability to not only assemble Python projects in the cloud using a smart server-first file-system, but also the ability to flag files (then folders later) as "common SDKs", SDK being a "software development kit" which will be shareable from any form of project in Your Data Place.

This means one major thing: your functions become importable anywhere, just like your datasets, and your functions/modules basically become a form of data.

Example use-cases for this feature

The reality is there's no use-case which will come anywhere near what we'll (cause I use this platform) end up using it for. A few basic use-cases I'll be using it for are:

  • Creating short and sweet scrapers which can be orchestrated in multiple projects.
  • Create common networking protocols or requests which can be invoked from multiple projects to send data somewhere.
  • Create API wrappers which serve for the most common calls in your organization/business.
  • Create email template functions with specific business HTML which can be called in multiple projects for marketing or business logic.

How will you take this feature to another level for developers?

I've got a few ideas for how to take this new feature, and attempt to push it to its limits and create a completely progressive experience. Some ideas as of now are:

  • The quick ability to use relative paths, to create atomic SDKs which work in the project they sit in, as well as the projects they don't even expect to be in yet.
  • YDP atomicism helper - I plan to write some smart hacky Python in the YDP lib to make the imports in the code layer, and not in the UI layer. The reason is because as I've stated before, is because I believe more can be expressed in code by providing SDKs than in UIs. Anyway... I plan to make it possible to expose scripts as shareable using a simple UI flag, then importable using the common YDP library at runtime. It may be as simple as ydp.common.script_name(1, 2).

What on earth is atomic?

Just a buzzword really. All it means is that the developer/programmer has decided he'd take some more time in making sure his code is able to be used in multiple areas. I use the idea of atomic code since I found the useability/repeatability a life-saver after learning React/NextJS and TailwindCSS (which is what YDP's UI is built in).


Comments

Post a Comment

What did you think about this post? Do you have any ideas you think would be helpful?

Popular posts from this blog

Shareable data and code in YDP Projects

-
Image
Datasets, everywhere... One major ability I want in YDP is the ability to create datasets that can be interpreted as global, reproducable datasets which were created as part of a dataset generator. Although I presume datasets will primarily be used inside their own projects, I think there's great value in the ability to make these datasets global, and accessible by any other project. The Problem At the moment, the problem is in the implementation. This was no issue when YDP was a sandbox test project in my local filesystem, where I could easily WRITE a file here, then READ a file here under the same name. But, in a real production architecture, we're using Docker as of this moment to manage our independent filesystems. When it comes to being able to have a dataset be generated in Project 1's file system (FS) then wants to be used in Project 2's FS, what has to occur is something along the lines of the following: Flag a Python script as a generator for a dataset. Execute
Read more

Siesta of YDP

-
Unfortunately, I've had to take a siesta in Your Data Place. I'm still extremely passionate, but most of my spare time has been consumed by a half-work half-play project which is niche in my industry which will allow me some more future freedom if it ends up working correctly. I'm using some of the innovations I've outlined here in that project, to make it something completely unique and separate to the current competitors. It's in the pharmaceutical space, and I may start to think through concepts in this blog, as some of the ideas are quite abstract but need to be represented simply and easily to the users. In the time since I've last written, I have not learnt Rust to a proficcient level. I've learnt Firebase and become better at frontend web design as well as better at software application building using Python. I will return to Rust, and YDP, as these are my raw passions. I believe my experience in this next project will greatly inspire more ideas in YD
Read more

The start: a week of Rust

-
Image
I've taken a small break to focus on my primary work obligations, and in that time I've had lots of time to ponder. During this time, I flew to Tasmania (737-800) and read through a few preliminary chapters of Manning Publications'  "Math for Programmers"  (Paul Orland). This book is a real workhorse for programmers who want and know  they wish to be able to implement more mathematics into their code, but find the conventional teaching methods difficult to fast-track. Home now, I plan to spend a week giving Rust a real go from the other Manning Publications book, "Rust in Action"  (Tim McNamara). In this time, I'll learn everything with a noble aspiration: "How could this benefit the users of YDP?" (as well as my primary work, and other interests of mine like genetics). Rust is fast, but from what I've experienced, it's cracked and goated with the sauce. What I find interesting about Rust, is the author's intent is as clear as d
Read more