Day 3 - Data Science App in 45 days - Spec the idea please!!

Day 3 - Data Science App in 45 days - Spec the idea please!!

Tags
Engineering
Product
Published
Published July 20, 2022
Author
notion image

Productive afternoon

We was at this coffee that we’ve recently discovered. A nice located on Dien Bien Phu street. Seems like the busy streets cannot stop the coffee to be overpopulated and hustling & bustling, instead, it was a quite place, good to start a group discussion.
Today we have Nick (this guy a Rust genius) and Bach (a frontend genius) joined the session hosted by a hippie (which is me)

Decision Fatigue

Moving from the traditional MVC model to micro services requires huge amount of effort, at least not like what I expected. I thought that simply we need a frontend in either React/Vue or Svelte, then bunch of REST API at the backend to access and compute the resource at the backend, which can eventually be bridged together by using axios at the frontend to make API calls.
It was more than that. Just to name a few:
  • Should users' datasets be stored at the frontend? Since we can leverage the users storage
    • Pros:
      • textual datasets (especially for local apps) are not too costly in terms of passing around for computing and storing (250 pages of 4000 words only cost 1mb).
      • If we adopt this design, our backend would be much simpler, they just revolves around AI prediction services but not database resources, because data querying and manipulating now happen at the frontend.
      • No postgres + No turtoiseORM, nothing @ the backend, resulting much
    • Cons:
      • Leave data processing responsibility to the frontend, (maybe) faster to implement but not optimal in terms of time.
      • Cannot verify the correctness of data flow schema. Say, with FastAPI, we can clarify a schema and it will automatically validate the data using Pydantic.
      • Frontend designers are stressed because cannot focus on the design, backend engineers are stressed because cannot do real backend sh**..
  • Which features are prioritized for the POC?
    • Most needed and the easiest to implement: general charts for sentiment analysis and grouping, word cloud
    • Advanced features:
      • marketplace for charts
      • customizable ML models
  • … I’m sleepy now, maybe next time

Specification

After any ideas discussion should very well be followed by the technical specification. We spec-ed our previously ideas on how we will implement this, technically:
  • Decide first on which features to have
  • Agree on the user roadmap for the frontend
  • Agree on the API specs (json schema for input/output): which backend resources will be requested by frontend?
  • Start implementing
Spec-ing the idea really helps us stay alerted with assessing the feasibility of each features for the app.
Until next time…