As a platform team leader that caters to other engineers within the organisation, extensive documentation is what enabled me and my team to be productive.
I answer almost all questions posed to me with a simple have you read this (confluence link )
I saw my interviewers notes in our teams SharePoint.
Mixed emotions after knowing that i was almost not hired for being younger / having less experience than expected for the role.
7 years now in the same company and the same person has claimed that I hired him multiple times over the years.
For big data systems, CSV is a arguably worse format
Row wise storage doesn't provide compression benefits like columnar storage which can significantly reduce storage needs.
Data could contain the same delimiters ( comma, newline ) as used by the parsers and introduce errors in computations. binary formats( parquet/ORC) can eliminate this issue while providing other benefits as well.
A data quality framework like deequ should help catch errors in data before you introduce the same to downstream applications in ELT.(not infalliable )
For ELT processes, always filter first before other processing.
Plan your partitions according to expected querying patterns. If reports are run country wise , then country is a good partition key or if date wise then date is a good parition key.
Be aware of data skews , you might introduce skew during partitions ( ex : partitioning by country and a few countries have large number of records compared to others.
You should also see the pathetic copycat feature YouTube has created called the shorts. Facebook's attempt with short videos and reels is not much better either.
Why does every company feel the urge to copy this and also fail so badly at this
YouTube shorts broke my workflow. My son likes to make short videos of himself singing songs or doing funny things and then we watch them on the TV together. YouTube retroactively made all of our videos “shorts” and then says you can’t cast shorts to the TV.
Are youtube shorts a failure? They seem to be immensely popular in terms of views/engagement.
Reminds me of how people would deride Instagram for copying Snapchat stories, yet far more people probably use Instagram stories than snapchat and are perfectly satisfied with the UX despite it being a blatant copycat.
>Are youtube shorts a failure? They seem to be immensely popular in terms of views/engagement.
I just think that's only because "the algorithm" prioritizes shorts at the moment to try and push them and make folks even bother making them. Once YouTube starts acting neutral towards them I don't think they'll hold up.
I've never come across a short intentionally and after seeing quite a few I haven't tried to find further ones (and outside of mobile it's quite difficult to do so).
That said of course, I'm probably not in the demographic shorts plays to either.
Because sometimes it works. Instagram stories being a great example of this. Though once the functionality of an app gets diluted enough, like with Instagram and Facebook, each additional feature becomes less competitive.
I answer almost all questions posed to me with a simple have you read this (confluence link )