Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

How are you able to see a datasets lineage across storage types. For example how are you able to see that an s3 buckets files are the ancestor of some table in Postgres?




Oh I see it uses open lineage. I thought it was able to handle discovery

It can handle discovery within a plugin if the asset types are related. You can also manually add lineage via the UI or use Terraform to create lineage links via IaC. It's pretty complicated to automatically handle discovery of asset lineage, I'm yet to find a nice way of doing it that can work for many use-cases



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: