"Think of it as Google search for data.
The project is named after Norwegian explorer Roald Amundsen,
the first person to discover South Pole."
It includes three microservices and a data ingestion library.
- amundsenfrontendlibrary: Frontend service which is a Flask application with a React frontend.
- amundsensearchsearchlibrary: Search service, which leverages Elasticsearch for search capabilities, is used to power frontend metadata searching.
- amundsenmetadatalibrary: Metadata service, which leverages Neo4j as the persistent layer, to provide various metadata.
- amundsendatabuilder: Data ingestion library for building metadata graph and search index. Users could either load the data with a python script with the library or with an Airflow DAG importing the library.
Amundsen — Lyft’s data discovery & metadata engine – Lyft Engineering
Lyft Data Discovery with Tao Feng and Mark Grover - Software Engineering Daily
using Neo4j graph database
No comments:
Post a Comment