...
Updates
1. 6 flat datasets are created by datateam (File’s are uploaded as it is, without any transformations and created as Flat Dataset)
2. Dashboards created by Content team and assigned Eldridge for review.
3. Based on feedback from Eldridge and approval from Emily, Decided to automate updates for datasets which used in building dashboards.
Automation Approach:
1. Data will be pulled at 6PM EST
2. Pull the LIVE data and compare the rows with already ingested data (using “ID” column). Drop the matching row from already ingested data and append LIVE data.
3. This will result in rows without duplicates.
Next task pending:
1. Ingest monthly screenshot as dataset?
2. What exactly client wants to see in monthly screenshot.
3. Knoema uses fixed dimension concept. If we get confirmation from client what exactly they need as dimension and what indicators they want to see then we can proceed
...