Retrieval sampling on open data

Goal:

Open data stored via Fil+ should be retrievable by anyone on the internet. Retrieval testing/sampling of open data onboarded will enable:

  • validation of work done by storage clients and SPs
  • identify high-trust clients
  • flag low-reliability SPs

KPIs (and links to dashboards):

  • Retrieval Bot stats: how many <cid, sp> pairs are being sampled (new dashboard?)
  • 100% of new open data DataCap applications should have public payload CIDs for retrieval testing
  • 100% of new open data DataCap applications have retrieval sampling done against SPs with results published periodically

Milestone exit criteria:

  1. May 25: Proposal + Solution design
    2. Propose standards for payload CID tracking
    Propose changes to DataCap app template that accounts for where CIDs will be shared and if bitswap/http retrieval is also in scope for testing
    Provide a path for DataCap applications to share payload CIDs
  2. Jun 5: Pilot
    3. Initial retrieval sampling on a subset of applications
    Validate E2E flow for results published via a bot to DataCap application
  3. Jun 19: Launch
    4. Stabilize and push on all open data applications
    Publish metrics and stats across program

Notes:

Twitter

Filecoin Foundation

FFDWeb

Filecoin