Repeated SnapDeals

[Kubuxu]

This is a proposal to allow ReplicaUpdate to be performed multiple times on the same sector.

Motivation

SnapDeals shipped with limited capability due to limitations of the larger system. These limitations are mitigated with the introduction of FIP-0045. Additionally, the ability to perform SnapDeals upgrade on the same sector multiple times would allow Storage Providers to accept a wider variety of deal terms and make sectors that went through their first SnapDeal still useful for data storage after the initial deal expires.
It is the last capability missing from enabling the full utility of all sectors in the network.

Proposal

Allow sectors which underwent the SnapDeals process in the past to repeat it. During the design of SnapDeals, this possibility was taken into account by keeping the OrginalSealedCID in SectorInfo, which is required for Repeated SnapDeals to function.

There are two possibilities for the scope of this proposal:

1. Ability to perform a SnapDeal on a sector where all deals and FIL+ claims have expired.
2. Ability to perform a SnapDeal on a sector where there are still active deals and claims by preserving active deals and claims.

The second one has the added complexity of having to verify that the data corresponding to active deals and claims was kept within the sector after a ReplicaUpdate.

Fortunately, the PieceCIDs and sizes of deals and claims are kept on-chain to facilitate such constraint.
New methods for Market and Datacap actors would have to be introduced to facilitate the status and data description access during the ReplicaUpdate process. This information is already planned to be at least partially exposed for the purpose of user-programmable actors interop.

As a way to limit the complexity of the ReplicaUpdate method, the expiration of the sector has to already suit new deals and allocations. The SP is free to perform sector extension before the ReplicaUpdate as long as it complies with constraints given by active FIL+ claims.

From the perspective of proof production and encoding, the ReSnap process decodes the existing data from the sector, a trivial operation of if an extra copy of the deal data was kept. The result of this operation is sector data corresponding to a CC sector which underwent the SnapDeals process. Then the SP packs new and existing (if we adopt scope 2) deal data into the clear sector (the position of the data within the sector can be chosen by the SP) and produces a ReplicaUpdate proof.

Repeated SnapDeals use the same proving circuit as the original SnapDeals, avoiding the need for a new proof type and trusted setup. The computation time for the proof is the same as regular SnapDeal. This also means that SnapDeal introducing small changes to the sector will have the same computational cost as SnapDeal affecting the whole sector.

During the process of the repeated ReplicaUpdate, for a short time, the SP will have to keep sector information for three copies of the sector, the currently active sector with user data, the clear sector and the new sector with new data. The clear sector can be removed after then the new sector and proof are produced. The "currently active sector" can be removed after the proof for the new sector is provided on-chain, validated and the 900 epoch delay period expires.

[ZenGround0]

This information is already planned to be at least partially exposed for the purpose of user-programmable actors interop.

👍

We should view it as a requirement on the first User Programmable markets FIP that this is supported. All markets implement a standard interface that takes in the sector identifier and the new CommD and return success/fail. I don't think we need old CommD but it's worth thinking about that and if so how we support proving that old CommD is correct( related #496). Knowing which markets to call based on a sector is a bit tricky but I think the ideas from #530 will serve us well: abstractly keep a commitment to the notifees and have SP provided data. For first implementation we can simply keep notifee list on the sector infos and carefully keep this detail out of all APIs.

[ZenGround0]

For QA power I think we can get away with no changes to verifreg actor even in scope 2. There are three different kinds of pieces with datacap: pieces being added, pieces being removed, pieces staying in place. We handle all three cases as of nv17 in extension + prove commit / replica update.

[jennijuju]

In this post #554 (reply in thread) it’s suggested that pre shark sectors cannot be snapped. Im having some trouble with understanding the complexity and the gap - @Kubuxu have you any thoughts into it? If so mind shedding me some light?

[anorth]

These limitations are mitigated with the introduction of FIP-0045

FIP-0045 fixes the main limitation, which was complex QA power calculation. But it only applies to new sectors.

An opt-in migration from old to new may be possible, but has limitations.

[jnthnvctr]

summarizing some addt'l threads and comments that may be useful in shaping discussion here:

• there's a number of paid storage on-ramps coming to market that would like to be able to resnap into sectors
• specifically this would simplify billing with SPs - for many on-ramps theyre paying SPs for capacity, and reselling that into clients
• the challenge is if you can't lifecycle deals inside of a snapped sector - the onramp needs to either (a) force customers to commit to 6 mo+ deals (fine for some class of enterprise deals, much harder for more dynamic data or for folks building saas-y products) (b) just buy more and more capacity (much less financially viable for an onramp, leads to less data directly shipping into sectors) (c) don't auto lifecycle deals into sectors (bad for the network, more expensive but more flexible for clients)

It's worth noting that there seems to be more recent discussions about deletion (intentionally overwriting data) which can help with some enterprise conversations as well. Folks from the client-wg might be able to weigh in (I will link them to this comment to thread) more details.

@anorth has put together a more encompassing revamp of the deal / sector relationship here, related to a talk he gave here.

I think there's probably two contexts that are worth discussing (and would love to help motivate):

• Short term: how do we ship something that unblocks enterprise on-ramps that are targeting bringing paid deals into the network (knowing many of these are going live in q1 and ramping) => this is important for bringing cashflows into individual providers (but also more broadly can help grow the ecosystem writ large). Would love to figure out what the MVP version of unblocking teams here are given the importance of timing. I think the challenge flagged is how we can balance client expectations around their data not being over written.

• Medium / Longer term: I think there's a lot of good insights for simplification Alex fleshed out - would be keen to see how we can orient the shorter term thing to ladder into the longer term direction (if possible)

cc @jbenet @laudiacay @nicola

(some other places discussions have been happening #sector-resnap-wg on FIL Slack, this thread in #fil-protocol )

[laudiacay]

you summarized this well @jnthnvctr

until this is fixed, we have to lock all cold data in for at least 6 months at a time (or whatever our default deal period is, determined by Banyan's network sealing capacity and burstiness of client demand- for now we expect 6 months).

this forces us to, instead of doing "rolling backups" where the latest version of customer data gets sealed to Filecoin, have the customer opt into sending data back to Filecoin by clicking "snapshot archive to Filecoin" to seal the current version. It makes things manual for the customer, which is behavioral friction that will result in less data coming onto Filecoin.

to be clear, this does not cause any problems for our hot storage, hot proofs, or ability to onboard relatively-static data onto Filecoin- we can still get (substantial) cashflows into individual providers, and we will still bring good amounts of data onto filecoin, but we will not sell as much sealed Filecoin capacity as we possibly could (especially for a substantial portion of users who will use us as drop-in replacement to S3).

we'd prefer to do "rolling/automatic backups", but that's not currently workable without either:

1. exposing high-write-frequency users to shockingly high bills (every block that ever enters the system stays stored for at least 6 months, no garbage collection until the deals expire, every file edit spawns a 6-month charge), or
2. leaving Banyan footing the bill when a user leaves a bunch of stuff on Filecoin that we can't garbage-collect for 6 months.

second, without resnapping, we can't hard-delete at all. this is extremely unusable for enterprise customers. realistically, "we deleted the encryption key and scheduled wiping it in a garbage collector within 24 hours" is acceptable (this is approximately what AWS does), but "it's going to be here for 6 more months" is not acceptable. paying miners out to get slashed is also not affordable.

adding these changes will let us flexibly garbage-collect as per user block-lifecycling policies in a way that makes sense for our bottom line and network sealing capacity, as well as be significantly more enterprise-friendly 🌈

[laudiacay]

cc @sstelfox

[dd45e640b42e6da7da96faee3996ef7c]

even with resnapping there will be no hard-delete possible. all the network/miners will be able to deliver is a "your data is not stored in a filecoin sector that is committed to the filecoin mainnet anymore"

better than nothing but hardly a hard-delete.

we might want to think about phrasing this properly to reflect what is actually happening instead of calling it hard-delete, even delete is wrong'ish, to avoid further confusion down the road for consumers of applications build on top of filecoin.

[stef-sf]

@f8-ptrk I agree this is a great start. This approach doesn't seem to be that much different than default behaviors in other traditional filesystems though.
I agree that some applications will require a specific level of secure deletion (aka hard-delete), though that could be an improvement in the future where the underlying filesystem is wiped multiple times.

How can we prioritize the implementation of this FIP. I believe it's crucial for all onramps.

[donkabat]

very old topic without implementetion :(
there is so many problems to implrement it, bla bla bla... but SPs have real problem with this stupid issue.
It's so easy from my point of view... if deal is expired then sector is back as possible to snapup again.

[jennijuju]

@donkabat I agree this is a nice feature to have in the network!

It's so easy from my point of view...

If you have an early idea/spec on how this could be implemented in the protocol - we would love to see it & work with you on that!

[donkabat]

no, after I read posts above I was able only to complain to get your attention and that's all what can I do. I know, that is sad, but I'm just a simple SP.

[jennijuju]

@donkabat don’t think you need to complain! Simply let us know this feature is important to you and why is useful to help us prioritize - glad you are raising your voice here 💙

[Shekelme]

So, a year has passed, and is there still no progress on this obviously necessary option?

[rvagg]

I think that work around snaps is held up by the fact that there hasn't been enough demand for it. This is similar to #752, which is ready to go but just not fully implemented, and nobody's really been asking for it. Then we have the problem of tooling to support any of this. I think the ideal might be for Curio to find their way to a point where they can make snaps a bit of a priority and champion pushing it forward, and if that happens, the rest of us can follow along to support it. I think NI-PoRep is another key part of this puzzle that needs support, so we end up with something like a pre-sealed sector pool (and/or marketplace), with an easy snap process to populate them.

[lucaniz]

Repeated SnapDeals — where this lands, and the structure for the FIP

TL;DR: we aim to re-vamp the re-snap effort described in this discussion. The introduction of Direct Data Onboarding (FIP-0076) changed what is available to solve it with, so the mechanism has to be different from what was first proposed. Below is what DDO foreclosed, and the structure we'll propose into the FIP draft.

What changed after DDO

The original idea was to enforce piece preservation by reading PieceCIDs out of market and verifreg actor state. That was reasonable when written, but post-DDO it no longer generalizes, for two independent reasons:

1. Not every piece has a deal or a claim. DDO lets pieces be activated via raw PieceActivationManifest entries with no f05 deal and no FIL+ claim. For those pieces there is simply nothing in market or verifreg state to read. The "PieceCIDs are on-chain" premise holds only for the f05 deal/FIL+ subset.
2. Ordering was never in actor state anyway. Even where DealProposals or Claims exist, they are not keyed by position within the sector. Piece-level integrity needs to know where a piece sits, and that placement is not reconstructable from market/verifreg state.

So the right problem and the right primitive (preserve pieces across a replica update), but the proposed source of truth does not survive DDO. What's needed is a piece-integrity construction that does not depend on deals or claims existing (which is what the FIP should specify).

The structure for the FIP draft

One FIP, CC sectors only. Re-Snap generalizes SnapDeals (FIP-0019) from a one-shot update to a repeatable one. The same encoding (R = R_0 + ρ·D) applied again on an already-snapped sector, decoding back to the CC key and snapping new data. The repeatable-update mechanics and the piece-integrity layer are one feature, not two: repeatable update is what makes "keep a proven subset of pieces" possible, and that capability is the reason to want repeatable update in the first place. They are specified together, in a single CC-scoped FIP.

Sectors sealed natively with data are out of scope and handled as a separate Future Work track. The two open items there are

• A formal space-hardness analysis of the cross-term
• Retrievability of the snapped data.

The latter is the harder one and is specific to native sectors: a native replica after a snap is R' = R_0 + D + ρ·D', where D is the original native data. Recovering the bare key R_0 is never the obstacle (it can always be regenerated by re-sealing) but R_0 alone is not enough. Subtracting it leaves R' − R_0 = D + ρ·D', a single sum containing both the original data D and the snapped data D', which cannot be separated without independently having access D, or to the initial replica R_0 + D ). In the CC case this problem does not arise (D = 0, so R' − R_0 = ρ·D' and D' decodes directly); in the native case, retrieving D' requires a way to obtain the base R_0 + D, which re-sealing does not provide.

The capability: keep a subset of pieces and prove they are still present. On a Re-Snap the SP declares which existing pieces it carries forward, and that claim is verifiable (i.e. the kept pieces are proven genuinely present and unchanged after the update). This is framed purely in terms of pieces and their commitments; it makes no distinction between active and inactive deals, and depends on no piece having a deal or claim, precisely because of the DDO points above.

Anchor: opt-in CommD in SectorOnChainInfo. CommD is written by the miner actor and bound by the replica-update proof, so it is an SP-unforgeable anchor on each side of an update.

The FIP draft will add an opt-in, prospective-only UnsealedCID field to SectorOnChainInfo. The opt-in field exists to make verification possible on-chain: an FVM actor can read state but not historical events, so for a preservation guarantee to be load-bearing on-chain, CommD must be in state. Where only off-chain verification is needed, the same CommD is already carried by the FIP-0083 sector-updated event and no new state is required. Storing CommD opt-in and prospective-only avoids the state-tree objection (Discussion #298), which applied to retroactive, always-on storage; this mirrors the opt-in CommD direction from "Context Free Sectors" (Discussion #496).

Proof: PoDSI inclusion proofs against CommD. Preservation is checked by proving kept pieces’ CommP into CommD_old and CommD_new via the PoDSI construction (Discussion #512). In case having a per-piece proof is too much of a burden we can opt for a randomized selection of those.

Method. The FIP draft will propose to extend ProveReplicaUpdates3 (method 35, FIP-0076) (the live replica-update method after the removal ProveReplicaUpdates ) by relaxing its previously-empty-sector precondition so it may run on an already-snapped sector, and by adding the opt-in CommD-tracking flag. FIP-0076's SectorUpdateManifest already anticipates this: its Pieces field is documented as requiring all-new pieces only because the sector was previously empty, pending re-snap support. This FIP draft lifts that restriction.

A full draft following this structure is ready and will be posted as the FIP draft. Comments on the structure above are more than welcome.

[ThatGuyLLC]

HI There! Happy to get this on the agenda for next Core Devs in July!