[Mirrored from Slack] At times, data is missing in archive node. How can we retrieve data from archive node in reliable way?

Developers
1

@tre
Data has gone missing before, and was only fixed because I replaced the entry in the database and presently it has happned again but I have not yet looked if it can be reconstructed.

2:362e09201bc016d61b69d2c672b19d15ceb0a2e24e69c77a3cca06b288ea768b with getTransaction API. The message body is missing from all nodes including archival nodes. If this piece was part of a NFT's image the data would be corrupted or completely missing if the first part of the chain requiring reassembly.

@thewiremaster
I can imagine that this could happen if, eg, the disk on which we upload the data has a full disk and therefore the data is corrupt at the source. Otherwise I agree it's a bug

@tre
This is not the first occurrence. Perhaps martis recalls the discussion we had on #developers about this same issue. In that case it was easy to discover the missing data because there were other copies with the same hash and I inserted the data into the platform, fortunately this did not mess up everyone's nodes. In the present case the sending account is Jelurida's and they posses the secret which can be used to help reconstruct the missing data. The next time perhaps the data will not be at all possible to recover.

@thewiremaster
So it means that the data can be lost also after long time after it has been uploaded? Or does this happen close to the upload time?

@tre
In the case discussed on #developers, the data does appear to have been lost at some point long after it was confirmed because the error did not show up before - although this may have been a change to the log detail.The present case the source is Jelurida's node reward account, so that it is odd that it has occurred from the developer's controlled account but convenient in that perhaps they can better investigate the issue and also they posses the source data to "fix" the missing database entry.In all cases the prunable data must have been broadcast and accepted to the block generator's node in order for the source transaction to have been confirmed.

@mrv777
The transaction hash can be found in a bundle transaction on a Jelurida node
https://ardor.jelurida.com/nxt?requestType=getFxtTransaction&fullHash=9613319c18ec2f85b1653600ae8c3a76d374d6913344220053322bf480dca161
But the transaction itself is "unkown" perhaps this detail will help the developers diagnose the issue.

@tre
This may be a serious issue because the non-prunable part of a money transfer transaction is missing so those nodes will not be able to rebuild and verify their blockchain.

2

There is no such transaction.

The closest one I find there is the first one with this hash:

362e09201bc016d61b69d2c672b19d15ceb0a2e24e69c77a3cca06b288ea76ab

Which is almost the same hash posted before:

362e09201bc016d61b69d2c672b19d15ceb0a2e24e69c77a3cca06b288ea768b

I hope that was just some copy/paste or transcription error. Otherwise someone is really close to cracking sha256 or extremely lucky. :sweat_smile:

And the transaction itself (the correct one) has the pruned data available, at least on my node.

3 Likes
3

About the reliability of the archival nodes I would say past problems where probably related to the very low number of archival nodes. Also the loading mechanism of pruned data from archival nodes was a bit slow.

Today we have an improved loading mechanism. It's faster, tries harder, and explains potential problems on the logs. And we have loads of archival nodes.

Pruned data can be lost if all archival nodes lose it which I think it's quite hard currently. But the same can be said about the blockchain itself. Maybe those past lost transactions happened when the network had very few archival nodes. If you had a blockchain with also very few nodes it will also fail. That's how it works. That's why having several nodes helps the network. And that's the goal, for example, of the node rewards program. To improve the network.