Unconfirmed pool full issue

Developers
1

So to sum up the issue that happened with Tarasca.
The were a bunch of transactions generated related to the game (over 2800), that the pool got filled on default nodes and transactions started being dropped. I'm not sure if the process of dropping transactions is what caused it, but there was also 100% CPU spikes on nodes then and in my log I just keep seeing the dropped txs

2021-12-04 14:36:38 FINE: pool-3-thread-2 Dropped unconfirmed transaction 2:c333fed60f6bef6861c3253a7c3bd04302b0393e939dc4f73baa891a069b14e0
2021-12-04 14:36:38 FINE: pool-3-thread-2 Dropped unconfirmed transaction 2:28b550f4add00b072aa0953ad3ea20ab18a693cadd948c3ba855428b162c5510
2021-12-04 14:36:38 FINE: pool-3-thread-2 Dropped unconfirmed transaction 2:95f8b2e85071a117bd275afa225c2248a578c8171314550e183186681632a9fb
2021-12-04 14:36:38 FINE: pool-3-thread-2 Dropped unconfirmed transaction 2:d92e927ceaed73c95d8b41e7f249b1920d528acdc819d5c811bd191849417dce
2021-12-04 14:36:38 FINE: pool-3-thread-2 Dropped unconfirmed transaction 2:a12abeaa5f193e6424742676419fafb61875e97cab8156c1c82e0422e82e4718
2021-12-04 14:36:38 FINE: pool-3-thread-2 Dropped unconfirmed transaction 2:79bafcbae3652ceae434bd59c020bbea2a0f170ace68dee500583d48ed59a731
2021-12-04 14:36:38 FINE: pool-3-thread-2 Dropped unconfirmed transaction 2:a4adc501d89dfa5b40f6a2d6995b0776fc847f338223ab3f5f923aabc6836313
2021-12-04 14:36:38 FINE: pool-3-thread-2 Dropped unconfirmed transaction 2:258fd1a4e63ee4fbe7d8105065043b99c9e1a259038aec4dc660fc6600d80b8e
2021-12-04 14:36:38 FINE: pool-3-thread-2 Dropped unconfirmed transaction 2:b7a41e9bf1b1761dca45febd091e465413eb2eb5c0d59809f00be1ad5fe06215
2021-12-04 14:36:38 FINE: pool-3-thread-2 Dropped unconfirmed transaction 2:00eaa2d37831ba7f906e8ff7afe8c2faff3308d34c18ed2c92c41d1c7c0ca3ec
2021-12-04 14:36:41 FINE: Message Handler 1 Resuming blockchain download - fork resolution required
2021-12-04 14:36:52 FINE: pool-3-thread-4 Will process a fork of 2 blocks, mine is 1
2021-12-04 14:36:52 FINE: pool-3-thread-4 Rollback from block 11676565732972113816 at height 2091028 to 10747899230743702133 at 2091027
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:6d0c282bd164c5f021f56a5dd765f5695dacc8e76fc0be84d2c291c20807a3b7
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:4455e4c4b78a46882313c889441206a986f1474eefd16fa24bdca4b251f630e0
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:451fc43ca1d4784a979a2d96f18347b41785d264391b8ee2b63e1e9cebfd9d45
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:816d0c203bd2ef5bfd25ea27958395f370160c041b6b27c277172b1c22603e45
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:8f7015ce3d1fff35b22e98b97b410995eb3781e830d28d2192a5f50494fa9a8d
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:08ab8b00b7b06fdf109849bf85d08378ddbcf83859b642b1bdac99d8cd355a63
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:23b49577814757d065b06780cf66fbd33e56b01f1d5aa11887ddf10882c62947
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:1e36d83d1d4785c886556ee8e8705d2cba65041373ced047219e0306b8cf5bb1
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:4da925d165a0519cf571e40350e1de7dc03859f5204632f87cb2138cdf569052
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:2e0e8a5456f01f2cabefa21ccb99711ed50f07c7388c167c21504fba0f30adb6
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:3c5b485bae521f5583ed32aed2ba5ad9cde3e8d2dbaa79e0fbab15d68844d511
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:f4730d29906aa037c5e10709c8eda2645b0efabd1eb55d038c881412ae692aed
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:1fe5ed7ce4b8ac06e5445b60183190afb4390fa56058001588e4ff7480feda7d
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:c8a3ec4d1d0b82382b3511be4949994b83bc0151ed66fc1997a4d4b3de0e1344
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:b4c01b22bf20be3f43ae269fba3448164e8121c2866c7e1a1d7f105bf01d6ed3
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:a5ac2d362065d55765d9d14cc80fb7f0651d2526180c32037fee78b73a48728f
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:829c42baa531cd05dfec28c65b9dc471a0af02a4c5bca211f60c0cb8fbe6abb5
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:dccd462da06b83b9728907fc3fc96774312371a766b2e5f3a4a18d93f76f6cd7
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:0ccff565f89ce40177a592b5d14a94484e192ee79a612bea5932402792ad2564
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:2de5ea9bd656fa7f32aa25fe712e9d2112a8923e06b6ecc6762eb6ab138b7bd8
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:52acef7d19409664c73a9782070ed3c6a3aa493b724dab7559584224bf53d395
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:3ecb5bdbf8c5be40d69b46fd5d94e43079eaaed73f7d8e2504dcca3ce16a92e4
2021-12-04 14:36:52 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:f1efa100b436d74c439a385878f20434bcabedff6e0877dd182678b183a5f427
2021-12-04 14:36:53 FINE: pool-3-thread-4 Switched to peer's fork
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:6186d292719e0b9d7a2a8ce872586e97bc025a2890958870b03fa89d5c951a66
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:287e185421d8af95773d1728b1ed11782f4bc55e26345d238a06363b5a87acec
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:e9772787da9cb9583419bbdb436715a87baf417eee6bd2d7620e25295777c8ea
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:55418f93bf19ce2f52d597a4f50000cf748b4eeeb5b4b27e2636f6a662179d6d
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:509ad3c42f4a182092e658f6b5f4a0b7cde9c61ccb7403cf9ecee434da335960
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:0e1b83d45a4972f2a31f22e109d46de41631b8f9b4fcbff27eadbd225764e640
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:33a6cf073d0a2676969c35102a6b9036d295b2e9f370107ea5eaf60c22af3107
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:2ee4eae2b590864b0d22076e30c8d1da03eadf4f721b9dbc8c808e91f08f85d8
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:c56554d73f0d5d422daad99b85ef9824515122ad67a3200bd26ce5af4a315d1a
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:0d866fc18103083925a08fe9f2deea2ae5537ecea2ff578c60bfed773a6733f4
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:1695ba95afd1effcb1ea70da3c02e43e2f6b4bf78f9b6d1f7422d91c9bf6cf12
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:aa225d8ed3a8cdc9ae7dc0e24e611caadf72c0a7bcc373ea12070ec60dfe2c77
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:1da3ee08b2a2aab41a1a55fe0adfc45c4bf9e0fdf431c8599f5271c424e4c2c5
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:314e8b7a177c5f95fcdfc63791a24bc432a22f6639b0ea2d814f8b0e9485ec08
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:f0248c8c6004777706c0e842a716658da4c841ff9aa1a5a510271dc7bc5bd68a
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:dcc8d110366d756ef7a4b40a610c4e23ab971eb75e077bff499aa0a8999b80f2
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:3fba520ba258a852ef69a601beda31b535bf1f0d1883dfda24837386033031dd
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:99e48c83b7cdf591a1ac49cdd0b8c888864051543953953de6c16914f3a3f2b2
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:ddc07246d3157f28c0eb84d34e231a141b7b97101facd9be1a4d412dc99b4795
2021-12-04 14:36:53 FINE: pool-3-thread-4 Dropped unconfirmed transaction 2:1eb57a4d0edf0671ea10bc32cd8e899c8d4723d3ebe0e4a4a5daba367d695e02

Yet, forgers were still forging some empty blocks as well. Best guess from tre was:

The theory is that this takes time to be available, or a performance bottle-neck. Then if the unconfirmed is over-loaded this may take much longer to be available.
We also know that a bundle will be rejected if any of the sub-transactions are not present in the pool (or appear not to be).

So between maxed out CPU nodes and transactions from the unconfirmed pool being dropped, forgers were not accepting the bundles and forging empty blocks.

Why does this not seem to be more of a concern for Jelurida? One project can mess with the whole network this much, causing nodes to slow down from high CPU, forks, disappearing transactions, empty blocks, etc.

Also, while it won't fix the problem I would suggest the following to help with these situations:

  1. Upping the default unconfirmed pool (why is it only 2500?),
  2. Upping the default deadline to at least 60min (gives time to notice there is an issue as well as time for bundlers to catch up or bundle manually. I remember noticing once my tx wasn't being bundled and not having enough time to manually do it, losing everything I had in the message of the transaction at the time. Very frustrating) (Bitcoin I believe is 24hrs before you can try to cancel a tx)
  3. Consider shortening blocktimes or doing something for more TPS on childchains. Right now IGNIS can only do 1.66TPS, if there was real world usage this bottleneck would be hit regularly if not always.

The average user does not understand bundling or how to manually bundle their transaction, they expect it to work and the event here is not the experience you want them to have if you expect to actually be used

2

For the sake of documenting it here, could you, or someone from Tarasca, summarize what are the transactions they're sending and how (contract, API...) are they sending them?

3
  1. Upping the default unconfirmed pool (why is it only 2500?),

Actually 2500 is the max size a node operator can set. The default is 1500. I'm not sure why the max is 2500.

As discussed in slack, I would prefer that it's a responsibility of the transaction issuer's node to store the transaction and re-broadcast it until it is accepted. But for that we must develop a functionality for whitelisting sender accounts. Then Tre suggested to also allow whitelist recipients. But I think this will allow DoS-ing the node with 0-fee transactions sent to the whitelisted account, unless we handle that somehow.

  1. Upping the default deadline to at least 60min (gives time to notice there is an issue as well as time for bundlers to catch up or bundle manually. I remember noticing once my tx wasn't being bundled and not having enough time to manually do it, losing everything I had in the message of the transaction at the time. Very frustrating) (Bitcoin I believe is 24hrs before you can try to cancel a tx)

We set the default deadline to 15 minutes because there were many complains for the following problem:

If the user issues a transaction with deadline 1440 minutes (the default in Nxt) and its fee is too low, the user have to wait for a whole day before being able to issue a transaction with higher fee

If you (or your service/contract) are not going to re-issue the transaction after 15 minutes, of course you should use longer deadline. I don't think we must change the default.