Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.460701863s trace: https://jaeger.dev.storj.io/trace/3b09f358d3a8cbe1 FAIL (2025-11-10 04:08:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZFTPBVHQ, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.661556198 (256h26m7.363148673s) [noise=true , address=77.90.31.15:4699 , piece_id=EVCDT3LRX36TFUTM77JIENYHMMD6A4DXK52NB7BXLYGXO6FGHLDA, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.665454063 (256h26m7.367046526s) [noise=true , address=145.79.227.10:4403 , piece_id=YTDBFQ5BNOO36YSSFB2CTJQDLL74FTZKRZSDZ23NQTNDGKN7NEWQ, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.668482532 (256h26m7.370074999s) [noise=true , address=145.79.227.27:4690 , piece_id=HMIQNZADYF5X3VGKQVUYBIT4QKAP3WP4MWFKSD6WYWIFLOU7HRHA, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.671502653 (256h26m7.373095116s) [noise=true , address=145.79.227.21:4473 , piece_id=4ZFP7SWM42VN3HQONESEYD53NJR4LKUFF3DMF2OLZ3TMD7PJFS3A, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.675154308 (256h26m7.376746772s) [noise=true , address=145.79.227.26:4655 , piece_id=DRWEQJH25FFBJ376NSRHSIKKTCQFQZFQETSKWB4OZOM4YSOCKRUQ, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.678758271 (256h26m7.380350731s) [noise=true , address=145.79.227.8:4330 , piece_id=FTSHIMVXJORWQSCUQLGFZUFDOASNBA46HDC3QA4AV2IZYSUU4HNQ, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.682994960 (256h26m7.384587423s) [noise=true , address=145.79.227.15:4875 , piece_id=DGHU2DHRFP2WXVHPHRRYSIABFUFPOJWCEHLBFHHSB6JSFDGOZ22A, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:05.306999595 (256h26m8.008592045s) [upload=225516]: Next returned nil segment Nov 10 04:07:05.307014203 (256h26m8.008606667s) [upload=225516]: Waiting for error group managing segments... Nov 10 04:07:05.444220454 (256h26m8.145812917s) [noise=true , address=77.90.31.15:4699 , piece_id=EVCDT3LRX36TFUTM77JIENYHMMD6A4DXK52NB7BXLYGXO6FGHLDA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: <nil> Nov 10 04:07:05.451767504 (256h26m8.153359957s) [noise=true , address=77.90.31.12:4592 , piece_id=6EEWGCA7Y3FBSLXW75RIFRS5XDS6X2NJVKD4QKP4UJRSMHYT7SJQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: <nil> Nov 10 04:07:05.459363315 (256h26m8.160955777s) [noise=true , address=77.90.31.18:4806 , piece_id=WZZWF4TQAJIKO2VZ2A25MSAMEIRM7MRWFGBDZW233YJMZFTPBVHQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: <nil> Nov 10 04:07:06.199674203 (256h26m8.901266653s) [noise=true , address=145.79.227.15:4875 , piece_id=DGHU2DHRFP2WXVHPHRRYSIABFUFPOJWCEHLBFHHSB6JSFDGOZ22A, seg_pos={0 0}, upload=225516]: Done uploading piece. err: <nil> Nov 10 04:07:34.173053276 (256h26m36.874645735s) [noise=true , address=145.79.227.16:4909 , piece_id=ZBFDJQRLYTGYMYIR5WIW7HN72YFZNRT27AETM6V6MATG66XOY5JA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 139.59.144.22:34048->145.79.227.16:4909: use of closed network connection; write tcp 139.59.144.22:34048->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34048->145.79.227.16:4909: use of closed network connection Nov 10 04:07:34.173221637 (256h26m36.874814096s) [noise=true , address=145.79.227.11:4730 , piece_id=6AB62ISS7NFCHEE26GPVBTXIAC7CCVBFK4V3VNP5CL7KBUPUSOPQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:33710->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:33710->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33710->145.79.227.11:4730: use of closed network connection Nov 10 04:07:34.173446017 (256h26m36.87503848s) [noise=true , address=145.79.227.7:4294 , piece_id=B2FWZ6PI22ZSCSJRZ2X4E6NMN6H36BC6AZIOTX7WIMT4EN5FVXSQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 139.59.144.22:42906->145.79.227.7:4294: use of closed network connection; write tcp 139.59.144.22:42906->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42906->145.79.227.7:4294: use of closed network connection Nov 10 04:07:34.173574406 (256h26m36.875166865s) [noise=true , address=145.79.227.21:4473 , piece_id=4ZFP7SWM42VN3HQONESEYD53NJR4LKUFF3DMF2OLZ3TMD7PJFS3A, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 139.59.144.22:46482->145.79.227.21:4473: use of closed network connection; write tcp 139.59.144.22:46482->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46482->145.79.227.21:4473: use of closed network connection Nov 10 04:07:34.173711501 (256h26m36.875303962s) [noise=true , address=145.79.227.26:4655 , piece_id=DRWEQJH25FFBJ376NSRHSIKKTCQFQZFQETSKWB4OZOM4YSOCKRUQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 139.59.144.22:47452->145.79.227.26:4655: use of closed network connection; write tcp 139.59.144.22:47452->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47452->145.79.227.26:4655: use of closed network connection Nov 10 04:07:34.173846065 (256h26m36.875438526s) [noise=true , address=145.79.227.13:4802 , piece_id=ASZHXU5NAGA3OHJNWSSYOGXL3NC6G2YRVP7DVQFKDERJZTDUKJ2A, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 139.59.144.22:40760->145.79.227.13:4802: use of closed network connection; write tcp 139.59.144.22:40760->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40760->145.79.227.13:4802: use of closed network connection Nov 10 04:07:34.174056958 (256h26m36.875649419s) [noise=true , address=145.79.227.14:4840 , piece_id=6DH252TCBW5KCRH4RAWCZUUFCYZBSTX7I67ROAL66CA2V3N3T2PQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 139.59.144.22:42448->145.79.227.14:4840: use of closed network connection; write tcp 139.59.144.22:42448->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42448->145.79.227.14:4840: use of closed network connection Nov 10 04:07:34.174345626 (256h26m36.875938079s) [noise=true , address=145.79.227.23:4545 , piece_id=J5WXBOJKO5PJ77E4GWFIBGADT6SUB2PXQUSEMPSRUEMTUOBIBBLA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 139.59.144.22:42072->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 04:07:34.174481151 (256h26m36.876073613s) [noise=true , address=145.79.227.26:4654 , piece_id=RFU6W5RZAQAISIYYBE42SER2WP3BCC4II6ECU7DNDFIVS4VKIDMA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 139.59.144.22:46802->145.79.227.26:4654: use of closed network connection; write tcp 139.59.144.22:46802->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46802->145.79.227.26:4654: use of closed network connection Nov 10 04:07:34.174666791 (256h26m36.876259257s) [noise=true , address=145.79.227.27:4690 , piece_id=HMIQNZADYF5X3VGKQVUYBIT4QKAP3WP4MWFKSD6WYWIFLOU7HRHA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:54484->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:54484->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54484->145.79.227.27:4690: use of closed network connection Nov 10 04:07:34.174965589 (256h26m36.876558045s) [noise=true , address=145.79.227.10:4403 , piece_id=YTDBFQ5BNOO36YSSFB2CTJQDLL74FTZKRZSDZ23NQTNDGKN7NEWQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: read tcp 139.59.144.22:49306->145.79.227.10:4403: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 04:07:34.175095672 (256h26m36.876688135s) [noise=true , address=145.79.227.8:4330 , piece_id=FTSHIMVXJORWQSCUQLGFZUFDOASNBA46HDC3QA4AV2IZYSUU4HNQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 139.59.144.22:55082->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 04:07:34.175131671 (256h26m36.876724111s) [seg_pos={0 0}, upload=225516]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 10 04:07:34.291181778 (256h26m36.992774235s) [upload=225516]: Done uploading segments... Nov 10 04:07:34.291260093 (256h26m36.99285255s) [upload=225516]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 2.460731935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.450953473s trace: https://jaeger.dev.storj.io/trace/3b09f358d3a8cbe1 FAIL (2025-11-10 04:07:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZFTPBVHQ, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.661556198 (256h26m7.363148673s) [noise=true , address=77.90.31.15:4699 , piece_id=EVCDT3LRX36TFUTM77JIENYHMMD6A4DXK52NB7BXLYGXO6FGHLDA, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.665454063 (256h26m7.367046526s) [noise=true , address=145.79.227.10:4403 , piece_id=YTDBFQ5BNOO36YSSFB2CTJQDLL74FTZKRZSDZ23NQTNDGKN7NEWQ, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.668482532 (256h26m7.370074999s) [noise=true , address=145.79.227.27:4690 , piece_id=HMIQNZADYF5X3VGKQVUYBIT4QKAP3WP4MWFKSD6WYWIFLOU7HRHA, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.671502653 (256h26m7.373095116s) [noise=true , address=145.79.227.21:4473 , piece_id=4ZFP7SWM42VN3HQONESEYD53NJR4LKUFF3DMF2OLZ3TMD7PJFS3A, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.675154308 (256h26m7.376746772s) [noise=true , address=145.79.227.26:4655 , piece_id=DRWEQJH25FFBJ376NSRHSIKKTCQFQZFQETSKWB4OZOM4YSOCKRUQ, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.678758271 (256h26m7.380350731s) [noise=true , address=145.79.227.8:4330 , piece_id=FTSHIMVXJORWQSCUQLGFZUFDOASNBA46HDC3QA4AV2IZYSUU4HNQ, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.682994960 (256h26m7.384587423s) [noise=true , address=145.79.227.15:4875 , piece_id=DGHU2DHRFP2WXVHPHRRYSIABFUFPOJWCEHLBFHHSB6JSFDGOZ22A, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:05.306999595 (256h26m8.008592045s) [upload=225516]: Next returned nil segment Nov 10 04:07:05.307014203 (256h26m8.008606667s) [upload=225516]: Waiting for error group managing segments... Nov 10 04:07:05.444220454 (256h26m8.145812917s) [noise=true , address=77.90.31.15:4699 , piece_id=EVCDT3LRX36TFUTM77JIENYHMMD6A4DXK52NB7BXLYGXO6FGHLDA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: <nil> Nov 10 04:07:05.451767504 (256h26m8.153359957s) [noise=true , address=77.90.31.12:4592 , piece_id=6EEWGCA7Y3FBSLXW75RIFRS5XDS6X2NJVKD4QKP4UJRSMHYT7SJQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: <nil> Nov 10 04:07:05.459363315 (256h26m8.160955777s) [noise=true , address=77.90.31.18:4806 , piece_id=WZZWF4TQAJIKO2VZ2A25MSAMEIRM7MRWFGBDZW233YJMZFTPBVHQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: <nil> Nov 10 04:07:06.199674203 (256h26m8.901266653s) [noise=true , address=145.79.227.15:4875 , piece_id=DGHU2DHRFP2WXVHPHRRYSIABFUFPOJWCEHLBFHHSB6JSFDGOZ22A, seg_pos={0 0}, upload=225516]: Done uploading piece. err: <nil> Nov 10 04:07:34.173053276 (256h26m36.874645735s) [noise=true , address=145.79.227.16:4909 , piece_id=ZBFDJQRLYTGYMYIR5WIW7HN72YFZNRT27AETM6V6MATG66XOY5JA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 139.59.144.22:34048->145.79.227.16:4909: use of closed network connection; write tcp 139.59.144.22:34048->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34048->145.79.227.16:4909: use of closed network connection Nov 10 04:07:34.173221637 (256h26m36.874814096s) [noise=true , address=145.79.227.11:4730 , piece_id=6AB62ISS7NFCHEE26GPVBTXIAC7CCVBFK4V3VNP5CL7KBUPUSOPQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:33710->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:33710->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33710->145.79.227.11:4730: use of closed network connection Nov 10 04:07:34.173446017 (256h26m36.87503848s) [noise=true , address=145.79.227.7:4294 , piece_id=B2FWZ6PI22ZSCSJRZ2X4E6NMN6H36BC6AZIOTX7WIMT4EN5FVXSQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 139.59.144.22:42906->145.79.227.7:4294: use of closed network connection; write tcp 139.59.144.22:42906->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42906->145.79.227.7:4294: use of closed network connection Nov 10 04:07:34.173574406 (256h26m36.875166865s) [noise=true , address=145.79.227.21:4473 , piece_id=4ZFP7SWM42VN3HQONESEYD53NJR4LKUFF3DMF2OLZ3TMD7PJFS3A, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 139.59.144.22:46482->145.79.227.21:4473: use of closed network connection; write tcp 139.59.144.22:46482->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46482->145.79.227.21:4473: use of closed network connection Nov 10 04:07:34.173711501 (256h26m36.875303962s) [noise=true , address=145.79.227.26:4655 , piece_id=DRWEQJH25FFBJ376NSRHSIKKTCQFQZFQETSKWB4OZOM4YSOCKRUQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 139.59.144.22:47452->145.79.227.26:4655: use of closed network connection; write tcp 139.59.144.22:47452->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47452->145.79.227.26:4655: use of closed network connection Nov 10 04:07:34.173846065 (256h26m36.875438526s) [noise=true , address=145.79.227.13:4802 , piece_id=ASZHXU5NAGA3OHJNWSSYOGXL3NC6G2YRVP7DVQFKDERJZTDUKJ2A, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 139.59.144.22:40760->145.79.227.13:4802: use of closed network connection; write tcp 139.59.144.22:40760->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40760->145.79.227.13:4802: use of closed network connection Nov 10 04:07:34.174056958 (256h26m36.875649419s) [noise=true , address=145.79.227.14:4840 , piece_id=6DH252TCBW5KCRH4RAWCZUUFCYZBSTX7I67ROAL66CA2V3N3T2PQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 139.59.144.22:42448->145.79.227.14:4840: use of closed network connection; write tcp 139.59.144.22:42448->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42448->145.79.227.14:4840: use of closed network connection Nov 10 04:07:34.174345626 (256h26m36.875938079s) [noise=true , address=145.79.227.23:4545 , piece_id=J5WXBOJKO5PJ77E4GWFIBGADT6SUB2PXQUSEMPSRUEMTUOBIBBLA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 139.59.144.22:42072->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 04:07:34.174481151 (256h26m36.876073613s) [noise=true , address=145.79.227.26:4654 , piece_id=RFU6W5RZAQAISIYYBE42SER2WP3BCC4II6ECU7DNDFIVS4VKIDMA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 139.59.144.22:46802->145.79.227.26:4654: use of closed network connection; write tcp 139.59.144.22:46802->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46802->145.79.227.26:4654: use of closed network connection Nov 10 04:07:34.174666791 (256h26m36.876259257s) [noise=true , address=145.79.227.27:4690 , piece_id=HMIQNZADYF5X3VGKQVUYBIT4QKAP3WP4MWFKSD6WYWIFLOU7HRHA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:54484->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:54484->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54484->145.79.227.27:4690: use of closed network connection Nov 10 04:07:34.174965589 (256h26m36.876558045s) [noise=true , address=145.79.227.10:4403 , piece_id=YTDBFQ5BNOO36YSSFB2CTJQDLL74FTZKRZSDZ23NQTNDGKN7NEWQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: read tcp 139.59.144.22:49306->145.79.227.10:4403: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 04:07:34.175095672 (256h26m36.876688135s) [noise=true , address=145.79.227.8:4330 , piece_id=FTSHIMVXJORWQSCUQLGFZUFDOASNBA46HDC3QA4AV2IZYSUU4HNQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 139.59.144.22:55082->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 04:07:34.175131671 (256h26m36.876724111s) [seg_pos={0 0}, upload=225516]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 10 04:07:34.291181778 (256h26m36.992774235s) [upload=225516]: Done uploading segments... Nov 10 04:07:34.291260093 (256h26m36.99285255s) [upload=225516]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 2.450986714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.119392847s trace: https://jaeger.dev.storj.io/trace/3b09f358d3a8cbe1 FAIL (2025-11-10 04:07:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== ZFTPBVHQ, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.661556198 (256h26m7.363148673s) [noise=true , address=77.90.31.15:4699 , piece_id=EVCDT3LRX36TFUTM77JIENYHMMD6A4DXK52NB7BXLYGXO6FGHLDA, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.665454063 (256h26m7.367046526s) [noise=true , address=145.79.227.10:4403 , piece_id=YTDBFQ5BNOO36YSSFB2CTJQDLL74FTZKRZSDZ23NQTNDGKN7NEWQ, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.668482532 (256h26m7.370074999s) [noise=true , address=145.79.227.27:4690 , piece_id=HMIQNZADYF5X3VGKQVUYBIT4QKAP3WP4MWFKSD6WYWIFLOU7HRHA, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.671502653 (256h26m7.373095116s) [noise=true , address=145.79.227.21:4473 , piece_id=4ZFP7SWM42VN3HQONESEYD53NJR4LKUFF3DMF2OLZ3TMD7PJFS3A, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.675154308 (256h26m7.376746772s) [noise=true , address=145.79.227.26:4655 , piece_id=DRWEQJH25FFBJ376NSRHSIKKTCQFQZFQETSKWB4OZOM4YSOCKRUQ, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.678758271 (256h26m7.380350731s) [noise=true , address=145.79.227.8:4330 , piece_id=FTSHIMVXJORWQSCUQLGFZUFDOASNBA46HDC3QA4AV2IZYSUU4HNQ, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:04.682994960 (256h26m7.384587423s) [noise=true , address=145.79.227.15:4875 , piece_id=DGHU2DHRFP2WXVHPHRRYSIABFUFPOJWCEHLBFHHSB6JSFDGOZ22A, seg_pos={0 0}, upload=225516]: Uploading piece... Nov 10 04:07:05.306999595 (256h26m8.008592045s) [upload=225516]: Next returned nil segment Nov 10 04:07:05.307014203 (256h26m8.008606667s) [upload=225516]: Waiting for error group managing segments... Nov 10 04:07:05.444220454 (256h26m8.145812917s) [noise=true , address=77.90.31.15:4699 , piece_id=EVCDT3LRX36TFUTM77JIENYHMMD6A4DXK52NB7BXLYGXO6FGHLDA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: <nil> Nov 10 04:07:05.451767504 (256h26m8.153359957s) [noise=true , address=77.90.31.12:4592 , piece_id=6EEWGCA7Y3FBSLXW75RIFRS5XDS6X2NJVKD4QKP4UJRSMHYT7SJQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: <nil> Nov 10 04:07:05.459363315 (256h26m8.160955777s) [noise=true , address=77.90.31.18:4806 , piece_id=WZZWF4TQAJIKO2VZ2A25MSAMEIRM7MRWFGBDZW233YJMZFTPBVHQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: <nil> Nov 10 04:07:06.199674203 (256h26m8.901266653s) [noise=true , address=145.79.227.15:4875 , piece_id=DGHU2DHRFP2WXVHPHRRYSIABFUFPOJWCEHLBFHHSB6JSFDGOZ22A, seg_pos={0 0}, upload=225516]: Done uploading piece. err: <nil> Nov 10 04:07:34.173053276 (256h26m36.874645735s) [noise=true , address=145.79.227.16:4909 , piece_id=ZBFDJQRLYTGYMYIR5WIW7HN72YFZNRT27AETM6V6MATG66XOY5JA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 139.59.144.22:34048->145.79.227.16:4909: use of closed network connection; write tcp 139.59.144.22:34048->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34048->145.79.227.16:4909: use of closed network connection Nov 10 04:07:34.173221637 (256h26m36.874814096s) [noise=true , address=145.79.227.11:4730 , piece_id=6AB62ISS7NFCHEE26GPVBTXIAC7CCVBFK4V3VNP5CL7KBUPUSOPQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:33710->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:33710->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33710->145.79.227.11:4730: use of closed network connection Nov 10 04:07:34.173446017 (256h26m36.87503848s) [noise=true , address=145.79.227.7:4294 , piece_id=B2FWZ6PI22ZSCSJRZ2X4E6NMN6H36BC6AZIOTX7WIMT4EN5FVXSQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: write tcp 139.59.144.22:42906->145.79.227.7:4294: use of closed network connection; write tcp 139.59.144.22:42906->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42906->145.79.227.7:4294: use of closed network connection Nov 10 04:07:34.173574406 (256h26m36.875166865s) [noise=true , address=145.79.227.21:4473 , piece_id=4ZFP7SWM42VN3HQONESEYD53NJR4LKUFF3DMF2OLZ3TMD7PJFS3A, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 139.59.144.22:46482->145.79.227.21:4473: use of closed network connection; write tcp 139.59.144.22:46482->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46482->145.79.227.21:4473: use of closed network connection Nov 10 04:07:34.173711501 (256h26m36.875303962s) [noise=true , address=145.79.227.26:4655 , piece_id=DRWEQJH25FFBJ376NSRHSIKKTCQFQZFQETSKWB4OZOM4YSOCKRUQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 139.59.144.22:47452->145.79.227.26:4655: use of closed network connection; write tcp 139.59.144.22:47452->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47452->145.79.227.26:4655: use of closed network connection Nov 10 04:07:34.173846065 (256h26m36.875438526s) [noise=true , address=145.79.227.13:4802 , piece_id=ASZHXU5NAGA3OHJNWSSYOGXL3NC6G2YRVP7DVQFKDERJZTDUKJ2A, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 139.59.144.22:40760->145.79.227.13:4802: use of closed network connection; write tcp 139.59.144.22:40760->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40760->145.79.227.13:4802: use of closed network connection Nov 10 04:07:34.174056958 (256h26m36.875649419s) [noise=true , address=145.79.227.14:4840 , piece_id=6DH252TCBW5KCRH4RAWCZUUFCYZBSTX7I67ROAL66CA2V3N3T2PQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 139.59.144.22:42448->145.79.227.14:4840: use of closed network connection; write tcp 139.59.144.22:42448->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42448->145.79.227.14:4840: use of closed network connection Nov 10 04:07:34.174345626 (256h26m36.875938079s) [noise=true , address=145.79.227.23:4545 , piece_id=J5WXBOJKO5PJ77E4GWFIBGADT6SUB2PXQUSEMPSRUEMTUOBIBBLA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 139.59.144.22:42072->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 04:07:34.174481151 (256h26m36.876073613s) [noise=true , address=145.79.227.26:4654 , piece_id=RFU6W5RZAQAISIYYBE42SER2WP3BCC4II6ECU7DNDFIVS4VKIDMA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 139.59.144.22:46802->145.79.227.26:4654: use of closed network connection; write tcp 139.59.144.22:46802->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46802->145.79.227.26:4654: use of closed network connection Nov 10 04:07:34.174666791 (256h26m36.876259257s) [noise=true , address=145.79.227.27:4690 , piece_id=HMIQNZADYF5X3VGKQVUYBIT4QKAP3WP4MWFKSD6WYWIFLOU7HRHA, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:54484->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:54484->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54484->145.79.227.27:4690: use of closed network connection Nov 10 04:07:34.174965589 (256h26m36.876558045s) [noise=true , address=145.79.227.10:4403 , piece_id=YTDBFQ5BNOO36YSSFB2CTJQDLL74FTZKRZSDZ23NQTNDGKN7NEWQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: read tcp 139.59.144.22:49306->145.79.227.10:4403: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 04:07:34.175095672 (256h26m36.876688135s) [noise=true , address=145.79.227.8:4330 , piece_id=FTSHIMVXJORWQSCUQLGFZUFDOASNBA46HDC3QA4AV2IZYSUU4HNQ, seg_pos={0 0}, upload=225516]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 139.59.144.22:55082->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 04:07:34.175131671 (256h26m36.876724111s) [seg_pos={0 0}, upload=225516]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 10 04:07:34.291181778 (256h26m36.992774235s) [upload=225516]: Done uploading segments... Nov 10 04:07:34.291260093 (256h26m36.99285255s) [upload=225516]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.119419794s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.313434788s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:36:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.313470858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.245377602s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:34:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.245430986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.467024602s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:34:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.467140154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.368588544s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:33:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.368611383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.404027319s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:32:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.404062087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.402366007s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:31:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.402396277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.65972128s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:30:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.659769352s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.22672854s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:30:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.226762299s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.451243375s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:29:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.451273227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.445677588s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:29:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.445717992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.358857499s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:28:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.358875614s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 2.436334526s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:28:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.436452856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.517505801s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:28:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.517534498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 2.400117463s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:28:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.400150333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 2.488078817s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:27:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.488110283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.212167635s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:27:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.212199135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 2.404231159s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:26:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.404264901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 2.325757342s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:26:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.325786781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.968527648s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:26:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.968558831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 2.421024068s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:25:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.421054039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.547431873s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:25:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.547466821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 2.548674837s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:25:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.548705454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.365726244s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:24:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.365761612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 2.371847933s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:24:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.371981074s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 2.29576876s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:23:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.295801384s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.232483234s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:23:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.23253114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 2.424084964s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:23:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.424117512s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..........XX...............................................] duration: 2.374627822s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:22:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.374662338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.471592508s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:22:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.471626319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..........XX..............................................] duration: 2.368541554s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:22:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.368600426s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..........XX.............................................] duration: 2.421412536s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:21:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.421443934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.619847131s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:21:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.619897449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..........XX............................................] duration: 2.2648826s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:21:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.264914965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.508781786s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:21:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.508885489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..........XX...........................................] duration: 2.352269216s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:20:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.352302508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.384412903s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:20:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.384450031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..........XX..........................................] duration: 2.373200817s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:19:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.373250494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..........XX.........................................] duration: 2.374653072s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:19:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.374691006s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.367823718s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:18:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.367853371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..........XX........................................] duration: 2.289334522s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:18:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.289356964s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.622587081s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:18:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.622645325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..........XX.......................................] duration: 2.444778904s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:17:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.444809914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.319082248s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:17:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.319142605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.397064675s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:17:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.39716071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..........XX......................................] duration: 2.494235998s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:16:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.494266435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.383120865s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:16:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.383217201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..........XX.....................................] duration: 2.421816482s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:15:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.421862965s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)