Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.341108645s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:56:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.34113163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.384925859s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:55:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.38494727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.377789613s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:55:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.377819281s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.371710786s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:54:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.371739061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.336862166s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:52:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.33688528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.348518151s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:51:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.348544907s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.37436405s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:51:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.374387137s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.411132208s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:50:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.411175515s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.414122782s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:49:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.414152585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.42023825s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:48:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.420264338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.395040606s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:47:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.395065611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.371994056s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:47:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.372021421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.440120232s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:46:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.440147255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.371944663s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:45:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.371970352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.320133767s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:45:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.320162044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.308838149s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:44:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.308910049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.367390161s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:43:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.367421097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.419911504s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:43:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.419940782s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.395458672s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:42:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.395483227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.452685563s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:41:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.452711139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.382967903s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:40:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.383009015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.338676567s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:39:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.338705414s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.414569602s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:38:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.41459791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.326754355s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:37:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.326820049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.364320844s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:36:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.364360297s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.373858003s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:35:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.373887741s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.323677486s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:34:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.323722809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.424602044s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:33:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.424626853s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.35464094s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:32:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.354668015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.441389387s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:31:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.441421619s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.472821863s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:31:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.472845984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.56499043s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:30:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.565022192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.431275693s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:29:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.431303235s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.50979652s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:28:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.509815943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.511703825s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:27:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.511733036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.123428064s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:27:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.123449506s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.72089561s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:22:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.72092444s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.605116968s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:22:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.605151333s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 1.58992486s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:21:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.589953148s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 1.66098384s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:21:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.661019607s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 1.696574561s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:20:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.696610884s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 1.649568969s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:20:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.649676187s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 1.668588346s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:19:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.668616807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.62139138s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:18:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.621454478s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.682662759s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:17:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.682696492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 1.694280215s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:16:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.694300575s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 1.693018156s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:15:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.693046379s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 1.721488553s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:14:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.721582612s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 1.70876017s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:14:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.708792808s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 1.676700689s trace: https://jaeger.dev.storj.io/trace/7c99c1239082ddd FAIL (2025-11-26 21:13:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.676723717s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)