Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.951299305s trace: https://jaeger.dev.storj.io/trace/dbcc8e0ac84a41 FAIL (2025-11-27 04:08:21 UTC) orbiter warning: read tcp 139.59.144.22:36358->136.0.77.3:443: read: connection reset by peer Duration: 2.951324706s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 3.028301138s trace: https://jaeger.dev.storj.io/trace/dbcc8e0ac84a41 FAIL (2025-11-27 04:07:12 UTC) orbiter warning: read tcp 139.59.144.22:36358->136.0.77.3:443: read: connection reset by peer Duration: 3.028329447s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.942604908s trace: https://jaeger.dev.storj.io/trace/dbcc8e0ac84a41 FAIL (2025-11-27 04:06:37 UTC) orbiter warning: read tcp 139.59.144.22:36358->136.0.77.3:443: read: connection reset by peer Duration: 2.942722293s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 3.068999558s trace: https://jaeger.dev.storj.io/trace/dbcc8e0ac84a41 FAIL (2025-11-27 04:05:48 UTC) orbiter warning: read tcp 139.59.144.22:36358->136.0.77.3:443: read: connection reset by peer Duration: 3.069020394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.491268953s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:13:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.491314112s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.452796499s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:12:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.452820634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.405535259s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:11:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.405585181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.272727399s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:10:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.272751s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.340855439s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:10:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.340879104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.326950836s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:09:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.326981957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.41247138s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:09:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.412571446s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.416010545s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:08:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.416037659s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.350174049s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:07:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.350204716s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.61191492s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:06:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.611940241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.418277301s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:06:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.418303813s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.408173002s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:05:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.408198283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.343122177s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:04:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.343137022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.281236278s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:03:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.281277898s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.440558769s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:03:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.440584898s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.440086861s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:02:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.440115601s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.394410931s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:01:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.394438869s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.50262411s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:01:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.502659761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.439298427s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 01:00:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.439407634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.466466312s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:59:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.466494635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.398588078s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:59:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.398632671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.403230386s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:58:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.403311724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.436780019s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:57:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.436805383s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.458422364s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:57:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.458451975s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.341108645s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:56:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.34113163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.384925859s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:55:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.38494727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.377789613s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:55:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.377819281s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.371710786s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:54:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.371739061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.336862166s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:52:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.33688528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.348518151s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:51:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.348544907s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.37436405s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:51:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.374387137s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.411132208s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:50:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.411175515s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.414122782s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:49:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.414152585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.42023825s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:48:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.420264338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.395040606s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:47:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.395065611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.371994056s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:47:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.372021421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.440120232s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:46:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.440147255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.371944663s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:45:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.371970352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.320133767s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:45:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.320162044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.308838149s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:44:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.308910049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.367390161s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:43:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.367421097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.419911504s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:43:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.419940782s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.395458672s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:42:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.395483227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.452685563s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:41:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.452711139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.382967903s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:40:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.383009015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.338676567s trace: https://jaeger.dev.storj.io/trace/78543768cdc866c5 FAIL (2025-11-27 00:39:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 00:26:50.030052716 (374h42m35.388711375s) [upload=330026]: Starting upload Nov 27 00:26:50.030154989 (374h42m35.38881364s) [upload=330026]: Done starting upload Nov 27 00:26:50.030336556 (374h42m35.388995233s) [upload=330026]: Uploading segments... Nov 27 00:26:50.030582500 (374h42m35.389241157s) [upload=330026]: Got next segment. Inline: false Nov 27 00:26:50.030641473 (374h42m35.389300137s) [upload=330026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 00:26:50.030649779 (374h42m35.389308441s) [upload=330026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 00:26:50.141771065 (374h42m35.500429719s) [seg_pos={0 0}, upload=330026]: Begin upload segment... Nov 27 00:26:50.141833799 (374h42m35.500492465s) [seg_pos={0 0}, upload=330026]: Done begin upload segment. Nov 27 00:26:50.141879210 (374h42m35.50053787s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.141899342 (374h42m35.500557995s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.142644924 (374h42m35.501303584s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143208398 (374h42m35.501867057s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.143631459 (374h42m35.502290115s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144137917 (374h42m35.502796575s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.144798742 (374h42m35.503457399s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145316372 (374h42m35.503975035s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.145683343 (374h42m35.504342s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146054186 (374h42m35.504712844s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146432692 (374h42m35.505091362s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146494647 (374h42m35.505153314s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146843309 (374h42m35.505501962s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.146997913 (374h42m35.505656572s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.147219514 (374h42m35.505878168s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.149208265 (374h42m35.507866919s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Uploading piece... Nov 27 00:26:50.783592790 (374h42m36.142251433s) [upload=330026]: Next returned nil segment Nov 27 00:26:50.783612368 (374h42m36.142271032s) [upload=330026]: Waiting for error group managing segments... Nov 27 00:26:50.922295626 (374h42m36.28095428s) [noise=true , address=145.79.227.41:4086 , piece_id=6LKAEL5MUWMKKIAEIQLARH3DH4X2U7HRRR6RYRPX666TJLRGUEGQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.925351661 (374h42m36.284010322s) [noise=true , address=145.79.227.7:4295 , piece_id=Q2YH6U2LI3JZMQGJFUZFGAI7OFNQHNC7IRVFTLFXAM7RJBBIA53Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.929125519 (374h42m36.287784175s) [noise=true , address=145.79.227.6:4257 , piece_id=76T3JR7SS7I4KOZMFXSGBYK5DJAYY2X7FEQ6LAAW3XNZRBORT6TQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.942099103 (374h42m36.30075776s) [noise=true , address=145.79.227.38:4979 , piece_id=PHP3TEUL7OSRFLZ52DT5H34APC7RZ6ZMIJCWZHAMD6Y43BBNNPTA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948409107 (374h42m36.307067763s) [noise=true , address=145.79.227.17:4972 , piece_id=QP6ZZLGPFSSIGABMXJ2QWZDFDQP4F77LWDNTKTZ2MZIGYFREM6MQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.948763010 (374h42m36.307421668s) [noise=true , address=145.79.227.19:4079 , piece_id=B5FC5KPYFWFBVIJFVY4SX2Y6PQESSUWWY2PT2ELMUCYUSY2EKQCA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.949882225 (374h42m36.30854088s) [noise=true , address=145.79.227.11:4732 , piece_id=A3Z4UQM3YTTYZKD653ZY4ZN5WYNPQMXO4F6LPT3QJ7BV2L7DBEHA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.950027506 (374h42m36.30868616s) [noise=true , address=145.79.227.22:4511 , piece_id=HCNWYOCM7GIHCGYQXN4ACSHJIGCMXRFMJYINDRZBFMZSMNJ63X7A, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:50.970017107 (374h42m36.32867575s) [noise=true , address=145.79.227.2:4113 , piece_id=HNLJWNB5TOZYSMD5H4NHGQU2AF3QIWYCT74Z5PO4WOMKBD6WGLBA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:51.048106490 (374h42m36.406765134s) [noise=true , address=145.79.227.40:4051 , piece_id=CQJCJQ3JGHSMSWZTWFIZCFUYOTTZ664FOLYTCHHXWH7QWPN5TVOQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:52.725148271 (374h42m38.083806929s) [noise=true , address=145.79.227.16:4909 , piece_id=6N5C44DAURQNPZJSURK4MT3CXQTHM4N4HUDELFUDFNSXFR7JDREA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:26:55.075827515 (374h42m40.43448617s) [noise=true , address=145.79.227.30:4762 , piece_id=EZVEHRLDJIHHKQYZUCTPW5ALEP6KR2FNTPZVZTUVGE56GLZGONNQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: <nil> Nov 27 00:27:19.749363092 (374h43m5.10802176s) [noise=true , address=45.91.116.9:4263 , piece_id=KQ7OEZYEYH7EOF6O52BH3QLH6WXY2EV2YQSWEWHUAA22OWYMNRKQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53548->45.91.116.9:4263: use of closed network connection Nov 27 00:27:19.763581648 (374h43m5.122240315s) [noise=true , address=77.90.31.10:4519 , piece_id=TKSWUCLHTLANVJTV3IDRC2WC5DSFCKOBV7EVOLCYWV4IVC727YFA, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39808->77.90.31.10:4519: use of closed network connection Nov 27 00:27:19.763760859 (374h43m5.122419523s) [noise=true , address=77.90.31.17:4771 , piece_id=AEATL5IEAPN7OFT7SQSH6PUTV2UGCHEQUMU5TDN272OCMDPELB6Q, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53286->77.90.31.17:4771: use of closed network connection Nov 27 00:27:19.763919553 (374h43m5.122578214s) [noise=true , address=77.90.31.14:4661 , piece_id=UF7TDEI5ZYXDO2V6KLTGXVOWHBBFTVPTKFRL7HJSVCNINV37MZAQ, seg_pos={0 0}, upload=330026]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38234->77.90.31.14:4661: use of closed network connection Nov 27 00:27:19.763951810 (374h43m5.122610476s) [seg_pos={0 0}, upload=330026]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 00:27:19.870976147 (374h43m5.229634804s) [upload=330026]: Done uploading segments... Nov 27 00:27:19.871013129 (374h43m5.22967179s) [upload=330026]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.338705414s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)