Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.378892001s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:53:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.378929851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.388249248s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:52:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.388281495s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.430500809s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:51:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.430525927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.455873056s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:50:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.455921409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.365680029s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:49:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.365704196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.340578495s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:49:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.340609093s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.425180717s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:48:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.425226803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.353478218s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:47:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.353505739s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.377616389s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:47:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.377645132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.394625219s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:46:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.394678661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.39095558s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:45:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.390992537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.334210095s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:45:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.334237563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.34926469s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:44:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.349293802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.3710703s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:44:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.371108235s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.319619529s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:43:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.319648415s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.372973062s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:42:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.372995293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.37366647s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:41:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.37370246s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.436995189s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:41:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.437021778s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.355885639s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:40:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.355912863s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.325735838s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:39:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.325762876s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.329501804s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:38:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.329544519s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.318674321s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:37:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.318700701s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.337928662s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:37:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.337986886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.412824514s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:36:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.412852997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.373774462s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:35:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.373802225s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.355249604s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:34:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.355273703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.341175157s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:33:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.341199537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.413539414s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:32:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.413568159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.484991917s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:32:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.485026748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.394401883s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:31:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.39448637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.345032214s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:30:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.345072901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.332480887s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:29:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.332497899s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.317187265s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:29:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.317214088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.419064258s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:28:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.419095161s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.459870525s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:27:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.45989724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.388541697s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:27:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.388567034s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.397100932s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:26:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.397137287s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.461798897s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:26:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.461831356s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.37900413s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:25:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.379143883s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.373176705s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:24:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.37320798s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.460278102s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:24:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.460306934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.361182499s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:23:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.36121607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.575466215s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:23:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.575554622s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.360449522s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:22:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.360470998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.1164861s trace: https://jaeger.dev.storj.io/trace/3554b0f003d717c1 FAIL (2025-12-08 21:22:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 21:22:12.647550988 (77h53m52.627903628s) [upload=68436]: Starting upload Dec 8 21:22:12.647714857 (77h53m52.628067498s) [upload=68436]: Done starting upload Dec 8 21:22:12.648023817 (77h53m52.628376456s) [upload=68436]: Uploading segments... Dec 8 21:22:12.648466582 (77h53m52.628819221s) [upload=68436]: Got next segment. Inline: false Dec 8 21:22:12.648617152 (77h53m52.628969791s) [upload=68436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:22:12.648626126 (77h53m52.62897877s) [upload=68436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:22:12.761687878 (77h53m52.742040515s) [seg_pos={0 0}, upload=68436]: Begin upload segment... Dec 8 21:22:12.761894065 (77h53m52.742246701s) [seg_pos={0 0}, upload=68436]: Done begin upload segment. Dec 8 21:22:12.761965122 (77h53m52.742317768s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762620974 (77h53m52.742973613s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762642262 (77h53m52.742994909s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.762997033 (77h53m52.743349671s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763191884 (77h53m52.743544521s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763362727 (77h53m52.743715364s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763689981 (77h53m52.744042636s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.763711677 (77h53m52.744064313s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764109403 (77h53m52.744462038s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764173966 (77h53m52.744526608s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764412945 (77h53m52.744765581s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764645883 (77h53m52.744998522s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.764728198 (77h53m52.745080845s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765058377 (77h53m52.745411013s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765127252 (77h53m52.745479889s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:12.765565309 (77h53m52.745917951s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Uploading piece... Dec 8 21:22:13.142971751 (77h53m53.1233244s) [noise=true , address=77.90.31.15:4699 , piece_id=GBOLI6RBKGYCKWHMBKUQ2OLGF3TARFG5WHODPYIQ5OPSEDCVWL3Q, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.150272987 (77h53m53.130625634s) [noise=true , address=77.90.31.13:4627 , piece_id=5BNVBYTFKCLVMC7RPA665YJWWCXYJADO4GFQL7U4KIRUFNYXJK3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:13.372969513 (77h53m53.353322192s) [upload=68436]: Next returned nil segment Dec 8 21:22:13.372987277 (77h53m53.353339925s) [upload=68436]: Waiting for error group managing segments... Dec 8 21:22:13.520298502 (77h53m53.500651145s) [noise=true , address=145.79.227.24:4582 , piece_id=UKJKEEOVT6YHA6X4P7V5IZSG6I32AREI7AYVP4TRZD5P4MH2FUHQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.521087749 (77h53m53.501440388s) [noise=true , address=145.79.227.42:4121 , piece_id=AS6YDZYFRUW6UVWY5VOO5ONHVZXQO4ZUAHHMVF6OWNAA4TSPU66A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522403776 (77h53m53.502756417s) [noise=true , address=145.79.227.14:4840 , piece_id=LZIKKXMFMSB3AEVZFVAQGHXJU6RNTFYOHMUDOWMQDMZ3C6YLZJSQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.522994299 (77h53m53.50334693s) [noise=true , address=145.79.227.21:4473 , piece_id=CP6BWDFD4CIK3HOVHCXNE5VUZ467JFNIOX7TFLYOITNCNO76UWMA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.532075444 (77h53m53.512428081s) [noise=true , address=145.79.227.43:4157 , piece_id=35CSVNGFCQGAXZX4OCSRPH4MEN3SJRRSFAZJBGA4O2KRRXQLMOUQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.534681159 (77h53m53.515033788s) [noise=true , address=145.79.227.41:4087 , piece_id=PXBFX3UYZPNKSQD7ETGBLW4K32M22ZINP5VDYGI6JGVW2CRC7YWQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.536267604 (77h53m53.516620252s) [noise=true , address=145.79.227.21:4474 , piece_id=3JL7DIKXXBW5BMAD6RP6INIUYQKGPFNRHPZCJYJGDTACB67GMKCA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.537048848 (77h53m53.517401489s) [noise=true , address=145.79.227.31:4798 , piece_id=YDHJPE7KGU6EZITCXOWYFW54N5ADYUMPHDWQWWTN5QTYD6TWP6XQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.538963860 (77h53m53.519316503s) [noise=true , address=145.79.227.34:4907 , piece_id=YGC77JJO2ABYOO5P5YOYTX5EZEE6YJH3WCK3LWJ2ESD7VSFPES3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553082108 (77h53m53.533434753s) [noise=true , address=145.79.227.25:4619 , piece_id=KQYD4HBFFHEA4655UCXT3RFJ46JH75GK47Z4FGYQSHJAGHM5TSQA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.553442890 (77h53m53.533795522s) [noise=true , address=145.79.227.40:4049 , piece_id=SK6JZIQRFPZ2QK7BSEHMS6LNWCKWUZMHUCHR7NQJNVP3MEGQWOSA, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:13.555749231 (77h53m53.536101871s) [noise=true , address=77.90.31.14:4661 , piece_id=PMJGSLE6MJJP2GTPEOB22ITRAXBVDNFCYF2N4Q4Z3KUXAILMYZQQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: <nil> Dec 8 21:22:14.069953329 (77h53m54.05030598s) [noise=true , address=77.90.31.18:4806 , piece_id=HTNB4SXWX2WMICD7P62I53ZCX3USIB5H3C3TFJUKFQHH24N4NS3A, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: storage node overloaded, request limit: 2000 Dec 8 21:22:42.347656621 (77h54m22.328009269s) [noise=true , address=77.90.31.12:4589 , piece_id=PDRY4BRYUBFMCCLJDS56XN6OUUZRB5MCUGJ2NUA7FV67PRWQAZJQ, seg_pos={0 0}, upload=68436]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:44556->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 21:22:42.347721381 (77h54m22.328074025s) [seg_pos={0 0}, upload=68436]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 21:22:42.459989213 (77h54m22.440341863s) [upload=68436]: Done uploading segments... Dec 8 21:22:42.460042007 (77h54m22.440394657s) [upload=68436]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.116504492s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.724725873s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:44:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.72482651s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.53703244s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:43:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.537060836s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.613529461s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:43:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.613553868s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.729512101s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:42:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.729554073s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.402996897s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:42:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.403261095s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)