Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.442436771s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 13:02:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.442472318s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.564538114s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 13:01:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.564571206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.565434077s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 13:00:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.565465764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.361326746s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:59:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.361420357s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.464293707s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:57:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.464322576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.385388574s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:57:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.385435491s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.469747688s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:56:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.469773242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.262786636s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:55:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.262815722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.337973945s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:54:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.338063893s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.486977248s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:53:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.487004885s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.230783904s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:53:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.230809312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.25202454s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:52:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.252055511s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.444602406s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:50:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.444631184s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.285165687s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:50:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.285201445s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.398302945s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:49:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.398339707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.320612227s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:49:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.320641282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.290029819s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:48:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.290130343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.290490517s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:47:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.290516425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.303310504s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:46:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.303354262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.413652935s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:45:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.413683664s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.472866547s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:44:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.472892607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.336758171s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:43:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.336778496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.272148305s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:43:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.272171968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.326459115s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:41:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.326499293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.11901208s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 12:41:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.119034705s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.673871283s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 17:09:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.673897712s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 2.01364223s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 17:08:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.013690777s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 1.670619931s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 17:07:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.670703338s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 1.599637344s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 17:07:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.599691591s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 2.125286563s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 17:06:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.12537946s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 1.577138335s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 17:05:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.577164354s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 2.198162199s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 17:04:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.19818462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.790282675s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 17:03:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.790311076s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.674307514s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 17:03:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.674333791s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 1.593447683s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 17:02:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.593501237s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 1.631160812s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 17:01:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.631187762s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 1.628887218s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 17:00:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.628912627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 1.720588446s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 16:59:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.720620315s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X............X..............................................] duration: 1.663426049s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 16:58:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.663470546s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X............X.............................................] duration: 1.70410676s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 16:57:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.704133959s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X............X............................................] duration: 1.612053034s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 16:56:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.612079754s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X............X...........................................] duration: 1.632163058s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 16:56:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.632718668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X............X..........................................] duration: 1.590898524s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 16:55:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.590926919s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X............X.........................................] duration: 1.595078098s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 16:54:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.595107373s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X............X........................................] duration: 1.675256037s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 16:53:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.67530091s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X............X.......................................] duration: 1.661633212s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 16:53:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.661652212s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X............X......................................] duration: 1.529472597s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 16:52:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.529505441s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X............X.....................................] duration: 2.222594716s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 16:51:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.222624831s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X............X....................................] duration: 1.629332489s trace: https://jaeger.dev.storj.io/trace/661c738c90f3b3b1 FAIL (2025-12-13 16:50:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.629377476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.311950899s trace: https://jaeger.dev.storj.io/trace/41b9e339794a76fe FAIL (2025-12-13 16:49:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.311978573s core uplink ap1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)