Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.769543596s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:11:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.769594722s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.647465598s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:11:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.647492808s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.39130147s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:11:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.391342155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.342536719s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:10:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.342562866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.362028596s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:10:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.362054417s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.746916683s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:10:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.746945189s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.466702634s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:10:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.466729138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.8868275s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:10:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.886860969s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.944456608s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:09:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.944484798s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.42687645s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:09:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.426918582s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.649431332s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:08:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.649573547s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.359041976s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:08:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.359075097s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.87877687s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:08:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.878808792s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.315840668s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:07:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.315873081s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.621120303s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:07:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.621147618s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.288937516s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:06:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.28896506s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.81161891s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:06:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.811645255s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.443722541s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:06:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.443742923s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.687145819s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:05:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.687166397s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.408994459s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:05:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.409024764s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.621459442s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:04:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.621481228s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.481668633s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:04:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.481696883s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.386024789s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:03:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.386052749s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.604031259s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:03:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.604061118s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.603020508s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:02:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.603034866s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.494857089s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:02:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.49488425s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.485258899s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:01:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.485288276s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.656214074s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:01:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.656257839s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.43680789s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:01:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.436830998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.402391231s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:00:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.402409389s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.898137075s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:00:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.898238634s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.8088949s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:00:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.809015399s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.829767865s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 18:59:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.829790184s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.304831991s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 18:59:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.304858731s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001587281s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 18:59:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001611818s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.349970779s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 18:57:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.34999386s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.331322193s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 18:57:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.331344956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.351403103s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 18:56:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.351423796s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.397819522s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 18:55:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.397849469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.114388537s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 18:55:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.114411228s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.498048439s trace: https://jaeger.dev.storj.io/trace/17a981e9c178eaf2 FAIL (2025-12-08 13:12:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.498095033s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.917855892s trace: https://jaeger.dev.storj.io/trace/17a981e9c178eaf2 FAIL (2025-12-08 13:11:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.917891091s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.639618068s trace: https://jaeger.dev.storj.io/trace/17a981e9c178eaf2 FAIL (2025-12-08 13:10:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.639646029s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.866549008s trace: https://jaeger.dev.storj.io/trace/17a981e9c178eaf2 FAIL (2025-12-08 13:10:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.866583299s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.955759366s trace: https://jaeger.dev.storj.io/trace/17a981e9c178eaf2 FAIL (2025-12-08 13:09:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.955782911s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.558166528s trace: https://jaeger.dev.storj.io/trace/17a981e9c178eaf2 FAIL (2025-12-08 13:08:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.558188912s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.681873286s trace: https://jaeger.dev.storj.io/trace/17a981e9c178eaf2 FAIL (2025-12-08 13:08:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.681898301s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.592751974s trace: https://jaeger.dev.storj.io/trace/17a981e9c178eaf2 FAIL (2025-12-08 13:07:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.592779406s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.72600525s trace: https://jaeger.dev.storj.io/trace/17a981e9c178eaf2 FAIL (2025-12-08 13:07:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.726036346s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.740883392s trace: https://jaeger.dev.storj.io/trace/17a981e9c178eaf2 FAIL (2025-12-08 13:06:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.74090494s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)