Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.637944168s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:22:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.637968994s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.592418889s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:22:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.592448577s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.408275379s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:22:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.408304013s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.769385372s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:21:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.76948449s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.676332877s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:21:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.676356937s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.450338192s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:21:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.450370399s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.790499087s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:20:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.790525604s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.593932849s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:20:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.593960216s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.754383611s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:20:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.754701449s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.843838947s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:19:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.843866745s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.415898961s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:19:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.415929741s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.645475862s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:18:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.645499736s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.963809016s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:18:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.963835018s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.534237318s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:17:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.534259688s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.381382848s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:17:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.381412232s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.651086731s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:16:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.651112551s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.491699793s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:16:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.491724728s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.775880122s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:15:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.775901811s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.516560977s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:15:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.516592553s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.790763116s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:14:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.79078601s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.372870639s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:14:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.372903657s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.779507458s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:13:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.779537253s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.390894345s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:13:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.390915856s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.777507471s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:12:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.777532527s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.407951407s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:12:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.407975532s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.769543596s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:11:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.769594722s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.647465598s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:11:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.647492808s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.39130147s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:11:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.391342155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.342536719s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:10:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.342562866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.362028596s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:10:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.362054417s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.746916683s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:10:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.746945189s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.466702634s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:10:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.466729138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.8868275s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:10:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.886860969s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.944456608s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:09:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.944484798s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.42687645s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:09:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.426918582s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.649431332s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:08:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.649573547s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.359041976s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:08:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.359075097s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.87877687s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:08:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.878808792s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.315840668s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:07:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.315873081s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.621120303s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:07:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.621147618s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.288937516s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:06:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.28896506s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.81161891s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:06:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.811645255s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.443722541s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:06:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.443742923s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.687145819s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:05:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.687166397s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.408994459s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:05:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.409024764s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.621459442s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:04:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.621481228s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.481668633s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:04:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.481696883s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.386024789s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:03:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.386052749s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.604031259s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:03:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.604061118s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.603020508s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:02:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.603034866s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)