Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.56765576s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:41:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.567681966s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.287918325s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:41:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.287944019s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.829197128s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:41:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.829237188s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.353853109s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:40:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.35388141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.402990807s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:40:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.403019002s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.792196822s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:40:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.792241748s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.459381336s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:40:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.459408027s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.758685778s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:39:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.758713588s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.67079151s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:39:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.670820908s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.489870003s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:38:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.489898483s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.53572433s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:38:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.5357509s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.777363185s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:38:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.777389958s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.482600079s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:37:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.482631705s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.66000073s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:37:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.660026102s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 8.63310805s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:36:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.633139719s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.528696872s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:36:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.52872675s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.38094289s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:35:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.380973667s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.643245034s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:35:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.643336325s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.352095118s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:35:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.352124139s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.650318413s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:34:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.650340055s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.351783731s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:34:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.351808707s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.541755905s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:34:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.541773308s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.361367707s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:34:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.361466474s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.610487697s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:33:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.610514668s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.464086046s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:33:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.464121629s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.727028817s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:32:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.727054298s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.383386586s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:32:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.383435418s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.720404666s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:32:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.720501103s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.674317053s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:31:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.674406833s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.384498985s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:31:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.384525217s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.454842158s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:30:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.454866817s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.6592057s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:30:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.659233425s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.433599522s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:30:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.433706196s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 3.077890889s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:29:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.077937096s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.33999159s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:29:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.340020196s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.627268259s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:28:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.627300971s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.402836577s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:28:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.40286434s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.633960742s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:27:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.63398368s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.382856587s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:27:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.382883452s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.784427072s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:27:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.78452038s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.520124894s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:26:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.520148117s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.395979722s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:26:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.396004273s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.581941147s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:26:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.582022236s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.361041666s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:25:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.361065681s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.747000286s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:25:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.747029569s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.399530408s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:24:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.399557708s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.663608166s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:24:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.663637736s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.402812042s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:23:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.402844412s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.411791838s trace: https://jaeger.dev.storj.io/trace/655d188af1aa270 FAIL (2025-12-08 19:23:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.411815698s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.62926445s trace: https://jaeger.dev.storj.io/trace/6086529041566513 FAIL (2025-12-08 19:23:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 18:54:57.674974846 (75h26m37.655327486s) [upload=66264]: Starting upload Dec 8 18:54:57.675038729 (75h26m37.655391369s) [upload=66264]: Done starting upload Dec 8 18:54:57.675330133 (75h26m37.655682761s) [upload=66264]: Uploading segments... Dec 8 18:54:57.675406307 (75h26m37.655758943s) [upload=66264]: Got next segment. Inline: false Dec 8 18:54:57.675485211 (75h26m37.655837857s) [upload=66264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 18:54:57.675493430 (75h26m37.655846073s) [upload=66264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 18:54:57.785146518 (75h26m37.765499144s) [seg_pos={0 0}, upload=66264]: Begin upload segment... Dec 8 18:54:57.785240478 (75h26m37.765593118s) [seg_pos={0 0}, upload=66264]: Done begin upload segment. Dec 8 18:54:57.785485278 (75h26m37.765837924s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.785679403 (75h26m37.766032052s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.786641527 (75h26m37.766994177s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787110826 (75h26m37.767463468s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787261208 (75h26m37.767613854s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787489023 (75h26m37.76784167s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787843253 (75h26m37.768195901s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.787852122 (75h26m37.768204761s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788192899 (75h26m37.768545534s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788412033 (75h26m37.76876468s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788516029 (75h26m37.768868665s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.788933691 (75h26m37.769286334s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789013511 (75h26m37.769366149s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.789551986 (75h26m37.769904633s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790354423 (75h26m37.770707062s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:57.790934325 (75h26m37.771286971s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Uploading piece... Dec 8 18:54:58.172348027 (75h26m38.15270068s) [noise=true , address=77.90.31.12:4590 , piece_id=P6H4KDKDRZZWGB3FVK4KB42KORBAHHHJES65VCTFD72D4JH5VJVA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.172831033 (75h26m38.153183679s) [noise=true , address=145.79.227.30:4763 , piece_id=5YDXU37WXFV36D2XH76XQ6BNY45SYZYTWGV3HBEFNVVT5LBW3NNQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.178590990 (75h26m38.158943635s) [noise=true , address=77.90.31.12:4589 , piece_id=XA2OR2VJMLK6LGEABP7UBIQNZG6BXVMUX5XP7SVMLBEKJE5YDP7Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 18:54:58.392481216 (75h26m38.372833849s) [upload=66264]: Next returned nil segment Dec 8 18:54:58.392499713 (75h26m38.372852355s) [upload=66264]: Waiting for error group managing segments... Dec 8 18:54:58.531890881 (75h26m38.512243518s) [noise=true , address=77.90.31.11:4554 , piece_id=MJEVYVGP5NQZENUOE3NNBBS3BYC52642XB5OGF2K6J7YHNYORNPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532848367 (75h26m38.513201014s) [noise=true , address=77.90.31.17:4772 , piece_id=Q4QNOKUPLIVMP7DVRAQSYKVCBIFYVVL7CYTVOKYYLQZSXD2JMRCQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.532895902 (75h26m38.513248545s) [noise=true , address=145.79.227.41:4085 , piece_id=HKKDK2YHRBBULRLBYAVC7QZQR2HGFH4DXZDJAS6EYR654U7SV2RA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.536318192 (75h26m38.516670836s) [noise=true , address=145.79.227.27:4690 , piece_id=S6DIBWGQA3NAM3E3AS5QIB7QJL2SYSDFGANJYOZA3MTT6MQZS7AQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.538479769 (75h26m38.518832418s) [noise=true , address=145.79.227.21:4474 , piece_id=4PXKGNK6IBHWO7TKWG2O3BJI5P4MSYUWZD4NJ7JMOXP2TCY3VVWA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.539308517 (75h26m38.519661165s) [noise=true , address=145.79.227.39:4014 , piece_id=XPW63KAMFGYPJULSRSYPO3V5PIJX5DH7BLGFCDSYRSU46AXCYYRA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.541778178 (75h26m38.522130824s) [noise=true , address=77.90.31.21:4878 , piece_id=7PHBVJ5HENMIOKLSRV34B6LOBJUGG6O3LUY4J3M6EFHV57JPXLSA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.553574259 (75h26m38.533926895s) [noise=true , address=145.79.227.25:4619 , piece_id=IPCP5WU2OE2JZDF47UKPU7GFPEIKLLBETHPV6UOQ7EYZG7QXUZJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.555434734 (75h26m38.535787374s) [noise=true , address=145.79.227.20:4439 , piece_id=HDPIFW4OJ6CY5LKLJ57IPHRPFDTXHVDTRJ7FKAXDY7PER6QPXQAA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556116312 (75h26m38.536468949s) [noise=true , address=145.79.227.32:4835 , piece_id=AFJ6KUBQP5LSAJUSUN7DOSNJ5TBWBBQRRYBTARKHKMOCWGDHDRJQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.556487637 (75h26m38.536840274s) [noise=true , address=145.79.227.22:4511 , piece_id=HE3JRGJIDNHPZFAXR2Y3HOXYZSCOKQ4OP6VPCLBLLZUFMBYU3WOA, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:54:58.714141565 (75h26m38.694494204s) [noise=true , address=145.79.227.33:4871 , piece_id=HCUMHPIMTFBWYNYL5NHFR5AQSYSIDIN2D7EO7ZYHG5GUCAYMLAPQ, seg_pos={0 0}, upload=66264]: Done uploading piece. err: <nil> Dec 8 18:55:27.392456241 (75h27m7.372808888s) [noise=true , address=77.90.31.16:4735 , piece_id=P77WZF5A3CNM7ZXREICQQUGBDYJPHYQIQ4K23NKEXXBY4XFZV73Q, seg_pos={0 0}, upload=66264]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 18:55:27.392541526 (75h27m7.372894167s) [seg_pos={0 0}, upload=66264]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 18:55:27.504922383 (75h27m7.485275021s) [upload=66264]: Done uploading segments... Dec 8 18:55:27.504964045 (75h27m7.485316691s) [upload=66264]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.629294695s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)