Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.362925201s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:03:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.362962155s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.73151081s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:02:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.731546909s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.461694542s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:02:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.461751327s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.54211254s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:02:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.542189724s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.526004193s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:02:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.526034561s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.396917105s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:01:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.396945309s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.503483421s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:01:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.503520014s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.421445371s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:00:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.421488456s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.641448963s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:00:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.641466562s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.405559904s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:59:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.405589837s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.347056676s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:59:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.347136941s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.829376585s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:59:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.829423522s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.359582174s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:58:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.359606726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.327800421s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:58:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.327829474s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.764526298s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:58:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.764551383s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.446992772s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:57:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.447021644s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.628701632s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:57:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.628729753s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.363156374s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:56:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.363179743s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.434864361s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:56:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.434894685s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.543200088s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:56:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.543351913s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.354290994s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:55:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.354323885s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.361894951s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:55:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.361917816s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.623788076s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:55:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.623838919s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.402808528s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:54:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.402839599s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.387085098s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:54:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.387114571s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.580238942s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:53:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.580271462s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.39588779s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:53:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.395919087s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.742567654s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:53:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.742622393s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.357302958s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:52:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.35732798s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.296730552s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:52:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.296762464s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.495557187s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:52:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.495585221s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.386257207s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:51:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.386286377s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.530695026s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:51:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.530716943s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.422710273s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:51:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.42281026s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.408219629s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:50:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.408234458s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.314862961s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:49:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.314890332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.371123079s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:49:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.371154949s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.529665638s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:49:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.52970913s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.829848762s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:49:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.829971459s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.398842748s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:48:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.398871954s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.457280912s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:47:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.45737647s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.317453234s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:47:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.317476974s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.535336743s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:46:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.53536571s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.358410631s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:46:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.358499361s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.580676295s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:46:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.580783591s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.52231321s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:45:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.522344664s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.3538105s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:45:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.353839129s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.809141116s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:45:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.809239193s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.41259646s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:44:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.41270938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.365703123s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:44:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.365730595s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)