Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.37445643s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:43:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.374483146s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.388290897s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:43:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.388312583s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.607123665s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:43:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.60715217s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.353920386s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:42:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.353957712s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.820090364s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:42:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.820122914s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.399829254s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:42:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.399857192s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.470851357s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:42:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.471015431s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.517897139s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:41:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.517998451s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.49825112s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:41:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.498280576s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.383536643s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:41:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.383587149s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.532336011s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:40:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.532363062s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.419497306s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:40:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.419546955s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.43931056s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:40:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.439456512s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.368250626s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:39:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.368281849s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.584102019s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:39:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.584132281s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.421299125s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:39:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.421327967s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.461061223s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:38:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.461085711s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.345530086s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:37:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.345573279s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.465179643s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:37:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.465229708s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.449953845s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:37:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.449984034s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.483690636s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:36:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.483716645s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.870817727s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:36:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.870959445s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.626599226s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:35:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.627137851s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.388276558s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:34:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.388314785s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.474614148s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:34:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.4746406s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.801103001s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:34:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.801130846s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.431989224s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:33:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.432017247s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.774352955s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:33:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.774386654s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.409137273s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:33:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.409164604s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.333719969s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:32:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.333810388s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.637714615s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:32:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.637749397s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.845614077s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:32:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.845647017s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.477872164s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:31:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.477901527s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.11982925s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 03:31:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.11985534s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.504525307s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:31:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.50456484s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001159256s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 03:31:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001247137s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.566657011s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:28:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.566678159s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.641257191s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:28:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.64128549s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 1.590423817s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:27:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.590449504s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 1.50992202s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:27:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.50994812s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 1.680551817s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:26:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.681359766s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 1.548764045s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:25:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.548782672s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 2.053332268s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:25:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.053367437s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.502704354s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:24:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.502744222s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.658845858s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:23:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.658883794s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 1.896101893s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:22:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.896129778s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 1.625663807s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:21:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.625695676s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 1.640746235s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:20:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.64082375s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 2.103011405s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:19:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.103043294s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 2.127680265s trace: https://jaeger.dev.storj.io/trace/7889f4ab501e1032 FAIL (2025-12-19 22:18:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.127708678s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)