Orbiter History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.55606711s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 05:08:21 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.556097579s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.634245374s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 05:07:49 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.634273869s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.556279188s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 05:07:07 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.556307335s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.467646771s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 05:06:38 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.467693088s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.512863708s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 05:05:59 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.512888893s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.54582791s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 05:05:29 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.545852376s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.484276227s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 05:05:11 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.484303902s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.506900239s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 05:04:16 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.506933239s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.474185815s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 05:03:40 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.474212483s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.593638597s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 05:03:04 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.593673598s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.541897526s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 05:01:57 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.541930372s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.859455108s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 05:00:54 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.859485339s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.575007006s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 04:59:43 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.575046049s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.514116186s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 04:59:12 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.514210737s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.56443224s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 04:58:38 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 1.564503856s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001182584s trace: https://jaeger.dev.storj.io/trace/25d4caaac4d9207a FAIL (2025-12-20 04:57:53 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 30.0011998s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.78527966s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:16:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.78537624s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.717576721s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:15:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.717602225s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.475317477s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:15:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.475339094s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.663708555s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:14:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.663769551s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.423984236s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:14:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.424017425s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.666119321s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:14:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.666148756s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.334234123s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:13:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.334257301s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.532950414s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:13:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.532983602s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.338147491s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:12:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.33817559s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.376798419s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:12:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.376831308s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.703174907s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:12:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.703266172s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.499454s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:11:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.499484155s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.880829021s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:11:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.880855799s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.547238709s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:11:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.547268248s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.505470438s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:10:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.505496491s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.398248585s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:09:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.398280114s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.591680662s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:09:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.591704309s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.521312591s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:09:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.521348802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.370592856s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:08:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.370616346s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.382187735s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:08:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.382214598s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.671726902s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:08:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.671755098s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.541025962s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:07:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.541045569s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.372889859s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:07:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.372913816s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.359874932s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:07:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.359899911s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.32812316s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:06:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.328154386s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.589679564s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:06:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.589707835s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.353202728s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:06:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.353239511s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.421486523s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:05:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.421511051s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.46550439s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:05:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.46553067s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.348143314s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:04:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.348170687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.439178887s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:04:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.439203668s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.722670904s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:04:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.722695485s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.371808224s trace: https://jaeger.dev.storj.io/trace/5c49b785065a6f85 FAIL (2025-12-20 04:03:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 20 03:31:18.860282677 (348h2m58.840635313s) [upload=306077]: Starting upload Dec 20 03:31:18.860349072 (348h2m58.840701697s) [upload=306077]: Done starting upload Dec 20 03:31:18.860434055 (348h2m58.840786679s) [upload=306077]: Uploading segments... Dec 20 03:31:18.860628304 (348h2m58.840980926s) [upload=306077]: Got next segment. Inline: false Dec 20 03:31:18.860675566 (348h2m58.841028203s) [upload=306077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 03:31:18.860680540 (348h2m58.841033175s) [upload=306077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 03:31:18.970120706 (348h2m58.950473351s) [seg_pos={0 0}, upload=306077]: Begin upload segment... Dec 20 03:31:18.970183005 (348h2m58.950535646s) [seg_pos={0 0}, upload=306077]: Done begin upload segment. Dec 20 03:31:18.970222730 (348h2m58.950575375s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970239506 (348h2m58.950592151s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.970815365 (348h2m58.951168011s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971224115 (348h2m58.951576754s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971322709 (348h2m58.951675353s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971740786 (348h2m58.952093422s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.971794555 (348h2m58.952147191s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972171460 (348h2m58.952524096s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972261642 (348h2m58.952614288s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972530181 (348h2m58.952882823s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972732301 (348h2m58.953084947s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.972829451 (348h2m58.953182091s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973098019 (348h2m58.953450656s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973248446 (348h2m58.953601089s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.973730327 (348h2m58.954082973s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:18.974248149 (348h2m58.954600795s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Uploading piece... Dec 20 03:31:19.352515767 (348h2m59.332868413s) [noise=true , address=145.79.227.30:4762 , piece_id=ULEOBNHH7N7VXOXSX7VMXIBL53IIGZR6FPZAKMSIHYS435C5VYHA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.353255868 (348h2m59.333608531s) [noise=true , address=145.79.227.43:4158 , piece_id=IPHO73DIGBK5RGI2KEYEQJNVNFXJPGEYIXGIOQ4L7VPU4E6QHM7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: storage node overloaded, request limit: 2000 Dec 20 03:31:19.574491252 (348h2m59.554843935s) [upload=306077]: Next returned nil segment Dec 20 03:31:19.574511303 (348h2m59.554864016s) [upload=306077]: Waiting for error group managing segments... Dec 20 03:31:19.719299885 (348h2m59.699652521s) [noise=true , address=145.79.227.52:4446 , piece_id=VYJUBXMRPSLVP253GY2CMVM7HK4TX6ZZJAN7TZQP7YKM2XWIIYDA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720129285 (348h2m59.700481925s) [noise=true , address=145.79.227.45:4229 , piece_id=LBSYQIO6SXLQ4LTWFG6T6AZJQY74ETIPA3FQABALACLQNXA4X7HQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720211522 (348h2m59.700564158s) [noise=true , address=145.79.227.16:4909 , piece_id=DSF4ZD75EJP3Q6NYJ3ATXESR57QHLP6PY5D2CFZCIY627NIDHAKA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.720930461 (348h2m59.701283097s) [noise=true , address=145.79.227.2:4114 , piece_id=HIMQ6375GF5244NHWUY4UBW5M35A3VH3URXF5J5S47XTOICMR3TQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.723228185 (348h2m59.703580831s) [noise=true , address=145.79.227.42:4122 , piece_id=SK3MTOI7P2J3EJSRAMSYSDBN2X5RMIIVWAOFQIOLDUCV4FESJKFA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.731915478 (348h2m59.712268118s) [noise=true , address=145.79.227.36:4518 , piece_id=D5KCKZ73VIKFEZZJU7R3XVPAA4ANT2P3DX5B7M6GM6RKMO7LUT2A, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733738530 (348h2m59.714091166s) [noise=true , address=145.79.227.51:4411 , piece_id=TMNMCAUUTFEUOUFQ2LINSG4NER5EJUHOICNWNVWUWBKESAHJJZDQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.733951910 (348h2m59.714304549s) [noise=true , address=145.79.227.33:4871 , piece_id=3KBTHI6G23HCJ4TWDSK66IBTSSC2ZO64WM4JZCG3A7PXLIH2JEWQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.734318272 (348h2m59.714670908s) [noise=true , address=45.91.116.7:4011 , piece_id=ANMJAY5QIKMYSV52QLIBDQHWQZWDBOVJJJNNS7S77NMLPYEFGEJQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.736092066 (348h2m59.716444703s) [noise=true , address=145.79.227.35:4483 , piece_id=VX6RNUG3IRMVE4NAZGAGCYJURZUJRZN3ZXJTVUC63GZKU6YDLBXA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:19.862267756 (348h2m59.842620402s) [noise=true , address=77.90.31.14:4663 , piece_id=A6RJC2X2GLUY6NFSIGYAUDOZJDQQWC46ANWI2FJVPXHBPED2OREQ, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:28.080568510 (348h3m8.060921147s) [noise=true , address=145.79.227.27:4689 , piece_id=KKSGIS2WQGYBUIVKM44WGPLNKEPAEOI4V6KPCPX4QBGI52REXB7Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: <nil> Dec 20 03:31:48.585308722 (348h3m28.565661369s) [noise=true , address=145.79.227.40:4050 , piece_id=RM4TDV4OXWWJ2EHPBTSJAMJGDA2CGQNL7LY6FOC56CMUA4SIETQA, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585753119 (348h3m28.566105756s) [noise=true , address=145.79.227.29:4943 , piece_id=O3PCGPVCTZTKCPKH6QFBQOVSCFQ4S557OB3MBV5ON4UJRJQF5G3Q, seg_pos={0 0}, upload=306077]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 03:31:48.585784463 (348h3m28.566137099s) [seg_pos={0 0}, upload=306077]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 20 03:31:48.703237459 (348h3m28.683590099s) [upload=306077]: Done uploading segments... Dec 20 03:31:48.703270572 (348h3m28.683623217s) [upload=306077]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.371899706s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.617503087s trace: https://jaeger.dev.storj.io/trace/7e24689593751449 FAIL (2025-12-20 04:03:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.617520923s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)