Orbiter History

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 289.142543ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9ee9e5385f36196 FAIL (2025-12-02 18:05:04 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.57.18:443: connect: connection refused Duration: 289.165715ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 868.241451ms trace: https://jaeger.dev.storj.io/trace/3b7a4a4ca5a9a00a FAIL (2025-12-02 18:04:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer ======LOGS====== Dec 2 17:58:45.945515236 (512h14m31.304173892s) [upload=451109]: Starting upload Dec 2 17:58:45.945584251 (512h14m31.304242898s) [upload=451109]: Done starting upload Dec 2 17:58:45.945681031 (512h14m31.304339673s) [upload=451109]: Uploading segments... Dec 2 17:58:45.945742287 (512h14m31.304400949s) [upload=451109]: Got next segment. Inline: true Dec 2 17:58:45.945747963 (512h14m31.304406605s) [upload=451109]: Waiting for error group managing segments... Dec 2 17:58:45.945800260 (512h14m31.304458906s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:45.945808052 (512h14m31.304466716s) [upload=451109]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:45.945812113 (512h14m31.304470753s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:31.489436266 (512h15m16.848094929s) [upload=451109]: Done uploading segments... Dec 2 17:59:31.489469001 (512h15m16.848127665s) [upload=451109]: Upload finished. err: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer Duration:* 868.27345ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 300.808524ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9ee9e5385f36196 FAIL (2025-12-02 18:04:20 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.57.18:443: connect: connection refused Duration: 300.829846ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.668985291s trace: https://jaeger.dev.storj.io/trace/7eb4a8f9dded1e1f FAIL (2025-12-02 18:04:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.669010784s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 973.485811ms trace: https://jaeger.dev.storj.io/trace/3b7a4a4ca5a9a00a FAIL (2025-12-02 18:03:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer ======LOGS====== Dec 2 17:58:45.945515236 (512h14m31.304173892s) [upload=451109]: Starting upload Dec 2 17:58:45.945584251 (512h14m31.304242898s) [upload=451109]: Done starting upload Dec 2 17:58:45.945681031 (512h14m31.304339673s) [upload=451109]: Uploading segments... Dec 2 17:58:45.945742287 (512h14m31.304400949s) [upload=451109]: Got next segment. Inline: true Dec 2 17:58:45.945747963 (512h14m31.304406605s) [upload=451109]: Waiting for error group managing segments... Dec 2 17:58:45.945800260 (512h14m31.304458906s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:45.945808052 (512h14m31.304466716s) [upload=451109]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:45.945812113 (512h14m31.304470753s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:31.489436266 (512h15m16.848094929s) [upload=451109]: Done uploading segments... Dec 2 17:59:31.489469001 (512h15m16.848127665s) [upload=451109]: Upload finished. err: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer Duration:* 973.508482ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 322.938084ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9ee9e5385f36196 FAIL (2025-12-02 18:03:49 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.57.18:443: connect: connection refused Duration: 322.963017ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.712048464s trace: https://jaeger.dev.storj.io/trace/7eb4a8f9dded1e1f FAIL (2025-12-02 18:03:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.712076212s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.697559487s trace: https://jaeger.dev.storj.io/trace/7eb4a8f9dded1e1f FAIL (2025-12-02 18:03:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.69758232s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 992.721972ms trace: https://jaeger.dev.storj.io/trace/3b7a4a4ca5a9a00a FAIL (2025-12-02 18:02:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer ======LOGS====== Dec 2 17:58:45.945515236 (512h14m31.304173892s) [upload=451109]: Starting upload Dec 2 17:58:45.945584251 (512h14m31.304242898s) [upload=451109]: Done starting upload Dec 2 17:58:45.945681031 (512h14m31.304339673s) [upload=451109]: Uploading segments... Dec 2 17:58:45.945742287 (512h14m31.304400949s) [upload=451109]: Got next segment. Inline: true Dec 2 17:58:45.945747963 (512h14m31.304406605s) [upload=451109]: Waiting for error group managing segments... Dec 2 17:58:45.945800260 (512h14m31.304458906s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:45.945808052 (512h14m31.304466716s) [upload=451109]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:45.945812113 (512h14m31.304470753s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:31.489436266 (512h15m16.848094929s) [upload=451109]: Done uploading segments... Dec 2 17:59:31.489469001 (512h15m16.848127665s) [upload=451109]: Upload finished. err: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer Duration:* 992.747887ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 297.160442ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9ee9e5385f36196 FAIL (2025-12-02 18:02:35 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.57.18:443: connect: connection refused Duration: 297.260626ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 880.087258ms trace: https://jaeger.dev.storj.io/trace/3b7a4a4ca5a9a00a FAIL (2025-12-02 18:02:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer ======LOGS====== Dec 2 17:58:45.945515236 (512h14m31.304173892s) [upload=451109]: Starting upload Dec 2 17:58:45.945584251 (512h14m31.304242898s) [upload=451109]: Done starting upload Dec 2 17:58:45.945681031 (512h14m31.304339673s) [upload=451109]: Uploading segments... Dec 2 17:58:45.945742287 (512h14m31.304400949s) [upload=451109]: Got next segment. Inline: true Dec 2 17:58:45.945747963 (512h14m31.304406605s) [upload=451109]: Waiting for error group managing segments... Dec 2 17:58:45.945800260 (512h14m31.304458906s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:45.945808052 (512h14m31.304466716s) [upload=451109]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:45.945812113 (512h14m31.304470753s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:31.489436266 (512h15m16.848094929s) [upload=451109]: Done uploading segments... Dec 2 17:59:31.489469001 (512h15m16.848127665s) [upload=451109]: Upload finished. err: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer Duration:* 880.177846ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.9311685s trace: https://jaeger.dev.storj.io/trace/7eb4a8f9dded1e1f FAIL (2025-12-02 18:02:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.931251925s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 290.071815ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9ee9e5385f36196 FAIL (2025-12-02 18:01:40 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.57.18:443: connect: connection refused Duration: 290.100667ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.149390847s trace: https://jaeger.dev.storj.io/trace/3b7a4a4ca5a9a00a FAIL (2025-12-02 18:01:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer ======LOGS====== Dec 2 17:58:45.945515236 (512h14m31.304173892s) [upload=451109]: Starting upload Dec 2 17:58:45.945584251 (512h14m31.304242898s) [upload=451109]: Done starting upload Dec 2 17:58:45.945681031 (512h14m31.304339673s) [upload=451109]: Uploading segments... Dec 2 17:58:45.945742287 (512h14m31.304400949s) [upload=451109]: Got next segment. Inline: true Dec 2 17:58:45.945747963 (512h14m31.304406605s) [upload=451109]: Waiting for error group managing segments... Dec 2 17:58:45.945800260 (512h14m31.304458906s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:45.945808052 (512h14m31.304466716s) [upload=451109]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:45.945812113 (512h14m31.304470753s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:31.489436266 (512h15m16.848094929s) [upload=451109]: Done uploading segments... Dec 2 17:59:31.489469001 (512h15m16.848127665s) [upload=451109]: Upload finished. err: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer Duration:* 1.149417329s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 298.526994ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9ee9e5385f36196 FAIL (2025-12-02 18:01:17 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.57.18:443: connect: connection refused Duration: 298.552025ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.670720476s trace: https://jaeger.dev.storj.io/trace/7eb4a8f9dded1e1f FAIL (2025-12-02 18:01:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.670806595s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 921.479813ms trace: https://jaeger.dev.storj.io/trace/3b7a4a4ca5a9a00a FAIL (2025-12-02 18:00:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer ======LOGS====== Dec 2 17:58:45.945515236 (512h14m31.304173892s) [upload=451109]: Starting upload Dec 2 17:58:45.945584251 (512h14m31.304242898s) [upload=451109]: Done starting upload Dec 2 17:58:45.945681031 (512h14m31.304339673s) [upload=451109]: Uploading segments... Dec 2 17:58:45.945742287 (512h14m31.304400949s) [upload=451109]: Got next segment. Inline: true Dec 2 17:58:45.945747963 (512h14m31.304406605s) [upload=451109]: Waiting for error group managing segments... Dec 2 17:58:45.945800260 (512h14m31.304458906s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:45.945808052 (512h14m31.304466716s) [upload=451109]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:45.945812113 (512h14m31.304470753s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:31.489436266 (512h15m16.848094929s) [upload=451109]: Done uploading segments... Dec 2 17:59:31.489469001 (512h15m16.848127665s) [upload=451109]: Upload finished. err: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer Duration:* 921.509083ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 326.32036ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9ee9e5385f36196 FAIL (2025-12-02 18:00:19 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.57.18:443: connect: connection refused Duration: 326.364491ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 932.522763ms trace: https://jaeger.dev.storj.io/trace/3b7a4a4ca5a9a00a FAIL (2025-12-02 17:59:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer ======LOGS====== Dec 2 17:58:45.945515236 (512h14m31.304173892s) [upload=451109]: Starting upload Dec 2 17:58:45.945584251 (512h14m31.304242898s) [upload=451109]: Done starting upload Dec 2 17:58:45.945681031 (512h14m31.304339673s) [upload=451109]: Uploading segments... Dec 2 17:58:45.945742287 (512h14m31.304400949s) [upload=451109]: Got next segment. Inline: true Dec 2 17:58:45.945747963 (512h14m31.304406605s) [upload=451109]: Waiting for error group managing segments... Dec 2 17:58:45.945800260 (512h14m31.304458906s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:45.945808052 (512h14m31.304466716s) [upload=451109]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:45.945812113 (512h14m31.304470753s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:31.489436266 (512h15m16.848094929s) [upload=451109]: Done uploading segments... Dec 2 17:59:31.489469001 (512h15m16.848127665s) [upload=451109]: Upload finished. err: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer Duration:* 932.654413ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.793992565s trace: https://jaeger.dev.storj.io/trace/7eb4a8f9dded1e1f FAIL (2025-12-02 17:59:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.794008376s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001505746s trace: https://jaeger.dev.storj.io/trace/7eb4a8f9dded1e1f FAIL (2025-12-02 17:59:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001532231s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 288.144561ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9ee9e5385f36196 FAIL (2025-12-02 17:59:36 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.57.18:443: connect: connection refused Duration: 288.167719ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 45.835298856s trace: https://jaeger.dev.storj.io/trace/3b7a4a4ca5a9a00a FAIL (2025-12-02 17:59:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer ======LOGS====== Dec 2 17:58:45.945515236 (512h14m31.304173892s) [upload=451109]: Starting upload Dec 2 17:58:45.945584251 (512h14m31.304242898s) [upload=451109]: Done starting upload Dec 2 17:58:45.945681031 (512h14m31.304339673s) [upload=451109]: Uploading segments... Dec 2 17:58:45.945742287 (512h14m31.304400949s) [upload=451109]: Got next segment. Inline: true Dec 2 17:58:45.945747963 (512h14m31.304406605s) [upload=451109]: Waiting for error group managing segments... Dec 2 17:58:45.945800260 (512h14m31.304458906s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:45.945808052 (512h14m31.304466716s) [upload=451109]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:45.945812113 (512h14m31.304470753s) [upload=451109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:31.489436266 (512h15m16.848094929s) [upload=451109]: Done uploading segments... Dec 2 17:59:31.489469001 (512h15m16.848127665s) [upload=451109]: Upload finished. err: metaclient: manager closed: closed: read tcp 139.59.144.22:40954->35.212.57.18:7777: read: connection reset by peer Duration:* 45.835322529s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 91.786772ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/9ee9e5385f36196 FAIL (2025-12-02 17:59:13 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.57.18:443: connect: connection refused Duration: 91.811013ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.363473718s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:22:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.36349857s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.405313138s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:21:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.405339063s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.403418475s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:20:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.403459811s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.348276401s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:20:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.348320122s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.333034781s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:19:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.33306032s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.24205596s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:19:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.242075653s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.293438994s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:18:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.293472645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.301136401s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:17:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.301260385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.316577883s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:17:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.316602694s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.374321059s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:17:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.374358126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.401788419s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:15:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.401818956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.440940655s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:15:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.440965212s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.370607504s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:14:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.370634755s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.38833719s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:14:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.388369657s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.269643136s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:13:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.269667079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.397875367s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:12:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.397902926s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.334297599s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:12:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.334324319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.393795868s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:11:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.393820182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.394280681s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:10:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.394302555s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.356752511s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:09:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.35678707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.330413492s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:09:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.33043571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.453249161s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:08:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.453278126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.428327222s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:08:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.42835253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.252399178s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:07:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.252427451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.397699513s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:06:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.397732274s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.232059276s trace: https://jaeger.dev.storj.io/trace/2ffe8f5892e19f49 FAIL (2025-12-02 16:06:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 2 15:39:52.392007388 (509h55m37.750666054s) [upload=449040]: Starting upload Dec 2 15:39:52.392102781 (509h55m37.750761452s) [upload=449040]: Done starting upload Dec 2 15:39:52.392305989 (509h55m37.75096465s) [upload=449040]: Uploading segments... Dec 2 15:39:52.392711356 (509h55m37.751370001s) [upload=449040]: Got next segment. Inline: false Dec 2 15:39:52.392823756 (509h55m37.751482418s) [upload=449040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 15:39:52.392830880 (509h55m37.751489542s) [upload=449040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 15:39:52.505601339 (509h55m37.864260002s) [seg_pos={0 0}, upload=449040]: Begin upload segment... Dec 2 15:39:52.505683181 (509h55m37.864341844s) [seg_pos={0 0}, upload=449040]: Done begin upload segment. Dec 2 15:39:52.505705240 (509h55m37.8643639s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.505747612 (509h55m37.864406256s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506719799 (509h55m37.865378464s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.506895039 (509h55m37.865553693s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507261884 (509h55m37.86592054s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507305797 (509h55m37.865964477s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507578508 (509h55m37.866237164s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507898679 (509h55m37.866557344s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507916888 (509h55m37.866575548s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.507995355 (509h55m37.866654026s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508291219 (509h55m37.866949876s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508548472 (509h55m37.867207134s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508607537 (509h55m37.867266189s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.508991378 (509h55m37.867650034s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509080273 (509h55m37.867738934s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.509341008 (509h55m37.867999664s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Uploading piece... Dec 2 15:39:52.893602632 (509h55m38.252261292s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ5SCGOUAAKTWEWJAUBAKNV2QEYLNMUTWMS7FXNWFV2VK5NY3TQA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 2 15:39:53.109975868 (509h55m38.468634531s) [upload=449040]: Next returned nil segment Dec 2 15:39:53.109995023 (509h55m38.468653687s) [upload=449040]: Waiting for error group managing segments... Dec 2 15:39:53.247005971 (509h55m38.605664624s) [noise=true , address=145.79.227.16:4912 , piece_id=4CVNXYVSVELUY3HNAG5JDTCPVI2GTRLCQAJR4SOT4BZKJHQLXLJA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.247605367 (509h55m38.60626401s) [noise=true , address=145.79.227.14:4839 , piece_id=XO67NDOS6JUKGFEILV24CAJTPW6JGILUCITJO7DNEZ4OGBKY5HRA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.249529916 (509h55m38.608188569s) [noise=true , address=145.79.227.42:4122 , piece_id=CXAKRHCYQBP7TVRH7OGTSWQY7HNIALLVZY2TGI5DIILH6L4JBOAA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251164571 (509h55m38.609823231s) [noise=true , address=145.79.227.22:4510 , piece_id=D53ILYPLISUJFK6GYDB7VDFQRGPCZ2IO2LWWMYN5BFXXHRU2TIZQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.251199454 (509h55m38.609858099s) [noise=true , address=145.79.227.11:4732 , piece_id=TVT4SVNXPJFJUZFNZDDVAS43RV7K2NRWKOPTRKWMW6ZW4DYGULTQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.253753533 (509h55m38.612412188s) [noise=true , address=145.79.227.32:4836 , piece_id=KH3B4YHCINMBSIJOQL2LWIP7PDGR2KR67JKE4M5QCVU23UTO63IA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.255282872 (509h55m38.613941533s) [noise=true , address=145.79.227.30:4761 , piece_id=DHAWQLBMHDLXGGBW62HNJBYQYGKERADHZLRY5VH6CMCTTNEQSGWA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.260847838 (509h55m38.619506486s) [noise=true , address=145.79.227.28:4725 , piece_id=GYHJ4K464NYBTCKILMM2HVULR6XZBSRPRGZB6RSJS5M73NT6TKNQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.266854607 (509h55m38.62551326s) [noise=true , address=145.79.227.10:4402 , piece_id=A2EG7EBIK37YAJHM5VNTKEHZSNBJL2PCBZMR36NCJIHVOM6QA7VA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.267907844 (509h55m38.626566502s) [noise=true , address=145.79.227.40:4051 , piece_id=LHSPZJ74HRRA35YJ4AZA2ZSZUAA5AKFBLLFFEGYGA3DEIYVUHD5A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.271617855 (509h55m38.630276511s) [noise=true , address=145.79.227.34:4906 , piece_id=XFMTIRY5ACX36QQ3DEAZ5GVQKXUBXS24FDQDUQ4VSYCI7B6AOQMQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:39:53.301319613 (509h55m38.659978256s) [noise=true , address=145.79.227.15:4875 , piece_id=DMPVMP6MWFE5GKUSE5OPE2LOGXF43MZDVBRXIE7Y66TTZEYWB54A, seg_pos={0 0}, upload=449040]: Done uploading piece. err: <nil> Dec 2 15:40:22.106950271 (509h56m7.465608932s) [noise=true , address=77.90.31.13:4628 , piece_id=ULCE6RPRLQW3VULCE6JQMIJWEX4L77P5XTVZQFJ75LXFTP5ADCKQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107074744 (509h56m7.465733407s) [noise=true , address=77.90.31.17:4772 , piece_id=P3U5PEEHXZQPWTYP3CZJLPXOWBODLV7IUX5S774QWTZACU62T6WA, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107108017 (509h56m7.465766683s) [noise=true , address=77.90.31.12:4590 , piece_id=VDBM3CG2Y723ETXWVHNUWBZW7KPZU7VWGUNCRHZYEVCZMEWYTFHQ, seg_pos={0 0}, upload=449040]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 2 15:40:22.107140688 (509h56m7.465799336s) [seg_pos={0 0}, upload=449040]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 2 15:40:22.215394440 (509h56m7.574053097s) [upload=449040]: Done uploading segments... Dec 2 15:40:22.215426913 (509h56m7.574085566s) [upload=449040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.23208776s core uplink us-select-1

Orbiter Check History

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)