Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.600875276s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 12:09:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.600927863s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.614924869s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 12:08:03 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.614959042s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.589832917s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 12:07:43 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.589856718s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.583003583s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 12:07:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.583023572s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.471578282s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 12:05:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.471602332s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.748788101s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 12:04:46 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.748831129s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.570462535s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 12:04:06 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.57055342s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.848566128s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 12:03:03 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.84859335s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.682524357s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 12:01:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.682549765s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.613370345s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 12:01:25 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.613404746s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.577934354s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 12:01:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.578014139s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.748100221s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 12:00:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.748127628s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.637856569s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 11:59:44 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.637879558s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 3.58484384s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 11:58:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 3.584867799s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.710025319s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 11:57:56 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.710047501s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.476471844s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 11:57:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.476498564s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.580807811s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 11:57:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.580828247s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.746593973s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 11:56:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.746623011s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.778264019s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 11:55:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.77829465s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.718832123s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 11:54:29 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.718860687s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.580488691s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 11:53:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.580510924s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.709554955s trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 11:52:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 2.709571811s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 175.833412ms trace: https://jaeger.dev.storj.io/trace/2dc2a1f8af543fa2 FAIL (2025-11-30 11:51:35 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 139.59.144.22:54862->45.38.24.2:443: read: connection reset by peer Duration: 175.855517ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.231527104s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:48:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.231550714s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.627633935s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:47:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.627658137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.255884663s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:47:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.255908341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.223299574s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:46:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.223323545s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 2.750585669s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:46:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.750638311s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.653607307s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:45:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.653696856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 2.170254379s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:45:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.170281647s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 2.894003992s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:45:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.894027942s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [XX..........................................................] duration: 1.650182434s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:44:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.650221356s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 2.938004827s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:44:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.938025934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 2.238450637s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:44:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.23847663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 2.259565566s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:44:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.2595887s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [XXX.........................................................] duration: 1.606257679s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:44:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.606284701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XXXXX......................................................] duration: 2.216792039s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:43:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.216888574s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.XXXXX......................................................] duration: 2.701826783s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:43:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.701863922s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [XXXX........................................................] duration: 1.655048794s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:43:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.655068052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XXXXX.....................................................] duration: 2.251408404s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:43:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.251437378s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..XXXXX.....................................................] duration: 3.424529623s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:42:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42455419s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...XXXXX....................................................] duration: 2.823904741s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:42:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.823932919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XXXXX....................................................] duration: 2.753657771s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:42:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.753682179s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [XXXXX.......................................................] duration: 1.667848396s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:42:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.667868834s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....XXXXX...................................................] duration: 2.648381004s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:41:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.648441023s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [XXXXXX......................................................] duration: 1.53355015s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:41:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.533609082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XXXXX...................................................] duration: 2.357130122s trace: https://jaeger.dev.storj.io/trace/36e3bc8bda2ae888 FAIL (2025-11-30 10:41:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 2576266 (456h17m29.151234909s) [seg_pos={0 0}, upload=401887]: Done begin upload segment. Nov 30 10:01:43.792620520 (456h17m29.151279174s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.792633491 (456h17m29.151292131s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.793152014 (456h17m29.151810673s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794267287 (456h17m29.152925947s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.794838404 (456h17m29.153497068s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795320876 (456h17m29.153979536s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.795747053 (456h17m29.154405709s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796165504 (456h17m29.15482416s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796569328 (456h17m29.155227981s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.796946295 (456h17m29.155604949s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.797440109 (456h17m29.156098768s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.798005817 (456h17m29.156664474s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799199941 (456h17m29.157858604s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.799761456 (456h17m29.158420118s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.800613201 (456h17m29.15927186s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.801408218 (456h17m29.160066874s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.802041112 (456h17m29.160699773s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:43.803547763 (456h17m29.162206408s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Uploading piece... Nov 30 10:01:44.360918117 (456h17m29.71957677s) [upload=401887]: Next returned nil segment Nov 30 10:01:44.360933446 (456h17m29.71959209s) [upload=401887]: Waiting for error group managing segments... Nov 30 10:01:44.499773587 (456h17m29.858432244s) [noise=true , address=45.91.116.13:4168 , piece_id=X7JS23Q2FXCJ2P2GDU2ZYEYK2S2RI46YWCAEIHJDX2H6LPZKQIKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.505430570 (456h17m29.864089224s) [noise=true , address=145.79.227.6:4285 , piece_id=7DYS5X6EVGOO2HISQXD5WLR2LNPORSEHHR5UGOMWLHALUFHK37UQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.506366124 (456h17m29.86502478s) [noise=true , address=45.91.116.7:4027 , piece_id=36KFPMWAHSUDAHTAR4YJY3OK7RFQLKAF66TLOT7VCH3TW5BQUEIQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.510655401 (456h17m29.869314054s) [noise=true , address=45.91.116.11:4132 , piece_id=SYDKBCUNGHTRGV6AL5Y4ODEROJCKT76W2JWDBYCJ5NVFON2BEVAA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.515821839 (456h17m29.874480493s) [noise=true , address=45.91.116.14:4222 , piece_id=73OXOA2DTE6DWPXZZV22XPX5NSORXKZNHCH3FKO4ACUMN7KDPTTA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.516416836 (456h17m29.875075489s) [noise=true , address=45.91.116.12:4253 , piece_id=T2HYIYUXVDN33ZGSURFK3BRE5CXSNWFQY2QSJQRB6WHGSPP7XLNA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.522437373 (456h17m29.881096026s) [noise=true , address=45.91.116.9:4270 , piece_id=FM7NSFNGRDAM366ULEXUPLFWYVK7CSOFMW7TH7JZGLI43EWY5UMA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:44.540356778 (456h17m29.899015431s) [noise=true , address=145.79.227.2:4133 , piece_id=IOQEMS3KZBLVOPYGCKXKVNTLZF7YTM6WAANM3FOW6KGSZMNS5CKQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.026323057 (456h17m30.384981714s) [noise=true , address=64.49.13.170:4723 , piece_id=VQKRP7TXFNTYKKNZ6BUABBH5XBBCRTQG3AZXBJY3T2UJLHQLOMWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.067010906 (456h17m30.425669563s) [noise=true , address=64.49.13.138:4484 , piece_id=FDOU4SYBFZYQFJOWUT5FTGGGB4I2IPGSZN6SOBQ44JXVHQLQRSPQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:01:45.840961184 (456h17m31.199619846s) [noise=true , address=64.49.13.169:4716 , piece_id=2VB4N3T6KHMT5FXGFWX3MZQGMDXBVWAIY47MLQCXY7L6DUOTE6SA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: <nil> Nov 30 10:02:43.384772242 (456h18m28.743430905s) [noise=true , address=77.90.31.13:4629 , piece_id=R4QNY2ZLRBPMAWTEV3FJAHZQHJ4COZDMBNZXFYWE5YE4WSZNVK6A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: rpc: dial tcp 77.90.31.13:4629: i/o timeout; conn closed: op Nov 30 10:02:43.385027162 (456h18m28.74368582s) [noise=true , address=77.90.31.14:4667 , piece_id=VLNAVDHAOFG5SMD2YFOHX247M2S3YZHSI7KK64JMRTQXTQRBBRVA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 139.59.144.22:33148->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385252549 (456h18m28.743911212s) [noise=true , address=77.90.31.11:4575 , piece_id=UFSMDSSLIY5WVUM5C4D3AOQTJP3B7TKUJGWTJHPDXIFEEP5A2GSQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: rpc: dial tcp 77.90.31.11:4575: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385294122 (456h18m28.743952778s) [noise=true , address=77.90.31.17:4794 , piece_id=KXDC4JS3JHPDWROIPQWGDMFRGU3ZXZBVOBDVIBIN42JDVAJIPIDQ, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385457671 (456h18m28.744116324s) [noise=true , address=77.90.31.12:4619 , piece_id=6CBVOQPWFUPCZZKZYSNWOOVBSWFJV6ZORDRI4C3H2ASFKZ5QWKWA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 139.59.144.22:59426->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385605228 (456h18m28.744263892s) [noise=true , address=77.90.31.15:4700 , piece_id=357WQSBXD2B3JTUQMKGDQCPOXMKS4RPKUFXPEE5TQOARN6SLJBYA, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: rpc: dial tcp 77.90.31.15:4700: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.385977352 (456h18m28.744636015s) [noise=true , address=77.90.31.19:4863 , piece_id=VP6BHB3OVD5REX7ZY7GZAW2NDZYKOKSGJL3ROBQJG6RT362SSD4A, seg_pos={0 0}, upload=401887]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 139.59.144.22:57210->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:43.386029579 (456h18m28.744688219s) [seg_pos={0 0}, upload=401887]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:43.496092186 (456h18m28.854750849s) [upload=401887]: Done uploading segments... Nov 30 10:02:43.496131377 (456h18m28.854790035s) [upload=401887]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.357154355s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....XXXXX..................................................] duration: 2.93448487s trace: https://jaeger.dev.storj.io/trace/46d2658452036a42 FAIL (2025-11-30 10:41:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.934574174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [XXXXXXX.....................................................] duration: 1.681616115s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:40:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.681673928s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [XXXXXXXX....................................................] duration: 1.593349581s trace: https://jaeger.dev.storj.io/trace/1b8f0dae11fdbec0 FAIL (2025-11-30 10:40:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.593377852s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)