Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 1.684377044s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:37:46 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.684405131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 1.651573578s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:36:59 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.651622455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 1.656150956s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:36:36 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.65618052s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 1.614028126s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:36:16 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.614058131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 1.620499031s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:35:19 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.620543083s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 1.591984513s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:34:54 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.592040716s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 1.559247606s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:34:05 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.559370551s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 1.710188058s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:33:09 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.710285501s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 1.614985547s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:31:57 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.615028003s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 1.585002637s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:31:14 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.585024543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 1.622693056s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:30:44 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.622781182s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 1.608532574s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:29:34 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.608570687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 1.782261747s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:28:45 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.782284432s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 1.739496694s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:27:43 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.739528056s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 1.674353322s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:27:02 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.674402817s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 1.74070784s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:26:08 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.740735984s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 1.734284735s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:25:45 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.734389213s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 1.651383958s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:25:29 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.651414156s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 1.588674219s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:24:20 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.588690224s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 1.634032855s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:23:22 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.634061506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 1.667137403s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:22:26 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.667199877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 1.574811583s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:21:51 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.574908026s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 1.632680239s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:21:14 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.632733347s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 1.60389976s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:20:46 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.60395195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 1.658283912s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:19:44 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.658327682s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 1.593460718s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:18:55 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.593488269s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 1.755725074s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:17:40 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.755757579s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 2.191566799s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:16:59 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.191593322s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 2.070845579s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:16:27 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.070876282s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 1.632759017s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:16:03 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.632833954s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 1.594041443s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:14:53 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.594070907s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 1.627530403s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:14:32 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.627560016s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 2.199343107s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:13:31 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.199443669s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 1.67751448s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:12:23 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.677549095s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 1.623081756s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:11:38 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.623110861s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.000455215s trace: https://jaeger.dev.storj.io/trace/39d574be3359eb4b FAIL (2025-11-15 13:10:36 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.000483179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.474244819s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:22:51 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.474273029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.359014299s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:21:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.35904595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.362747918s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:21:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.362774763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.384675722s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:19:57 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.384706223s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.379229061s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:19:40 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.37925454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.485847059s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:18:49 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.485877272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.43297244s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:18:11 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.433048447s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.431981546s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:17:54 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.432026875s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.564048288s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:16:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.564079749s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.454208465s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:15:41 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.454238096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.525841096s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:15:06 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.525874895s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.355420109s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:14:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.35544658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.349688228s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:13:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.349730879s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.392822276s trace: https://jaeger.dev.storj.io/trace/67787e0cac7c3828 FAIL (2025-11-14 09:12:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 14 08:40:26.692841383 (70h56m12.051500041s) [upload=62364]: Starting upload Nov 14 08:40:26.692970457 (70h56m12.051629103s) [upload=62364]: Done starting upload Nov 14 08:40:26.693246503 (70h56m12.051905146s) [upload=62364]: Uploading segments... Nov 14 08:40:26.693294649 (70h56m12.051953293s) [upload=62364]: Got next segment. Inline: false Nov 14 08:40:26.693330647 (70h56m12.051989289s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 08:40:26.693335584 (70h56m12.051994229s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 08:40:26.808983574 (70h56m12.167642242s) [seg_pos={0 0}, upload=62364]: Begin upload segment... Nov 14 08:40:26.809046566 (70h56m12.167705237s) [seg_pos={0 0}, upload=62364]: Done begin upload segment. Nov 14 08:40:26.809084656 (70h56m12.167743323s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810175202 (70h56m12.168833877s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.810734490 (70h56m12.169393151s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811182432 (70h56m12.169841087s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811571433 (70h56m12.17023009s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.811956069 (70h56m12.170614714s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.812389686 (70h56m12.171048351s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813083897 (70h56m12.171742553s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.813648842 (70h56m12.172307501s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814395676 (70h56m12.173054334s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.814752808 (70h56m12.173411475s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.815470873 (70h56m12.174129539s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816142015 (70h56m12.174800676s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.816383714 (70h56m12.175042375s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.817087968 (70h56m12.175746625s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:26.818368786 (70h56m12.177027454s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Uploading piece... Nov 14 08:40:27.437707056 (70h56m12.796365715s) [upload=62364]: Next returned nil segment Nov 14 08:40:27.437724331 (70h56m12.796382992s) [upload=62364]: Waiting for error group managing segments... Nov 14 08:40:27.569530874 (70h56m12.928189531s) [noise=true , address=145.79.227.10:4402 , piece_id=CA36IXHMNLURLXBPT7WCMNJ74JRDKTFXERPUQP5E5YNJP5JEZ4GA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.571232736 (70h56m12.929891413s) [noise=true , address=145.79.227.4:4186 , piece_id=UZDOLJFECWAARBXQYPRHWAIP63AOJLV25EJ4FUNAHB7PRDVXZKJQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572208805 (70h56m12.930867472s) [noise=true , address=145.79.227.13:4804 , piece_id=BD7R4OASY3HAN4DEFHIZWVDECMYXIG7QSQMV76RL5MFP2OJODDAQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.572976155 (70h56m12.931634814s) [noise=true , address=77.90.31.13:4626 , piece_id=YNFXCZUTYLJZS4MQDXYKQZJRYNBWUMMUSDL523I5TTD2TVVEATEA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575728483 (70h56m12.934387146s) [noise=true , address=77.90.31.12:4589 , piece_id=LCW5FRSNNUD5LYJMDM4ETMWCDP66FSZDQ45VUQ5J6J35ANSLRVKA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.575918460 (70h56m12.93457716s) [noise=true , address=145.79.227.20:4438 , piece_id=MNR2QJONSMIKQGZMHWYHEFKIYIBUQTRNBLAYPNG6ZOLRME6536YA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.577875305 (70h56m12.936533967s) [noise=true , address=45.91.116.12:4227 , piece_id=AVGITE54UTGGANCKL2CVO2RBA5W2BOSGQYQGXQ4HDTK5IUG5CGLQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579308054 (70h56m12.93796675s) [noise=true , address=145.79.227.11:4731 , piece_id=7UZTZAE4KFSPZFUM6KB2YASKEEEY5BV6MGHLYC3XHZTSVK4QN4LQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.579390270 (70h56m12.938048936s) [noise=true , address=145.79.227.9:4367 , piece_id=SXRYK52XR4U2AMD5XYTYXOOC32H7TRA5WF7OI6SSL5C3SDXZBRGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.581638963 (70h56m12.940297621s) [noise=true , address=145.79.227.8:4329 , piece_id=QJMNQF6CYYEFYFK3QJ3FAKYX36CDG5V2P2NA3Z5O6MLDDWMHAHGA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.585832944 (70h56m12.944491603s) [noise=true , address=45.91.116.4:4939 , piece_id=ZMNYTEM2CS2WMT7NZOKJSZ2U6VVI5J6PH2AUPGHL6MY4PBILW6XA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.586634186 (70h56m12.945292844s) [noise=true , address=145.79.227.27:4691 , piece_id=YYAPWIT7ACP7CWPBCU3Z5WYQJQUKEGZCL2UMUYCJKXKC5SQ4F46A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587390241 (70h56m12.946048898s) [noise=true , address=145.79.227.8:4331 , piece_id=2IRPV56WKYI6M6ZXIKOD7DLJKWVGGQDXTNC2X5FPXBNYHRYTNR3A, seg_pos={0 0}, upload=62364]: Done uploading piece. err: <nil> Nov 14 08:40:27.587404611 (70h56m12.946063269s) [seg_pos={0 0}, upload=62364]: Segment reached optimal threshold of 13 pieces. Nov 14 08:40:27.587824879 (70h56m12.946483548s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.587842892 (70h56m12.946501555s) [noise=true , address=145.79.227.14:4840 , piece_id=HMYC7ZJZY7DUXAM7UIVCKG2Z2KSC2K63UWNVGTMBXH2OI7AZUCTA, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588520027 (70h56m12.947178694s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588529907 (70h56m12.947188568s) [noise=true , address=145.79.227.16:4911 , piece_id=2LZZHNF72FDPJHDW65GIU3EEZTDISKRJCPA656P6KQKETEHFEYHQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588814450 (70h56m12.947473116s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Nov 14 08:40:27.588824167 (70h56m12.947482826s) [noise=true , address=145.79.227.23:4546 , piece_id=3NROSGRLS4BKJGKHCXPAFPRWILSTUWSJB44TPXX2G25E3DY33XXQ, seg_pos={0 0}, upload=62364]: Long tail cancellation due to optimal threshold Nov 14 08:40:27.588844312 (70h56m12.94750296s) [seg_pos={0 0}, upload=62364]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 14 08:40:27.588946526 (70h56m12.947605181s) [upload=62364]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 08:40:27.588951905 (70h56m12.94761056s) [upload=62364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 08:40:27.747513897 (70h56m13.106172563s) [upload=62364]: Done uploading segments... Nov 14 08:40:27.747540051 (70h56m13.106198707s) [upload=62364]: Upload finished. err: <nil> Duration: 2.392841502s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)