Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 4.001446763s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:39:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.001474876s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 3.111072729s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:39:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.111114524s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 3.049363246s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:38:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.049397902s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.825947807s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:36:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.825983518s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.894822149s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:36:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.894850957s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.935672947s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:35:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.935749236s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 3.448500746s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:35:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.44861443s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.890538915s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:34:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.890570766s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 3.08162584s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:33:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.081655074s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.898326386s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:32:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.898360238s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.959487918s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:32:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.959534645s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 9.990809874s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:31:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.990847501s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 4.980864885s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:31:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.980882352s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 3.154723243s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:30:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.154923791s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 3.038989183s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:30:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.039020926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 3.02911769s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:29:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.029157226s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.392773606s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:28:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.392801738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.851302679s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:27:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.851323649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 17.082329156s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:27:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.082392479s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 20.438425237s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:27:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.438464067s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.790549952s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:25:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.790581066s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 3.472438925s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:24:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.472470807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 3.529773638s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:24:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.529816376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.935245815s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:23:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.935278919s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 3.632126611s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:22:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.63215627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.81516377s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:21:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.815198582s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.906427319s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:20:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.906492118s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.861707367s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:19:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.861752673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.763455973s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:18:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.76348577s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.801914714s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:17:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.801944747s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.905273002s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:16:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.90537744s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.904833398s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:15:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.904864035s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 3.462546792s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:14:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46259909s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.896412156s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:13:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.896450995s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.069753892s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:13:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.069793487s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.864595887s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:12:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.864622703s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 11.031569112s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:11:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.031622245s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.928917156s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:11:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.928957199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002247188s trace: https://jaeger.dev.storj.io/trace/5b2ca91c55ae8dcb FAIL (2026-04-18 15:10:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00227023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.148097912s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:23:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.148127715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.060488243s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:22:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.060519559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.374304287s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:22:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.374339125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 3.377740582s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:21:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.377778572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 3.282984623s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:20:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.283018897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 3.259596927s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:19:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.259621356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 3.091445387s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:19:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.091480627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 3.112139392s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:18:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.112231259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 3.306579538s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:17:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.306621018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 3.022040639s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:16:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.022072121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 3.105696473s trace: https://jaeger.dev.storj.io/trace/1906885ee3de6780 FAIL (2026-04-18 04:15:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== false Apr 18 03:34:19.382418238 (251h17m29.976529486s) [upload=241587]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 18 03:34:19.382425643 (251h17m29.976536889s) [upload=241587]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 18 03:34:19.489113571 (251h17m30.08322481s) [seg_pos={0 0}, upload=241587]: Begin upload segment... Apr 18 03:34:19.489164266 (251h17m30.083275505s) [seg_pos={0 0}, upload=241587]: Done begin upload segment. Apr 18 03:34:19.489191011 (251h17m30.083302251s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489420655 (251h17m30.083531905s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.489836704 (251h17m30.083947946s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490219834 (251h17m30.084331075s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490246383 (251h17m30.084357638s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490519068 (251h17m30.084630313s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490881441 (251h17m30.08499269s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.490984670 (251h17m30.085095912s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491405927 (251h17m30.085517173s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491500186 (251h17m30.085611433s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.491818491 (251h17m30.085929741s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492083388 (251h17m30.086194635s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492268518 (251h17m30.086379763s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492884090 (251h17m30.086995347s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.492980843 (251h17m30.087092082s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.493502906 (251h17m30.087614154s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494186874 (251h17m30.088298122s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.494829557 (251h17m30.088940819s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Uploading piece... Apr 18 03:34:19.583251845 (251h17m30.177363095s) [noise=true , address=145.79.227.43:4178 , piece_id=ULJRDROBFLH7CTEDPX6HHKVQRGBDWF3R2FJ4IWLUWFBEV6TXVLHA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:145.79.227.43:4178): protocol: rpc: dial tcp 145.79.227.43:4178: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.586345429 (251h17m30.180456685s) [noise=true , address=145.79.227.44:4223 , piece_id=4DY3T3GLORZUK2LKXBUN4PMWIZIDPJJROX5KYLT2SJZ6PWKUAGXQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:145.79.227.44:4223): protocol: rpc: dial tcp 145.79.227.44:4223: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:19.587384451 (251h17m30.181495694s) [noise=true , address=145.79.227.31:4818 , piece_id=TDVOR6X2ZHAZSUIKZJYN6F2Q6LKGOQXGJU6L6IC5BCLFGAEBNIFA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: rpc: dial tcp 145.79.227.31:4818: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:34:20.074197698 (251h17m30.668308944s) [upload=241587]: Next returned nil segment Apr 18 03:34:20.074245186 (251h17m30.668356436s) [upload=241587]: Waiting for error group managing segments... Apr 18 03:34:20.198233132 (251h17m30.792344378s) [noise=true , address=145.79.227.38:4990 , piece_id=FNEGJ6Q6T67LNHJJVJGDXDRZNBTFXUSGC3A74W6Q2JJLOPS6VS7Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.205170998 (251h17m30.79928224s) [noise=true , address=145.79.227.36:4541 , piece_id=LZZHFLVN5Z3LETWMXMX5FXI4BEXKT272OPTXO2MGVYKUCQ7CHB6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.212980107 (251h17m30.807091348s) [noise=true , address=145.79.227.46:4616 , piece_id=7K4RGYCVTYGOWNCP26BRVCRPTUNBENNXHDJGIBVLVDRTGRLJ3MAQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.311455208 (251h17m30.905566456s) [noise=true , address=145.79.227.34:4936 , piece_id=XHQAOVDOTEKEC2NB2S6QFGM3ACIIC6FXIZK3A23VGZK7R4NPN6OA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.555251091 (251h17m31.149362331s) [noise=true , address=167.17.72.69:4717 , piece_id=724S4YMBV3VXVMILPGFN23HBLCFE2PVVVTA4HWZSJJESMNYCUBZA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.638093469 (251h17m31.232204708s) [noise=true , address=167.17.72.71:4807 , piece_id=E6XKS72PAZYEIB66ZVMJWP3I5GKJX4E6IFYJLQXV53R7MDSUJFNQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.725727671 (251h17m31.319838922s) [noise=true , address=193.149.164.66:4976 , piece_id=MGFIR7KUOJXTFNSPKM7BBB2H7QXOMTXP5DSNINLXKALQV6CULMMQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.727286602 (251h17m31.321397914s) [noise=true , address=193.149.164.71:4305 , piece_id=SAEJ2LUAXLUBHB6LNBZSHMPQMWKFQMLTASC5CENVTOYWF23FJSHQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.816633773 (251h17m31.410745028s) [noise=true , address=79.127.205.234:4204 , piece_id=6GICF252TTY6ZX3DSN5PP3L46K6ZQBPBEBZWZVELS5KXCTRFTVSA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.828252631 (251h17m31.422363873s) [noise=true , address=79.127.205.233:4142 , piece_id=2G2NEYDUNILKKT3LOZL5BA6IUT6EEVADMIZ47LY6FWEMH5G3XXTQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.831306687 (251h17m31.425417933s) [noise=true , address=79.127.205.235:4279 , piece_id=XXNNYO4UROL2BACFWEKYPAWSCLMK2EQKGQP2MAVHL47SCKFKFCSQ, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:20.922081478 (251h17m31.516192719s) [noise=true , address=79.127.205.237:4226 , piece_id=3OYM7ZE5IME23EZGC2C4XH47AP7RPGRP6XXUJ36T7YFTAVKSOO6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:34:21.294400155 (251h17m31.888511403s) [noise=true , address=145.79.227.11:4748 , piece_id=QXK6G4TA7SID67SEGN5ALIURZPBOI74UJP7MPLW2IA4XJNKMBE6Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/84/log-0000000000022884-00005052: no space left on device; EOF Apr 18 03:34:21.480018086 (251h17m32.074129332s) [noise=true , address=145.79.227.33:4904 , piece_id=BRY4N3XO56TDP2LPGZDCNM6BQEIXVZ6ZE37A73D3UEQG46JDNW2Q, seg_pos={0 0}, upload=241587]: Done uploading piece. err: <nil> Apr 18 03:35:19.091643794 (251h18m29.685755045s) [noise=true , address=145.79.227.49:4350 , piece_id=PVXBW7DRG2HERIHOSTBFQJR2LLQVYQCGHKKKJ62FSHL2YHDJ4WWA, seg_pos={0 0}, upload=241587]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:145.79.227.49:4350): protocol: rpc: dial tcp 145.79.227.49:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 03:35:19.091823775 (251h18m29.685935004s) [seg_pos={0 0}, upload=241587]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 18 03:35:19.194354883 (251h18m29.788466133s) [upload=241587]: Done uploading segments... Apr 18 03:35:19.194403378 (251h18m29.788514638s) [upload=241587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.105726988s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)